Hiring Scorecards: How to Turn a Job Description Into Weighted Evaluation Criteria

A Director of Engineering at a 40-person AI startup and a Director of Engineering at a 4,000-person services company are not the same job. Same title, same seniority, often the same salary band. Entirely different bar. Any hiring scorecard that gives one candidate one score across both roles is measuring the title, not the job.

That is the practical failure behind most wasted submissions. Not bad sourcing. A definition of the role that was never specific enough to reject anyone against.

This post walks through one real role, from the job description to the eight weighted criteria the system generated, to the candidate scored against them.

Why a job description is not a hiring scorecard

A JD is a sales document. It describes the role attractively enough that someone applies. It is written for the candidate, and it succeeds when people want the job.

A scorecard is the opposite. It is written for the people making decisions, and its job is to make rejection consistent. Two recruiters reading the same JD will reject different people. Two recruiters working from the same weighted scorecard will not.

The gap between them is where the cost sits. A vague brief produces a search against nothing in particular, a screening call that asks generic questions, and a submission that gets rejected three weeks later for a requirement nobody wrote down. The same failure shows up in req qualification, where the questions that would have surfaced it were never asked.

One real role, eight weighted criteria

Take a live role: Director of Engineering, India (Site Lead), for an AI company standing up a new India site. Full-time, Bangalore, one position, ₹1.5 to 2 crore.

The job description covers the usual ground. Build and lead the engineering team. Own delivery and outcomes. Maintain technical standards. Stay hands-on. Collaborate with US-based researchers. Establish and run the site. Shape the culture from the start.

From that, the system generated eight evaluation criteria, each with a scoring guideline and a priority weight from 1 to 5. Candidates are rated out of 5 on each one, and the final score is the weighted average using those importance weights.

Why criteria weights decide the shortlist

Look at what the weighting does to this role.

Reinforcement Learning Knowledge sits at 2. On a role at an AI company, the instinct is to make that a must-have. But this is a site lead hire. The person has to build a team and stand up an office. A brilliant RL engineer who has never hired anyone fails this role, and the weights say so before anyone wastes a call on it.

Meanwhile India Site Leadership sits at 3, above the RL knowledge and above academic pedigree. That is the client's actual risk, and a list of unweighted criteria would have buried it among seven other things that all looked equally mandatory.

This is also the part clients are best at correcting. Most hiring managers cannot write a good brief from a blank page. Almost all of them can look at eight criteria with weights and say "swap those two." Send the scorecard, not the questionnaire.

How to get client sign-off on the weights

Send it before the search starts, not after the first rejection. Three practical rules:

• Send it as a decision, not a draft. "Here is how we will evaluate. Tell us what to change" gets a response. "What are your requirements?" does not.
• Force a trade-off. Ask which two criteria they would drop to fill the role a month sooner. The answer reorders the weights faster than any intake call.
• Log the answer against the role. Feedback recorded on the criteria improves the next scorecard for that client. Feedback recorded in an email thread improves nothing.

Adding your own criteria in plain English

The generated criteria cover what the JD says. They cannot cover what the client believes but did not write down, and that unwritten part is usually where the rejections come from.

So the criteria are not a fixed list. A recruiter can add their own, in ordinary language, and give it a weight. Something like:

That is a real hiring rule, it is genuinely how experienced recruiters think, and it is the kind of thing that normally lives in one person's head and dies when they leave. Written as a criterion, it becomes something the system reasons about: it works out the size and stage of the companies on a CV and scores accordingly.

Which produces the effect from the top of this post. The same candidate, evaluated for the same title at two different companies, gets two different scores, because the criteria and the weights are different. That is not a quirk. It is the entire point.

A second example, from a non-technical desk

The same mechanism applies away from engineering, and the weights shift in ways that are worth seeing.

Note that the CV-impressive criteria sit at the bottom on both scorecards. That is usually the sign that the weights are honest.

Testing a scorecard before you run it

A scorecard is a hypothesis about who succeeds in the role. Running it across 200 applicants before testing it means finding out you were wrong at 200 times the cost.

So test it against people whose outcome you already know. Drop in two or three benchmark CVs, one you would definitely hire and one you rejected, and watch how they score. Adjust the criteria descriptions, the weights and the scoring guidelines until the separation matches your judgement.

If the scorecard cannot tell your best hire from your clearest rejection, it is not measuring the job yet. Better to find that out with three CVs than with three weeks of submissions.

What a scorecard changes downstream

A scorecard is not documentation. It is the input to everything after it.

• The search runs against the criteria rather than a job title, which is what makes natural language search useful instead of approximate.
• Screening questions come from the criteria, so the call tests the things that actually decide the hire.
• The submission summary evidences against the criteria, which is why clients read it.
• Client feedback gets logged against the criteria, so the next role for that client starts better defined.

When our own team works this way, candidates move from submission to first interview at roughly 2.5x the previous rate on technical roles and 2x on non-technical ones. Those are our observed figures across our own delivery, not a controlled study, and they vary by client and by how well the role was defined to begin with. Treat them as directional.

The mechanism behind them is unglamorous. Fewer people get submitted. The ones who do are evidenced against criteria the client agreed to before the search started.

Frequently asked questions

How to build your first scorecard

Take your last three roles that closed slowly. For each one, find the requirement that only surfaced after a rejection.

That requirement should have been a weighted criterion before the search began. The fact that it was not is the entire cost of skipping the scorecard, and it is a cost most firms pay on every role without ever putting a number on it.

Then build one scorecard for a role you are working right now, test it against two candidates you already have an opinion about, and send it to the client before you send a single CV.