Explainable AI in Hiring: Why "Black Box" Scoring Is a Growing Risk

A hiring manager gets a candidate score from an AI tool. The number looks precise. It isn't obvious how the tool arrived at it, what it weighed, or whether it would produce the same score for an equally qualified candidate next week. That gap between a confident number and an explainable reason behind it is where the real risk in AI hiring sits, and it's getting harder to ignore.

An AI score that nobody can explain isn't evidence of anything. It's a liability waiting for someone, a regulator, a rejected candidate, a plaintiff's attorney, to ask the one question it can't answer: why.

What "explainable" actually means

Explainability, transparency, and interpretability get used interchangeably, but they aren't the same thing. Transparency is knowing that an AI tool was used. Interpretability is understanding, at a technical level, how a model works. Explainability is narrower and more practical: can a specific decision, for a specific candidate, be traced back to the specific reason it happened, in language a hiring manager, a candidate, or a judge can actually follow.

A "black box" tool can be transparent about its existence and still fail the explainability test completely. Plenty of vendors will tell you an AI tool is in use. Far fewer can tell you, for the one candidate who asks, exactly which part of their response or performance drove their score down.

Why most AI scoring falls short of this bar

Generic AI layered onto resumes, keyword matching, or personality-style questionnaires tends to produce a score without a legible reason. The model infers a prediction from patterns in data the employer never sees and can't audit. When a hiring manager or legal team asks why a candidate was ranked where they were, the honest answer is often some version of "the model decided," which satisfies nobody and defends nothing.

This is also where "proprietary algorithm" becomes a red flag rather than a selling point. A vendor that can't walk you through what their scoring actually measures, in plain language, isn't protecting a trade secret. They're asking you to accept legal and reputational exposure on faith.

A practical test for real explainability

Before trusting an AI hiring tool with decisions that affect real candidates, it's worth checking it against four questions. For a longer list to take into a vendor conversation, 15 Essential Questions to Ask When Evaluating AI in Talent Acquisition covers explainability to candidates, legal teams, and executives in more depth.

Traceability. Can every score be traced back to a specific, job-relevant task the candidate actually performed, rather than an inferred trait or a resume proxy?

Consistency. Does the same quality of response produce the same score for different candidates, and can that consistency be demonstrated rather than assumed?

Accessibility. Can a non-technical stakeholder, a hiring manager, a candidate, a member of the legal team, understand the explanation without a data science background?

Contestability. If a candidate or a reviewer disagrees with a score, is there a way to see the underlying evidence and challenge it?

A tool that fails any of these isn't necessarily unusable, but it's a tool you're using without the ability to defend it later, which is a decision worth making deliberately rather than by default.

The regulatory and legal picture is catching up

This isn't a hypothetical concern. In Mobley v. Workday, a federal court allowed discrimination claims against an AI hiring vendor to proceed on the theory that the vendor acted as an "agent" of its employer customers, reasoning that turned heavily on how automated the tool was and how much decision-making authority it held. The more a tool operates as a black box making real decisions, the more exposure it can create, for the employer and potentially the vendor.

Regulation is moving the same direction. The EU AI Act classifies most AI systems used in recruitment, candidate evaluation, and related employment decisions as high-risk, and gives affected individuals a right under Article 86 to request an explanation of the main factors behind a decision made about them, with full high-risk obligations for employment systems becoming enforceable in December 2027. Closer to home, New York City's Local Law 144 already requires employers to give candidates notice when an automated tool is in use and to describe what it assesses, a notice requirement that's hard to satisfy honestly if the employer itself can't explain what the tool is doing. Our Local Law 144 compliance checklist walks through those requirements step by step.

None of this means a compliant explanation process guarantees legal defensibility on its own. It means explainability is shifting from a nice-to-have to a baseline expectation, and the employers who treat it that way now will have less to untangle later.

Where Vervoe fits

Trust and explainability is one of the four principles Vervoe's assessments are built around: every decision should trace back to the work, so anyone can see why. See how this fits into our wider approach to trust and security. Vervoe's role-based assessments score candidates against real job tasks, not inferred traits or resume proxies, so a result connects to something a hiring manager, a candidate, or a legal team can actually examine. Bias testing is a separate question that often comes up alongside explainability, and our AI bias audit results are published for review.

This is a design principle, not a compliance guarantee. Explainable scoring makes a hiring decision easier to understand and discuss. It doesn't replace an employer's own legal review of how it's using any AI tool, under Local Law 144, the EU AI Act, or any other applicable law. For specific obligations, talk to counsel.

Next steps

Run your current AI hiring tools, whatever's screening, scoring, or ranking candidates today, through the four-question test above. The gaps it surfaces are usually a better starting point for a vendor conversation than any marketing claim about fairness or accuracy.

If you want to see what scoring tied to real job tasks looks like in practice, explore skills assessments.