The strongest AI on the market scores just 13.3% on a benchmark of legal practice. Hear that, and you conclude the technology isn't ready.

Yet the same model scores 93.4% on a different benchmark from the same company. The capability didn't collapse. The ruler changed.

That single fact sits at the center of today's argument about legal AI. This piece takes the Legal Agent Benchmark (LAB), released in May 2026, and reads it from the ground up: what it set out to measure, how the concrete ISDA and repo tasks are built, and why its scoring makes the headline number read far lower than the practical usefulness it reflects.

Legal Agent Benchmark Executive Summary

Throughout, we separate two registers. Facts—drawn from Harvey's official posts and press coverage—carry citations. Analysis is flagged as the editorial view.

1. What the Legal Agent Benchmark Is

Who built it, when, and why

LAB was published by the legal AI company Harvey on May 6, 2026 (source: Harvey's official blog).

The aim is clear. Earlier legal benchmarks—LegalBench, CUAD and the like—measured short tasks: inference over a passage, extraction of a clause, answers that resolve in seconds. LAB instead targets the long-horizon workflows that practicing lawyers spend tens of minutes to hours on.

That is the novelty. Drafting a contract from scratch, reviewing a counterparty's paper and writing an issues memo, surfacing policy deviations and producing an escalation document—work that ends with a submitted deliverable. Hand the whole job to an AI agent, and how usable is what comes back? That is what LAB tries to measure.

The implementation is open source under the MIT license, on GitHub at harveyai/harvey-labs. Anyone can read the scoring code and the task definitions. The leaderboard is run in partnership with an external evaluation outfit, so frontier models are compared side by side.

How it was made

This is the part everyone asks about, so we write it carefully.

The tasks originate from real client matters handled by practicing attorneys across each field (source: Harvey). Each is decomposed into the granularity of work an associate would be assigned. The instructions are short—around fifty words on average—reproducing the terse email a partner sends when handing a junior a job.

The documents inside each matter, however, are synthetic. You cannot use live confidential material, so the team builds fictional files with issues deliberately planted. The skeleton of a real matter, with the issues you want to grade seeded into it—that is the design.

The rubrics are expert-written. Harvey's acknowledgments name its in-house teams, including Applied Legal Research. What is not disclosed: the names of any outside law firms, the number or seniority of the lawyers involved, or the internal-to-external ratio. So no one can write "audited by firm X."

The defensible statement is narrower: experts wrote the rubrics starting from real attorney matters, and Harvey envisions lawyers reviewing tasks and rubrics on an ongoing basis. The time and effort that went into building it are also undisclosed.

Scale comes in two figures. At launch (the official May 6, 2026 announcement): over 1,200 tasks, 24 legal domains, over 75,000 expert-written rubric criteria. The repository has since grown, and the current GitHub badge reads 1,660 tasks. Read it as a benchmark that is still maturing.

What sets it apart from earlier benchmarks

Here the "93.4%" benchmark from the opening enters the picture. It is BigLaw Bench, published by Harvey on August 29, 2024 (source: Harvey's official blog)—the firm's older benchmark, a generation before LAB.

BigLaw Bench measures the "quality" of legal work product with two scores. One is the answer score—whether the substance of the work product is good enough. It asks "what percentage of a final, expert-lawyer-quality work product did the model complete?" The other is the source score—whether the citations are accurate. It checks whether the assertions in the deliverable are tied back to the correct documents.

And here is the first fork from LAB. BigLaw Bench is a short-horizon benchmark: from a given input, it has the model produce a single deliverable or answer and grades it. The question is bounded, and the answer comes back as one piece. Because there is partial credit—points added and subtracted by the importance of each issue—the latest models reach high scores. The 93.4% above (Fable 5), and Claude Opus 4.6's 90.2%, are BigLaw Bench numbers.

So what makes LAB different? Three points.

First, the horizon. BigLaw Bench is short-horizon—it returns a single answer to a bounded question and grades it. LAB is long-horizon and agentic. The model must walk a full matter folder on its own, judge which documents matter, and carry a multi-step piece of practice all the way to a finished deliverable. The very framing of the question differs.

Second, the scoring. BigLaw Bench evaluates "quality" as a percentage, layering partial credit onto its answer and source axes. LAB uses the all-pass scheme described below—1.0 only if every rubric criterion passes, 0.0 if a single one fails, no partial credit. BigLaw Bench measures "how well did you do," LAB asks only "did you finish all of it." The graduations on the ruler differ at the root.

Third, the consequence. The same Fable 5 scores 93.4% on BigLaw Bench and 13.3% on LAB. That gap speaks most eloquently of the difference between the two. It is the very distance between "can give a good answer to a bounded question" and "can carry a single matter through to completion, with nothing missed, against a supervising attorney's checklist."

Put another way: BigLaw Bench asks is the answer good? LAB asks can you do the whole job without dropping anything? The opening contrast—13.3% against 93.4%—was never a degradation of capability. It was born of this difference in the question.

How the scoring works—this is the crux

The most distinctive choice in LAB is how it grades.

First, grading is done not by humans but by an LLM. Each rubric criterion is scored individually by its own LLM judge—by default Claude Sonnet 4.6 at temperature 0.0—matching the deliverable against the criterion at the level of meaning. There is no golden answer file; the match_criteria written into each criterion is the test. To curb judge bias, the initial results average across multiple model families.

LLM-as-judge: a technique that hands the grading or evaluation itself to another AI (a large language model) rather than a human—think of it as an AI marking another AI's answers. / Temperature 0.0: a setting that suppresses output randomness (variability) to make the model more deterministic, so it returns almost the same judgment every time for the same input.

Second, and decisively: the all-pass scheme. A task scores 1.0 only if every criterion passes. Miss one, and the task is 0.0. There is no partial credit, and none within a criterion either—each one passes or fails.

The designers' reasoning: a due-diligence memo that catches eight of ten issues is not 80% useful. If it misses something material, it is effectively incomplete work product. For a professional who signs the deliverable, that logic holds.

Third, how instruction is given. LAB provides direction only, no playbook (directional instructions, no playbooks). The agent gets six general-purpose tools—bash, read, write, edit, glob, grep—and no normative guidance. bash, read, write, edit, glob, grep: reading and writing files, editing, searching, and running commands—an agent's most basic, general-purpose "hands and feet." No specialized legal tooling is provided; the model must move the matter forward with these alone (and, conversely, with these you can do most things). It must explore the document folder, design its own approach, and assemble the deliverable. The test is not "can you follow a given playbook" but "can you build a finished product from the instruction alone."

Where the scores stand

The published LAB all-pass scores read as follows.

LAB scores

Fable 5 leads at 13.3%. Then Claude Opus 4.8 at 10.4%, Opus 4.7 at 7.1%, Sonnet 4.6 at 5.4%, Opus 4.6 at 4.2%, GPT-5.5 at 2.1%, and Gemini 3.5 Flash at 0.8% (source: Harvey's official blog).

The smallness may surprise you. But here the opening contrast bites. The same Fable 5 scores 93.4% on Harvey's older BigLaw Bench. Capability did not fall by a factor of ten overnight. The severity of "all or nothing" moved the decimal point.

Harvey states the implication plainly: even frontier intelligence cannot yet deliver complete legal work product. That is what 13.3% means.

2. Inside the ISDA Tasks

Enough abstraction. We open the contents—four task families from the derivatives and repo corner: the ISDA Master Agreement, the CSA (credit support), the ISDA Confirmation, and repo and securities lending. Within finance, this is the hard territory: netting, collateral, regulatory capital.

ISDA: the International Swaps and Derivatives Association, which publishes the ISDA Master Agreement, the world-standard contract form for derivatives trading. / Repo: a sale of securities coupled with an agreement to buy them back later—effectively a short-term financing transaction.

How a task is assembled

The structure is shared. A title, the instructions, a list of criteria, and the documents that make up the matter folder. The instruction runs one or two sentences—a real inbound request email.

Tasks then branch by contract lifecycle stage: first-draft, counterparty-paper-review, playbook-escalation, redline, term-negotiation. Multiply that grid by instrument—Master, CSA, Confirmation, repo—and you get the lattice of tasks.

The criterion structure deserves note. In every task.json, a criterion carried exactly three keys: id, title, and match_criteria. No severity, no weight, no priority—none of these exists as a structured field. That fact matters later.

The ISDA Master pack

Take the counterparty-paper-review task (82 criteria). The instruction: acting as a named attorney for the client (Party B), review the counterparty's (Party A) draft agreement and annex against the client's negotiation playbook; cross-reference the standard form, instruction email, transmittal email, documentation checklist, adherence letter and term sheet; produce a redline and a comprehensive issues/risk memo with risk ratings on every deviation.

The folder holds an ISDA 2002 Master form, two counterparty drafts, the negotiation playbook, a term sheet, an IBOR fallbacks adherence letter, instruction emails (.eml)—the texture of a live matter.

The criteria open like this. C-001: does the memo flag that the Cross-Default Threshold is $75M in the draft versus $25M in the playbook? C-002: does it rate that deviation High risk? C-003: does it recommend reducing to $25M or CHF equivalent? One issue, split into find / rate / recommend.

The CSA (credit support)

The CSA governs collateral. Two stages, set side by side, reveal the design's range.

The first-draft task (74 criteria): using a precedent template, populate every bracketed placeholder and optional provision per the commercial term sheet and reference files, and produce a complete first version. The folder includes a CSA template (1995 English law), a commercial term sheet, an eligible-collateral and haircut schedule (Excel), the Master schedule, a ratings summary.

The criteria interrogate valuation percentages one by one—USD cash 100%, US Treasuries ≤1yr 99.5%, 1–5yr 98.0%, and so on—where numeric accuracy is itself the test.

The counterparty-paper-review task (62 criteria) shifts the question to "did you assign the right severity?" Did the memo flag, as Critical, that the Independent Amount deviates from the level the playbook requires? Did it assign Critical/High/Medium/Low to all 18 deviations? Did it catch the haircut figures that diverge from the playbook?

The ISDA Confirmation

A Confirmation memorializes a single trade. The counterparty-paper-review task (53 criteria) requires reaching back through the executed Master, the CSA and the underlying loan documents to surface the draft's deviations.

Enforceability issues are planted here. Can the memo flag that a Confirmation unilaterally amends the Schedule—and that, absent a formal amendment, a Confirmation cannot override the Schedule, raising an enforceability concern? Can it argue that a Swiss-law, Zurich-arbitration Confirmation conflicts internally with a New-York-law 2002 Master, creating a risk of clashing legal interpretation particularly in a close-out scenario? How a split in governing law bares its teeth at liquidation—the practitioner's pressure point.

Repo and securities lending

The richest is the repo escalation-memo task (69 criteria). The instruction: using the template, identify all policy deviations in a contract markup against the standard-terms policy, classify each by tier, and quantify incremental risk from the data sheet; present both the arithmetic sum and the diversification-adjusted total; address a concurrent credit-limit breach and request a temporary increase; and for the highest-tier deviation, include a General Counsel (GC—the head of legal) statement summarizing the netting opinion.

The folder holds a GMRA 2011 standard form, a standard-terms policy, a counterparty risk data sheet (Excel), an escalation request email, and a "netting-opinion summary memo"—a summary of the General Counsel's opinion letter.

Close-out netting: a mechanism that, when a counterparty defaults, offsets the claims and obligations across many trades down to a single net amount. If it is not legally valid, exposure can balloon many times over. / Netting opinion: a lawyer's formal opinion letter assuring that close-out netting will operate validly in the relevant jurisdiction.

Among the criteria sits what we take to be the heart of this piece. C-028: does the memo include a "GC statement" section—drawing on the netting-opinion summary memo from the folder noted above—summarizing that close-out netting is enforceable under English law? C-034: is GC sign-off on the top deviation conditioned on receipt of a confirmatory netting opinion? And—C-062: does the memo flag, as a residual risk, that the netting opinion does not extend to cross-border insolvency recognition scenarios?

C-062 is the protagonist of the chapters that follow.

3. What Year of Practice Is This?—Judging the Tasks Themselves

From here, the editorial assessment.

Plainly put: the LAB tasks are well made.

The material draws on real ISDA Masters, CSAs and definitions, and largely reflects practice. In the derivatives and repo world, this is the work assigned to a lawyer in their first through third year—reviewing a counterparty's paper, drafting a first version, surfacing deviations and writing the memo. The foundational reps an associate runs countless times under supervision.

And it is realistic. Almost no background is provided; you hand over a stack of documents and the task and say "deal with it." That is exactly the texture of a busy senior dropping work on a junior. The new lawyer must find the issues alone, armed only with the request email and the folder. LAB's choice to give direction without a playbook captures that reality well.

In other words, what it sets out to measure is on target. The tasks are strong. The next chapter steps into how those good tasks are scored—and there we find the reason not to take 13.3% at face value.

4. Everything Counts as One Point—So the Score Understates Practical Usefulness

Scoring treats everything as one equal point—while practice demands a hierarchy of severity

To read LAB's scores, there is a scoring mechanism worth holding in mind. In one line: catching a reservation in a netting opinion, and noticing that a table is missing a column, count as the same single point. This is less a defect than a structural property of the all-pass scheme—and because of it, the resulting score reads far lower than the work's practical usefulness.

What becomes equivalent inside the rubric

Reopen the repo escalation task (69 criteria) from Chapter 2. Within the same task, two criteria of utterly different character sit side by side.

One is C-062. Did the memo flag, as residual risk, the reservation in the netting opinion—"this opinion does not extend to cross-border insolvency recognition"? An issue that decides whether the trade lives or dies.

The other is C-061, in the same task. Are the required columns present in the deviation table? A pure formatting check.

Under all-pass, the two are exactly equivalent. One point each. Catch the former and drop the latter's column, and the total is zero. Get the formatting right and miss the reservation, and no points separate you. The ability to spot the enforceability of close-out netting, and the tidiness of a table, carry the same weight on the scoreboard.

The same shape recurs in the CSA review task (62 criteria). Flagging a $2.5M Independent Amount shortfall as Critical (the bedrock of eligible-collateral protection) and noting "the August 4, 2025 target execution date" (a scheduling detail) stand as one vote apiece.

It "knows" severity but never lets it touch the score

Here is the structure that shapes how the score should be read.

As Chapter 2 showed, severity labels appear throughout LAB's rubrics—Critical/High/Medium/Low, or Green/Amber/Red. At a glance it looks as though severity is being handled.

Read closely, and it is not. These labels are criteria asking "did you write the correct severity label into the deliverable," not weights on the score. When the CSA task's C-010 asks whether the Independent Amount was flagged as Critical, what is graded is whether the string "Critical" was written correctly. That Critical rating is not weighted more heavily than any other criterion in the math.

So LAB imports severity into the rubric's vocabulary while never using it in the score function. It knows severity, yet makes it neither a weight nor a gate. A material issue and a trivial format check are equally one vote. Holding this structure in mind is the first step to reading the score correctly.

Why the score understates practical usefulness

Consider the consequence.

All-pass rates sit at a few percent even for frontier models (7.1% for Opus 4.7). A practically capable model that caught every material issue still scores zero on a task if it drops a single trivial format check. An answer that, by intuitive and practical standards, is "good enough to use" is mercilessly sorted into zero under the all-pass ruler. That is why the resulting number reads far lower than the actual practical usefulness.

And one more thing. At this level, flat scoring cannot distinguish a model that caught the material issue (netting validity) from one that caught a trivial one (an execution date). Both go to zero the moment they miss any other single criterion. The very difference we most want to see—the capacity not to drop the material issue—is not resolved on the scoreboard.

13.3% does not, in itself, lie. But to read it, without checking what counts as one point, as "even the strongest AI is only 13% useful for legal work" is plainly wrong. This is a score to be received with reservations—one that discounts practical usefulness considerably.

5. Why a Netting Reservation Is an Absolute Failure—and Why That Calls for Reservations About the Score

Let us dig one layer deeper into the asymmetry from the previous chapter. It assumes finance-law background, so we lay it out carefully—why we call missing a netting-opinion reservation an absolute failure, independent of experience. This is also what underpins why 13.3% should be received with reservations.

A "clean," unreserved legal opinion on close-out netting is the foundation of credit and capital management in derivatives. To step into a reservation—say C-062's "does not extend to cross-border insolvency"—is to push the validity of netting outside that foundation. Two layers are in play.

One is enforceability in insolvency. If netting fails, claims and obligations gross up at liquidation, and real exposure to a defaulting counterparty surges. A net position of a few hundred million stands up, ungrossed, at billions.

The other is regulatory capital. Capital recognition of netting is permitted only where there is a supervisory "well-founded legal basis"—that is, a clean opinion. Step into a reservation, and netting drops out of the capital calculation and lands on risk-weighted assets. This is not a nicety of contract interpretation; it is a supervisory matter.

Because of these two layers, a lawyer who spots the reservation has no discretion. The moment it is seen, GC approval and escalation to the netting committee become a de facto duty—dependent on neither tenure nor individual judgment. That is why it is the failure you must never commit.

Here lies the asymmetry of practice. The "absolute no" issues—netting enforceability, breaches of margin or stay rules, errors in the instrument, quantity or amount of a trade—render a deliverable worthless if missed. By contrast, qualitative risk-allocation clauses, so long as they do not stray far from market convention, leave room to flag or not. The former is instant failure; the latter is discretion. That asymmetry is the reality of finance law.

LAB's flat all-pass levels that asymmetry. The design philosophy—include only what a supervising attorney actually checks, strip out nice-to-have padding—is itself sound. But because the scoring melts every included item down to one equal point, this is what happens: a practically capable model that catches every material issue scores zero if it drops a single trivial format check. An answer that practice would call "usable" is sorted into worthlessness on the scoreboard. This, precisely, is why the 13.3% score understates practical usefulness and should be read with reservations.

6. Toward Writing the Ruler—The Future of the Profession

To close, widen the lens. What this benchmark illuminates is the future of the profession itself.

Why legal work became LLM-friendly

A lawyer's core work is analysis, drafting and review, judged against the law and a given client's framework for legal risk appetite—plus gathering the relevant law, scholarship and precedent from a vast corpus and synthesizing it.

These are precisely the tasks LLM-based agents excel at. Lawyering was not always thought replaceable by AI. But once LLMs advanced, and once the work was articulated and broken into parts, its core turned out to be the machine's specialty.

It is also a domain where the upside of AI is legible—scarce senior talent, the cost of outside counsel. Little wonder legal AI has become one of the more crowded corners of the field.

Yet LAB measures only the "AI-ification of conventional work"

Pause here. What LAB measures is, precisely, how accurately conventional legal work can be automated. Drafting, reviewing, memo-writing—the precision of discrete tasks.

But as conventional work is automated, the nature of the work humans do shifts. Advising on the design of AI-friendly playbooks. Consulting on the judgment calls humans must still make on legal risk. The center of gravity moves there.

And the legal AI agent of the future will, under human supervision, run a larger loop: the AI researches, analyzes a new project or product, proposes improvements; the client then reworks structure, contracts and controls. Once the plan firms up, the AI moves to DD planning and execution, drafting, memoranda and opinions, and progress management—testing as it goes, all the way to completion, orchestrating a team of sub-agents. LAB does not design or evaluate that kind of higher-order agentic work. One day, building a project-specific legal AI agent may itself become a law-firm service. LAB, at least, points nowhere near that far.

So, write the ruler

Back to evaluation. How would we re-measure? Our proposals (not Harvey's view):

One: gate the scoring by severity. The issues you cannot miss—Critical criteria—become gates that zero the task if dropped. Discretionary items earn credit when caught and cause no mortal wound when missed. The all-pass spirit—no tolerance for a material miss—is preserved, while the asymmetry of weight finally reaches the score.

Two, more fundamental: the era of racing AIs to follow a playbook is quietly ending. What the law firm of the future is asked is whether it can write the playbook the AI consumes—designing, in machine-readable form, the procedure that never drops the material issue, and guaranteeing quality through adherence to it. From the side that receives the ruler, to the side that writes it. The advantage moves there.

Will the profession be replaced?

The final question.

Shallow expertise will struggle to add value. What will be wanted is hybrid command of both law and AI—the ability to support a client's AI maturation and the building of project-specific agents.

When entry-level work disappears, training the young demands a new design. The labor-intensive model—throwing armies of juniors at due diligence and billing by the hour—will be forced to transform.

But this is not only a story of loss. Through AI, the law shifts from the property of specialists toward the property of citizens—a genuine democratization. For society, that should be the welcome direction. The meaning of the monopoly the bar holds, the barrier to entry the profession enjoys, will be reexamined. And what remains is the value of the adviser in the more human sense.

13.3% does not lie. But once you check what it counts as a single point, what comes into view is not only the limit of a scoring scheme. Beyond it, you can see the quiet tectonic shift in the work of the profession itself.

References