We recently partnered with SemiAnalysis on their GDPx benchmark as the exclusive provider of legal tasks. The following is an excerpt from the full release detailing our legal expert base, task construction, judge validation, and a public task with its failure analysis.

About Chakra

Chakra is a research and design lab that builds task factories for useful intelligence. Although they make tasks for other domains as well, Chakra’s legal expert base includes corporate partners at Wachtell and other large US law firms, general counsels at startups, commercial counsels at Fortune 500 companies, and heads of legal operations at asset managers. The GDPx tasks were developed using this roster.

For GDPx, Chakra contributed tasks across eight practice areas: M&A, venture capital, capital markets, corporate governance, intellectual property, data privacy, administrative law, and dispute resolution. Across those areas, six work types recur: legal research, drafting, redlining, term extraction, issue spotting, and risk assessment.

Making tasks for GDPx

Authors begin with assignments drawn from their own practice and ask what the client or supervising partner needs to accomplish. The agent should understand its role, objective, and expected deliverable from the prompt. The complexity lies in the environment: interacting provisions, competing priorities, and facts requiring careful interpretation.

The environment is a set of Word and PDF files containing agreements, term sheets, offering documents, court filings, and client or partner emails, mixing full-length documents with excerpts. The files are generated, drawing on public sources such as SEC EDGAR filings when helpful. Experts also write the rubric the verifier grades against, and a reference solution showing one valid way to complete the task.

Validating the judge

To QC rubrics, Chakra measures judge error against expert labels before relying on its scores. The validation set includes known-good, deliberately flawed, and partially correct documents, labeled criterion by criterion, which gives false-positive and false-negative rates. Experts review close cases: whether equivalent wording receives credit, whether incomplete reasoning passes, and whether one criterion influences another.

Each output is graded five times and criterion-level verdicts are tracked, since a stable total score can hide inconsistent decisions. Judge-model ablations hold the rubric and outputs fixed while changing the grading model, which exposes differences in strictness and self-preference bias.

Public task

quillon-series-b-redline asks the agent to redline the draft amended and restated certificate of incorporation in a Series B financing on behalf of the lead investor, framed as a partner would assign it.

In 20 rollouts across four models, no redline was usable, for a binary score of 0% across the board. The mean partial score was just 18%, and two of the four models scored near zero because penalties for re-trading agreed terms erased their credit. Opus 5.5 averaged a partial score of 44%, GPT-6 Astra 25%, Gemini 4 Argon 2%, and GLM 5.3 0.5%. The highest-scoring run reached 57% (Opus 5.5).

The failed criteria trace to three causes: failure to synthesize different input documents, failure to anticipate how the counterparty could work around the client’s desired protections, and treating the client’s email as overriding the signed term sheet. 13 of 20 runs failed to notice from the capitalization table that the company’s other shareholders have enough voting power to strip away the client’s protections. 20 of 20 runs failed to protect against the company using its subsidiary to bypass the protective provisions of the charter. 14 of 20 runs applied the client’s emailed provisions without flagging that they re-trade already-agreed terms.

Written by Chakra, with editorial support from SemiAnalysis.

—

To read more about the benchmark, visit the GDPx website.

To inquire about frontier legal intelligence research, please contact info@chakra-labs.com.