Introducing MAGI Benchmark
Chasing 30%
of a Company’s Soul
ANATOMY OF A SOUL
Chakra Labs · 2026
MAGI Benchmark:
MCP and GUI Interaction
While frontier models have accelerated tremendously over the last 12 months, they still fall short of leveraging all tools at their disposal. These tools include not just MCP servers, APIs, or bash terminals, but human graphical user interfaces (GUIs) which enable models to finish the "last mile" of a particular task graduating agents from completing sub-routines to performing full jobs.
Chakra Labs has built a new benchmark to measure this capability directly: MAGI (MCP and GUI Benchmark).
MAGI consists of 1,000 (200 test / 800 train) tasks mimicking real work spread across 8 cloned enterprise applications. The large majority of tasks require both GUI interaction and MCP tool-use interleaved for successful completion, span between 2 and 7 applications , and start with a prompt that requests an outcome without a prescribed process.
Importantly, these tasks test five properties of real company work simultaneously:
Ambiguous: prompts specify required outcomes rather than a step-by-step process, resolved from the company's shared organizational state.
Long-horizon: trajectories run long before they resolve, a median of 87 steps, some past two hundred.
Tool-dense: tasks require sustained MCP coordination, five or more server calls within a single trajectory.
Cross-app: tasks span multiple applications at once, a mean of 4.38, up to seven.
Mixed modality: every application exposes GUI and/or MCP surfaces backed by identical state, so completing a task requires crossing between a visual interface and a structured API call within a single trajectory.
The result: frontier models complete fewer than 30% of MAGI tasks at pass@1, leaving substantial headroom before agents perform at the same level as a human in the workplace.
Engineering the soul of a company
Prompt ambiguity in MAGI is supported by rich context seeded into the application surfaces. Rather than over-specification, we opt for prompts that are simple but require the agent to explore the world to derive its ground truth. Every application is pre-seeded with a shared organizational state for a fictional startup, Terrace Labs, a seven-person developer-infrastructure company built specifically for MAGI. We call this a "Company Soul": colleagues, teams, initiatives, tickets, dashboards, and pending decisions, seeded consistently across all eight applications. The same colleagues appear in Slack and Linear; the same project names appear in Notion and Grafana.
An agent that searches carefully can resolve any prompt ambiguity; one that hallucinates context fails the checks deterministically. Most benchmarks trade ambiguity against verifiability: a prompt specific enough to check isn't truly ambiguous, and one ambiguous enough to reflect real work can't be checked.
Fig 1
Task authoring process
Tasks in MAGI are written by expert operators who spend two to three days immersed in the full Company Soul environment before writing a single prompt. They write delegation-style prompts alongside rubrics defining what must be true at completion, which artifacts must exist, which cross-app references must hold, what constraints cannot break.
We compile these rubrics into deterministic evaluation logic through a system we call Frankenstein where every check is converted into an explicit programmatic constraint before any agent run begins. Every task then goes through human QC: expert operators execute it end-to-end, produce golden traces, and confirm the verifier matches the original success criteria. No task ships until a human has completed it and the verifier agrees. From there, a task enters one of two pools: the training set stays available for lab use, while the test set stays closed, held back from publication so it can't enter a future pretraining scrape and lose its power to measure real capability.
Fig 2
Every model is run through one adapted OpenHands harness built on the Harbor framework, held constant across all models so the comparison isolates the model. This does not assess native computer use; native-harness evaluation is future work.
Scoring runs on two metrics, computed independently. Pass@1 is task-level binary: a task counts as solved only if every one of its checks passes. Each task also carries roughly a dozen to twenty binary sub-checks, fourteen on average, each tagged by type:
[STATE], the right artifact exists with the right fields
[CONTENT], the artifact's text is on-topic
[PROSE/GUI], a specific action was performed with no MCP equivalent.
Mean reward is the weighted fraction of sub-checks satisfied. Every check is deterministic and reproducible.
All main results in this report are evaluated at default reasoning settings, unless otherwise noted.
Not even halfway
A client account is up for renewal next month, and the account contact hasn't replied to a recent email. The agent is asked to dig through Linear for everything tied to the account across two teams, then deliver five outcomes with no order or process specified:
write a renewal brief in Notion synthesizing those findings plus service metrics
create a saved Linear view filtered to the account
message a stakeholder on LinkedIn about platform reliability
email the client contact with a status-and-timeline update
schedule a strategy meeting on the calendar
This is the Company Soul at work: the account's history isn't summarized anywhere, it's scattered across the same shared organizational state described above, and the agent has to reconstruct it and decide what belongs where, the same way a new hire would.
Opus 4.8 and GPT-5.5 score similarly on the task, receiving rewards of 0.33 and 0.364 respectively, but the deterministic verifier's sub-checks, eleven in total, show where the two models actually diverge. Tagged by type:
Fig 3
Both models stall on the same deliverable: neither produces the Notion renewal brief, the one artifact that requires synthesizing what Linear actually says about Greenline's account health into new, correctly labeled content, rather than confirming an action the prompt already specified. Both also fail the Linear saved view, though differently: Opus 4.8 never creates one, GPT-5.5 creates it but doesn't name or filter it correctly. From there the two models diverge completely on the remaining outreach steps.
No single failure mode explains this task on its own; the same categories that structure the results below show up here in miniature.
Losing the thread
In Q2, OpenAI and Anthropic each shipped new flagship models with increased capability on computer use. GPT-5.5 reports 78.7% on OSWorld-Verified. Anthropic's Opus 4.8 reports 83.4%, and its newer Fable 5 model pushes that to 85.0%, the current high mark on the benchmark.
Those scores measure isolated computer-use tasks. GPT-5.5 and Opus 4.8 both drop sharply on the work MAGI measures instead, ambiguous, long-horizon, tool-dense, cross-app, and mixed-modality all at once. Kimi K2.7-code sits alongside them, the weakest of the three.
All models run through the same harness, a single adapted OpenHands configuration, so the MAGI leaderboard below compares them under identical conditions. This is a limitation of the current benchmark, which will be addressed on future iterations to run models in their native harnesses.
Fig 4
Every model can open Slack, create a ticket, draft an email without issue. What breaks is carrying context across systems, across long time horizons, and across modality switches, the ambient complexity a human operator navigates without thinking.
Each model clears most of a task's individual checks but rarely all of them. Pass@1's low ceiling comes from the difference between partial and full credit, not from run-to-run variance.
Plotted against cost per task on default effort levels, a different pattern emerges.
Fig 5
GPT-5.5 and Opus 4.8 sit almost on top of each other, comparable reward at comparable price, while Kimi trades roughly a third of the reward for about a seventh of the cost. Neither the frontier pair nor Kimi is simply worse on both axes; they're different points on the same cost-reward frontier, not a strict hierarchy.
Five ways to fail
GPT-5.5, Opus 4.8, and Kimi K2.7 post pass rates 13 points apart and stall on almost entirely different tasks, only 21 of 200 test tasks are solved by both frontier models, but when a trajectory fails, it fails in nearly the same proportions no matter which model is running it.
If failure mode were mostly a model-specific quirk, the model with the worst pass rate should also have the most distinctive failure signature. Kimi doesn't. Its GUI Interaction and Termination shares sit within a few points of the other two models' (Figure 6), the same two categories dominate, in nearly the same order, despite a pass rate roughly half of Opus's. That points toward the five failure modes being closer to properties of the task distribution and evaluation harness than idiosyncrasies of any one model, which points any fix upstream rather than at a single model.
Fig 6
GUI Interaction Error: the most common failure mode across every model tested, averaging 51 percent of failed trajectories (52.0% GPT-5.5, 55.5% Opus 4.8, 46.7% Kimi). Misclicks, navigation loops, wrong element selection: failure in the physical act of using the interface. This is the category most directly downstream of visual grounding, the model's ability to map what it sees to the coordinates and elements it needs to act on.
Termination Error: the second most common, and the most revealing, averaging 40 percent of failed trajectories (40.0% GPT-5.5, 37.2% Opus 4.8, 42.6% Kimi). The agent declares success before satisfying every check, a calibration failure: the model doesn't know it hasn't finished the remaining steps.
State Management Error: context dropped between applications, IDs mismatched across systems, averaging 6.5 percent of failed trajectories (7.3% GPT-5.5, 5.8% Opus 4.8, 6.5% Kimi).
Ambiguity / Judgment Error: misread intent, wrong priority, tone that doesn't match organizational context. Under 1.5 percent of failed trajectories in every model (0.7% GPT-5.5, 1.5% Opus 4.8, 0.6% Kimi), the smallest category by a wide margin, despite ambiguity being one of the benchmark's five design properties.
MCP Tool Error: wrong tool, invalid parameters, hallucinated tool names, bad sequencing. Close to zero in this benchmark, 0.0% for GPT-5.5 and Opus 4.8, 3.6% for Kimi. Most tool-call breakdowns surface downstream as a termination instead, once a call fails enough times, the model gives up on the step rather than retrying with corrected arguments.
Fig 7
Every failure mode above costs the same to run. A trajectory that misclicks its way to a GUI Interaction Error and one that loses context across three apps both burn through the same $14 to $15 in tokens before the verifier ever runs.
To each their own failures
All three models miss the same broad categories of requirement: an artifact with the right structured fields, on-topic content inside it, a specific UI-only action taken. What differs is which one dominates, and why.
Opus 4.8 is the most careful and the slowest. GUI Interaction is its largest failure mode at 55.5% (Figure 6), the highest of any model. 47% of its failures are near-misses, scoring 0.7 or higher but dropping one or two final checks, split between imprecise Notion and email content and GUI-only confirmations like pinning a message or subscribing to a thread. That diligence is also what makes it slow: longest wall-clock time by a wide margin, despite not taking the most steps, with a small share of trials still productive at the time-limit cutoff.
GPT-5.5 fails for a related reason with a different cause. GUI Interaction still leads at 52.0%, but Termination follows close behind at 40.0% (Figure 6): it builds the right artifacts almost every time, then closes the task early once the primary deliverables exist, underweighting what's still outstanding. 41% of its failures are near-misses. The same GUI-only gap appears here too, plus occasional unproductive loops re-querying the same tool without new progress.
Kimi K2.7 shares the same broad failure shape, GUI Interaction and Termination dominate here too, at 46.7% and 42.6% (Figure 6). What's specific to Kimi sits underneath the termination number: malformed tool-call arguments the parser rejects outright, which it typically abandons after enough rejections rather than reformatting and retrying. The taxonomy records that as Termination, not an MCP Tool error, since the JSON error itself is rarely the final blocking event.
The step-count distribution below shows this pattern directly: Kimi carries a distinct low-step bucket that neither frontier model has, consistent with walking away from a task early rather than working through it.
Fig 8
The common thread: every model's weakest point is the step with no MCP equivalent which a human would do reflexively: the pin, the subscribe, the GUI-only confirmation.
What they’re spending time (and tokens) on
A natural hypothesis is that these models fail on different tasks because they allocate effort differently - that Opus times out from over-navigation, while GPT-5.5 under-delivers from insufficient understanding. The data do not support this. Understanding and Navigation are sufficiently similar across all three models to rule out a simple effort-allocation story: Understanding accounts for 18.2–26.2% of trajectory, Navigation 46.9–54.3% (Figure 6). Execution is the real outlier, ranging from 11.6% (Opus) to 24.5% (GPT-5.5), and it runs the wrong direction for the effort-allocation hypothesis: Opus spends the least on Execution and has the highest pass rate, GPT-5.5 spends the most and sits in the middle. Opus is also the only model that splits its remaining trajectory almost evenly between Execution and Final Polish (11.6% vs. 11.0%), rather than clearly favoring one over the other.
The gap between a 27.5% and a 14.6% pass rate comes from what happens within each phase rather than how the trajectory is allocated across phases. The performance gain accrues from training on the points at which models actually break, rather than from reallocating effort across phases.
Fig 9
These differences also show up in how each model sequences its actions. The heaviest traffic in the matrix below sits inside the GUI actions themselves: Navigate loops back into Navigate or Inspect far more often than it moves out to an MCP call, the same searching-without-finding pattern that shows up as GUI Interaction failures above.
Fig 10
Each cell shows the probability of the next action given the current one, row-normalized so each row sums to 100%. Computed over tool-call actions, not agent turns, so figures here aren't directly comparable to the turns-based step counts elsewhere in this report.
The coordination cliff
One might expect each additional required app to shave off a similar amount of reward, but with MAGI we find this isn't the case. Across all models, cost increases sharply once a task crosses six apps. Reward holds in the .70x range from two apps through four, dips modestly to .674 at five, then falls sharply to .472 at six and 38% at seven (Figure [11]).
A likely source is that app count is a proxy for trajectory length. More apps mean longer trajectories, which consume more of the context window and accumulate more instructions the model must continue to satisfy. Both pressures, context length and instruction-following, degrade gradually rather than at a fixed point, which is consistent with a convex decline rather than a sharp threshold. We don't read the drop-off as evidence that five apps is a hard limit, only as the point where these accumulating costs become large enough to dominate.
Fig 11
Table scoped to tasks spanning 2–7 apps. 1-app tasks are excluded, coordination cost isn't meaningful below two applications. n trials per bucket ranges from 30 (2 apps) to 173 (4 apps); see table for exact counts by app-span.
The same shape shows up in each model individually, not just in aggregate: reward stays roughly flat through four apps for all three, then drops by five or six. That consistency is what makes this a property of the benchmark's difficulty curve rather than one model's specific weakness.
Fig 12
The decline reflects a diminishing willingness to proceed rather than a loss of capability as app-count grows: when no MCP tool exists for the next step, models routinely treat it as impossible rather than falling back to the GUI. On v04-release-coordination, a seven-app task, Opus 4.8 halted after 13 steps, well below the seven-app average of 65, concluding: "I'll wrap up by summarizing what I've accomplished and noting the limitations, LinkedIn, email, and Google Calendar aren't available through my tools." All three were in fact reachable through the GUI.
Step-count variability also rises toward the upper end of the app-span range. Opus's standard deviation climbs from 32 steps at two apps to 63 at six (Figure 12), the highest in the dataset, before easing to 60 at seven; Kimi follows a similar path, reaching 62 at six. GPT-5.5 is the exception, holding a narrow 24-to-32 band across the entire range. All three models record their lowest reward at seven apps, one step past where Opus and Kimi's variability peaks, consistent with instability building before performance fully collapses.
We frame this as willingness rather than capability, which invites an obvious test: raising the reasoning-effort budget should reduce early termination if the models simply aren't trying hard enough. Preliminary runs against higher-effort configurations show no such reduction, evidence against the simplest "not trying hard enough" reading. Full results are reported separately; this preliminary finding is noted here only to bound the interpretation above.
All
/
Opus 4.8
/
GPT-5.5
/
Kimi K2.7
Fig 13
Opus's six-app column shows the split directly: trajectories bottom out near zero and others run past two hundred, the same spread the standard-deviation jump describes.
The cost of 30%
In 2026, cost-per-task has become nearly as closely scrutinized as accuracy in agentic benchmark reporting: evaluation suites that once reported only pass rate now routinely publish token expenditure alongside it. For MAGI, we evaluated all three models against a human baseline, priced using Upwork rates, in order to situate cost relative to performance.
Fig 14
Cost per task and cost per solved task diverge sharply once completion rate enters the picture. Human cost per solved task at $15.84 undercuts every model except Kimi K2.7-code. Opus 4.8 and GPT-5.5 both cost three to four times more per completed task than a person since most of what they run doesn't finish. Kimi K2.7-code is the exception only because it's cheap enough to absorb a low pass rate where failed attempts are sufficiently inexpensive that the cost doesn’t compound the way it does for Opus and GPT-5.5.
The runs themselves are what's expensive right now. A model that failed less often would flip this table without changing a single API price: every attempt costs the same $14 to $15 whether it finishes or not, so completion rate is the lever that moves cost per solved task.
The road next travelled
MAGI holds three things fixed that real organizations do not: the agent meets every task cold, the environment stays still while it works, and colleagues exist only as the state they left behind. Relaxing each is where the next version is headed.
Memory. Real operators carry the company with them between tasks, they know who owns what without rereading every app. We are building memory tools seeded with organizational context to measure whether an agent accumulates and reuses a working model of the company across a sequence of tasks.
Interaction. Work often gets resolved by talking to someone. We are adding multi-turn tasks where the agent coordinates with colleagues simulated by language models, so resolving ambiguity means asking the right person and reading an imperfect answer. Scale AI's SWE-INTERACT demonstrates persona-conditioned user simulators on the same Harbor framework and the same three model families in the software-engineering domain; we extend that approach to the surface MAGI measures.
Change. Environments move while you work in them: a ticket gets reassigned, a meeting shifts, a document is edited mid-task. We are introducing perturbations partway through a trajectory to measure whether an agent notices the ground has moved and replans.
If you want to run your model on MAGI, let us know.
Access MAGI
All benchmark results reported here are measured. Every model is evaluated through a single adapted OpenHands harness, built on the Harbor framework, held constant across all models, so scores reflect the model being tested rather than a native agent configuration.
If you use the MAGI Benchmark in your research, please cite:
@misc{chakra2026magibenchmark,
title = {MAGI: MCP and GUI Interaction},
author = {Chakra Labs},
year = {2026},
howpublished = {\url{https://chakra.dev/magi}},
note = {arXiv preprint forthcoming}
}