AGI Score Methodology

Read directly from the evaluator. A result counts only as the evaluator publishes it, for this exact model, on this exact version of the test. LiveBench publishes its two figures as components, so we apply LiveBench’s own formula to them (the global average is the mean of its seven category means; Agentic Coding is the mean of its JavaScript, TypeScript and Python tasks), and only when every component is published.

Never counted: a value converted from another evaluator, adjusted, estimated, or copied from another model or another test; a figure relayed by a third-party site; and a lab’s own report of its model. Lab announcements tell us where to look, never what to score.

One setting per model. Effort settings are not separate models. For each evaluator we take the setting the provider names as the model’s own, or else the highest effort setting the evaluator lists, never the one that scores best. If that setting has no result, the test is missing: a lower setting never stands in. A row that does not say which setting it ran does not count.

The model alone answered. Some providers offer a refusal fallback: when the model declines a request, another of the provider’s models can answer in its place. A result counts for the model alone when the evaluator states that the fallback was off (for that result, or in a standing statement of how it runs every model), when the evaluator marks every run that used it and this run carries no mark, when it shows the fallback never activated, or when it publishes a recount that scores every task the fallback helped with as a failure.

With refusal fallback. When some of a model’s results come from runs with its provider’s fallback on, or from runs that do not say, and nothing else sets them aside, the model is scored as “model with refusal fallback” on its results as published. That label goes wherever the model is named, and each of its results says how it was served. It comes off in the release after every one of the six tests has a result from the model alone. A fallback to another company’s model never counts.

Still published. A result counts while the evaluator still publishes it. When an evaluator revises a value, the revision enters in an evidence release, never silently.

What “at least” means. At least X is the AGI Score the model would have even with 0% on every missing test. It is a fact about the evidence, not an estimate of the model. It is never called an AGI Score and never carries a rank. The one comparison it supports: a model at “at least X” is certainly ahead of every ranked model listed below it. It says nothing about the models above it. We publish no ceiling, range or “could reach” figure, because each of those would assume a value for a test nobody has published. With three or more tests missing, a minimum says too little to place a model honestly, so it shows as early results.

Ranks count fully measured models only. A rank is a position among the models measured on all six tests.

Why it works this way. Our earlier method scored a model with a gap on the tests it had. That is the same as filling each gap with the model’s own average elsewhere, so a model could rise by being measured less. Now a missing result can only hold a model back, and each missing test is named with the evaluator that would complete it, for example “Terminal-Bench 4.0: not yet tested by Vals”.

Holds. A model is held only by a recorded ruling, with its reason, who ruled and when. A held model shows the reason and no figure.

Agency, Reasoning, Knowledge and Visual each show one part of the AGI Score on its own, from that part’s tests, under the same rules: a rank needs every test in the part, and “at least” counts a missing test as 0%. They are not the AGI Score.

Custom weights, set with the presets and sliders in the Explorer on the main page, let you choose the five part weights yourself; the leaderboard then shows them under a Custom tab. They use the same results and the same rule for missing tests, so nothing is estimated, but they are yours alone: a custom board is shown on 0 to 10, it is not the AGI Score, and it changes no official rank.

Agents (planned for v3.1) will rank models on AutomationBench, from Zapier, and Terminal-Bench Science, from Vals, at equal weight, with a rank only when both are measured. The Agents tab is outside the AGI Score and never changes it.

No derived values. Before every release, a check rebuilds the board from the published data and refuses to publish if any figure rests on a converted, estimated or filled value, if a score is missing a test, or if an “at least” figure carries a rank. Every changelog entry states the count it found: “Derived values scored: 0”.

Recomputed in your browser. The page recomputes every figure from the published board data before showing it. If it cannot reproduce them exactly, it shows no figures at all.

Coverage review. Ninety days after v3.0, and every ninety days after that, we check each test against the models released at least sixty days earlier. Any test missing for more than a third of them gets a written review within thirty days. Keeping the test is a valid outcome; replacing it takes a signed methodology change and a calibration release, never an automatic switch.