Assurance
How well do AI models do ordinary business work, what does it cost, and how often do they get it wrong? We are building a public test program to answer that on real tasks, with a published method and cross-vendor grading.
Status: stage one, demonstration run
Stage one ran the five public tasks three times on each of four models. It is a demonstration of the method, not a finding: five tasks are too few to rank models, and the grading has not been checked against people. The held-back tasks run in stage two.
Why this exists
California enacted two laws in September 2026 that create independent verification organizations and a registry of AI auditors. Colorado's own automated decision law takes effect on January 1, 2027. Businesses are being asked to vouch for AI tools they have little independent evidence about. Useful, public test data can exist before any certificate does, and that is where we start.
Details on both states' laws are in the California and Colorado policy tracker entries.
What we test
Business tasks only. No trivia, puzzles or general knowledge. Ten tasks across five kinds of work, two each. One task per kind is public, and its answer key is kept on file and available on request. The other will stay private until the first report, and a fingerprint of it will be published before stage two runs, so it cannot be changed afterward.
Contract clause extraction
Example: Pull the payment terms, renewal, notice period, liability cap and governing law from a supplier agreement, citing the clause for each.
Graded by: Exact match on each field, and a check that every cited clause appears in the document.
Customer email drafting
Example: Answer a customer about a delayed shipment using the facts in the order record, promising nothing the policy does not allow.
Graded by: Required facts checked against the record; two blind graders mark acknowledgment, next step and tone as yes or no.
Meeting summaries
Example: Summarize a sales handoff meeting into decisions, owners and due dates.
Graded by: Each decision, owner and date checked against the answer key; a rejected proposal reported as a decision counts as a fabricated fact. From stage two, answer keys will be signed before any run.
Spreadsheet analysis
Example: From twelve months of sales, report net revenue, the top product and last month's change.
Graded by: Totals to the cent, percentages to a tenth of a point. From stage two, an invented value for a blank cell will count as a fabricated fact.
Small business coding
Example: Write a function that cleans an invoice file: duplicates, inconsistent dates, blank amounts.
Graded by: Hidden unit tests, run in an isolated environment with no network access.
What we measure
- Accuracy
- The share of runs that pass every required check, and the share of checks passed, so a nearly right answer is visible.
- Cost
- Actual billed tokens times the vendor's published price on the run date, per passing run. Stage two will add cost per task and will report human review time separately.
- Speed
- Wall-clock time from request to full response. Stage one shows the median; stage two will add the range.
- Consistency
- Every task runs three times per model with identical settings. Stage one shows how many of the three passed; stage two will publish all three scores and their spread.
- Failure modes
- Each failed run is labelled by what went wrong. Stage two will use the full set of labels: missed field, fabricated fact, wrong number, stale clause, format failure, failed test, refusal, timeout or truncation.
How cross-vendor grading works
- No model grades its own vendor's output.
- Wherever an answer can be checked mechanically, it is: exact matches, numeric tolerances, source checks and unit tests. These decide whether a run succeeded.
- Items a check cannot judge, such as tone or whether a summary invents a decision, go to an ensemble of two AI graders from the other vendor: OpenAI models grade Claude's work and Claude models grade OpenAI's. Graders do not see which model produced the output. An item passes only if both graders pass it, and their agreement rate is published.
- Answer keys, tests and rubrics are frozen before any model runs. A corrected key gets a new version, every model is rescored against it, and the earlier scores stay on record.
What we will publish
In stage two we will publish the test harness, scoring scripts, public tasks and answer keys, prompts and settings, the exact model identifiers the vendor returned, run dates, raw scores, token counts, cost, speed and failure labels. Held-back tasks will be released after the first report if that does not compromise a repeat test. Until then, these files are kept on file and available on request.
Models in stage one:
- Anthropic: claude-sonnet-5, claude-opus-5-5
- OpenAI: gpt-6-sol, gpt-6-astra
The model name each vendor reports back is recorded with every run. Codex does not report one, so OpenAI runs record the model that was requested.
Stage one demonstration results
Five public tasks, three runs each, run on 2026-09-24. A run succeeds only if every required check passes. Read these as a test of the method: with five tasks, one task can move a model's total by a fifth.
| Model | Runs that passed | Checks passed | Median time | Cost per passing run | Graded items passed |
|---|---|---|---|---|---|
| claude-opus-5-5 | 15 of 15 | 100% | 4.2 s | $0.0157 | 21 of 21 |
| claude-sonnet-5 | 15 of 15 | 100% | 3.6 s | $0.0063 | 18 of 21 |
| gpt-6-astra | 15 of 15 | 100% | 13.5 s | not comparable | 18 of 21 |
| gpt-6-sol | 15 of 15 | 100% | 10.7 s | not comparable | 18 of 21 |
OpenAI times include Codex start-up and OpenAI costs are not comparable; see the disclosures below. Graders agreed on 21 of 21 items for claude-opus-5-5, 19 of 21 items for claude-sonnet-5, 20 of 21 items for gpt-6-astra, 20 of 21 items for gpt-6-sol.
Every model passed every required check on every run. These five public tasks are too easy to separate the models, and stage two's held-back tasks need to be harder. Graded items and times cannot be compared across vendors in this run, because each vendor was graded by a different pair of models, the graders changed some verdicts when re-grading unchanged outputs, and OpenAI times include Codex start-up. Within a vendor they can be compared, because both models shared a grader pair: claude-sonnet-5 against claude-opus-5-5, and gpt-6-sol against gpt-6-astra.
Grading corrections
The first scoring failed 6 of 60 runs. On inspection those failures were faults in our grading, not in the models: a check that flagged the correct phrase “not a guaranteed delivery date” as a promise, and a meeting task whose instructions and format asked for different wording. Separately, one Claude-side grader was weaker than the rest and failed correct work its partner passed, so it was replaced. All of this was fixed, every model was rescored against the same corrected keys, and the first version's scores are kept on file and available on request.
The grader swap was made after the results were seen. The inspection that found these faults was done by the builder, a Claude model, not by a person.
| Model | Graded items passed, first | Graded items passed, rescored | Agreement, first | Agreement, rescored |
|---|---|---|---|---|
| claude-opus-5-5 | 21 of 21 | 21 of 21 | 21 of 21 items | 21 of 21 items |
| claude-sonnet-5 | 18 of 21 | 18 of 21 | 19 of 21 items | 19 of 21 items |
| gpt-6-astra | 16 of 21 | 18 of 21 | 16 of 21 items | 20 of 21 items |
| gpt-6-sol | 15 of 21 | 18 of 21 | 17 of 21 items | 20 of 21 items |
Runs that passed, by task
| Task | claude-opus-5-5 | claude-sonnet-5 | gpt-6-astra | gpt-6-sol |
|---|---|---|---|---|
| contract extraction | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| email drafting | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| meeting summary | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| spreadsheet analysis | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
| business coding | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 |
How runs failed
- claude-opus-5-5: no failed runs
- claude-sonnet-5: no failed runs
- gpt-6-astra: no failed runs
- gpt-6-sol: no failed runs
Where this fits
Others already test AI on professional work. Vals.ai publishes scores, cost and latency on legal and finance tasks. Scale's READY measures enterprise workflows, including the cost of human review. OpenAI's GDPval uses blind experts to grade professional work. Artificial Analysis sells custom task testing. We learn from all of them.
What we add is a public-interest program, independent of any model vendor, that tests everyday small-business work and reports how models fail as well as how often they succeed.
Disclosures
- John Francis, a co-founder of the institute, personally uses products from both vendors being tested.
- Claude models run through Anthropic's paid API. OpenAI models run through OpenAI's Codex tool on a ChatGPT plan, because no OpenAI API key was available. Codex wraps each request in its own agent instructions, so OpenAI times include tool start-up and OpenAI costs are not comparable and are not shown.
- The graders are among the models being tested, so a grader may favour its own vendor's style.
- The tasks were written by an OpenAI model and checked by a Claude model. Either could favour its own vendor's style; the mechanical checks are there to limit that.
- Any funding, credits or early access from either vendor will be listed here before results are published.