Assurance

Customer service

An agent handling a customer's request with business tools: changing a booking, fixing an account, answering a billing question. Good means the request is done correctly. Sierra's tests also report how often a model succeeds on every one of several tries.

Sources last checked 2026-10-23.

Key evidence rings earned, out of 4Agent works through the task with tools, as the words after it sayFilled: best score in that test. Dashed: tested, figures not shown here.

At a glance

independent tester: An independent group tested this, earned scores shown here, from the last year: We show their numbers, published in the last year, earned two or more testers: Two or more groups tested it, not yet repeat runs, some or all models: Results checked by repeat runs, for some or all models, not yet

Vendor claims have not been collected for any category yet.

T2R 81.6%T2R 81.6%T2A 84.0%T2A 84.0%T2T 92.3%T2T 92.3%T3B 48.7%T3B 48.7%EOG ↗GAL ↗
ⓘ More about this category
  • Reliability figures shown
  • 6 sources, 15 models with figures
  • Sources last checked 2026-10-28
  • Ring 4: needs rings 1 to 3 first
  • T2R = tau2-bench retail, T2A = tau2-bench airline, T2T = tau2-bench telecom, T3B = tau3-bench banking, EOG = EnterpriseOps-Gym-AA, GAL = Agent Leaderboard v2

Who does well

TestBestBest scoreNextNext score
tau2-bench retailGPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high81.6%Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high79.6%
tau2-bench airlineClaude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort highroughly tied84.0%GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high83.0%
tau2-bench telecomClaude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high92.3%Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high91.2%
tau3-bench bankingClaude Opus 5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max48.7%GPT-5.6 Sol Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh46.9%
Place within each test. Places are never added up across tests.
Modeltau2-bench retailtau2-bench airlinetau2-bench telecomtau3-bench banking
Claude Opus 4.52 of 5Agent with the domain's tools, talking to an AI-played customer, effort high1 of 5Agent with the domain's tools, talking to an AI-played customer, effort high1 of 5Agent with the domain's tools, talking to an AI-played customer, effort high13 of 15Agent with Sierra's standard search tools, talking to an AI-played customer, effort high
Claude Sonnet 4.55 of 5Agent with the domain's tools, talking to an AI-played customer, extended thinking on5 of 5Agent with the domain's tools, talking to an AI-played customer, extended thinking on5 of 5Agent with the domain's tools, talking to an AI-played customer, extended thinking on12 of 15Agent searching with a terminal only, talking to an AI-played customer, extended thinking on
Gemini 3 Flash (preview)3 of 5Agent with the domain's tools, talking to an AI-played customer, effort high3 of 5Agent with the domain's tools, talking to an AI-played customer, effort high2 of 5Agent with the domain's tools, talking to an AI-played customer, effort high9 of 15Agent searching with a terminal only, talking to an AI-played customer, effort high
Gemini 3 Pro (preview)4 of 5Agent with the domain's tools, talking to an AI-played customer, effort high4 of 5Agent with the domain's tools, talking to an AI-played customer, effort high3 of 5Agent with the domain's tools, talking to an AI-played customer, effort high14 of 15Agent searching with a terminal only, talking to an AI-played customer, effort high
GPT-5.21 of 5Agent with the domain's tools, talking to an AI-played customer, effort high2 of 5Agent with the domain's tools, talking to an AI-played customer, effort high4 of 5Agent with the domain's tools, talking to an AI-played customer, effort high8 of 15Agent with Sierra's standard search tools, talking to an AI-played customer, effort high

The tests here put 19 pairs of models in a different order.

Show all 19 pairs placed in a different order
  • In tau2-bench retail, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench airline, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench retail, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, Gemini 3 Flash (preview), agent searching with a terminal only, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench airline, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, Gemini 3 Flash (preview), agent searching with a terminal only, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, Gemini 3 Flash (preview), agent searching with a terminal only, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench retail, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench airline, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench retail, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench telecom, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench airline, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, GPT-5.2, agent with Sierra's standard search tools, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, GPT-5.2, agent with Sierra's standard search tools, talking to an AI-played customer, effort high, placed above Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high.
  • In tau2-bench retail, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Gemini 3 Pro (preview), agent searching with a terminal only, talking to an AI-played customer, effort high.
  • In tau2-bench airline, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Gemini 3 Pro (preview), agent searching with a terminal only, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on; in tau3-bench banking, Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on, placed above Gemini 3 Pro (preview), agent searching with a terminal only, talking to an AI-played customer, effort high.
  • In tau2-bench retail, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench telecom, Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench airline, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench telecom, Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, GPT-5.2, agent with Sierra's standard search tools, talking to an AI-played customer, effort high, placed above Gemini 3 Flash (preview), agent searching with a terminal only, talking to an AI-played customer, effort high.
  • In tau2-bench retail, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench telecom, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench airline, GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high, placed above Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high; in tau2-bench telecom, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high.
  • In tau2-bench telecom, Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high, placed above GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high; in tau3-bench banking, GPT-5.2, agent with Sierra's standard search tools, talking to an AI-played customer, effort high, placed above Gemini 3 Pro (preview), agent searching with a terminal only, talking to an AI-played customer, effort high.

Findings for this category aren't written yet.

The tests behind this

tau2-bench retail

best 81.6%Sierra115 tasksNewest result 2026-03-02reliability figures showncaveat

tau2-bench, built by Sierra, puts an AI agent in a customer-service job: it talks with a customer, played by another AI model, and must follow the business's policy while using the business's tools to finish the task. This source is the retail domain: orders, returns and exchanges.

Results

Each model's best setting in this test

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2681.6%

    Completes it 4 tries out of 4: 51.8%

  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2679.6%

    Completes it 4 tries out of 4: 51.8%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0276.8%

    Completes it 4 tries out of 4: 51.8%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0275.9%

    Completes it 4 tries out of 4: 47.4%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2672.4%

    Completes it 4 tries out of 4: 39.5%

Show all 6 results (1 not shown above) from tau2-bench retail
  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2681.6%

    Completes it 4 tries out of 4: 51.8%

  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2679.6%

    Completes it 4 tries out of 4: 51.8%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0276.8%

    Completes it 4 tries out of 4: 51.8%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0275.9%

    Completes it 4 tries out of 4: 47.4%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort none, Run date not published; posted 2026-02-2675.0%

    Completes it 4 tries out of 4: 45.6%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2672.4%

    Completes it 4 tries out of 4: 39.5%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra does not publish when it ran each model; the date shown is when the results were posted. The newest models on Sierra's site were run only on a separate banking domain, shown as its own test on this page. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability: Sierra runs each task more than once. The reliability figure is pass^4: the chance the model completes the same task on 4 tries out of 4. A big gap between the two numbers means the model is inconsistent.

Sierra built tau-bench and runs every result shown here. Sierra sells AI customer-service agents for retail, airline and telecom, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 10 models

Claude Fable 5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Gemini 2.5 Pro, Gemini 3.1 Pro (preview), GPT-5.4, GPT-5.5, GPT-5.6 Sol.

More about this test: tau2-bench retail

What's in the test

Each task is a customer conversation that ends with the store's records either matching the right outcome or not. Sierra runs each task more than once, so it can report both how often the agent succeeds and how often it succeeds every time.

115 tasks

  • Customer conversations with tool use
  • Policy following
Technical details for tau2-bench retail

Success rate (pass^1): The share of the 115 retail customer-service tasks the agent completed, averaged over Sierra's repeated runs of each task.

  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort none: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: none.
  • Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Extended thinking on.

Source version: Sierra tau-bench submissions, gpt-5-2_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, claude-opus-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-flash_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-pro_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2-none_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, claude-sonnet-4-5_sierra_2026-02-26, downloaded 2026-10-23

Results posted 2026-02-26 to 2026-03-02

Run dates not published by the tester; the dates are when results were posted.

License: Display permitted by Sierra in writing, 2026-09-28. Checked 2026-10-23.

tau2-bench airline

best 84.0%Sierra50 tasksNewest result 2026-03-02reliability figures showncaveat

tau2-bench, built by Sierra, puts an AI agent in a customer-service job: it talks with a customer, played by another AI model, and must follow the business's policy while using the business's tools to finish the task. This source is the airline domain: bookings, changes and cancellations.

Results

Each model's best setting in this test

  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2684.0%

    Completes it 4 tries out of 4: 70.0%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2683.0%

    Completes it 4 tries out of 4: 72.0%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0282.5%

    Completes it 4 tries out of 4: 68.0%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0280.5%

    Completes it 4 tries out of 4: 66.0%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2672.0%

    Completes it 4 tries out of 4: 48.0%

Show all 6 results (1 not shown above) from tau2-bench airline
  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2684.0%

    Completes it 4 tries out of 4: 70.0%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2683.0%

    Completes it 4 tries out of 4: 72.0%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0282.5%

    Completes it 4 tries out of 4: 68.0%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0280.5%

    Completes it 4 tries out of 4: 66.0%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2672.0%

    Completes it 4 tries out of 4: 48.0%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort none, Run date not published; posted 2026-02-2652.5%

    Completes it 4 tries out of 4: 22.0%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra does not publish when it ran each model; the date shown is when the results were posted. The newest models on Sierra's site were run only on a separate banking domain, shown as its own test on this page. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability: Sierra runs each task more than once. The reliability figure is pass^4: the chance the model completes the same task on 4 tries out of 4. A big gap between the two numbers means the model is inconsistent.

Sierra built tau-bench and runs every result shown here. Sierra sells AI customer-service agents for retail, airline and telecom, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 10 models

Claude Fable 5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Gemini 2.5 Pro, Gemini 3.1 Pro (preview), GPT-5.4, GPT-5.5, GPT-5.6 Sol.

More about this test: tau2-bench airline

What's in the test

Each task is a customer conversation that ends with the business's records either matching the right outcome or not. Sierra runs each task more than once, so it can report both how often the agent succeeds and how often it succeeds every time.

50 tasks

  • Customer conversations with tool use
  • Policy following
Technical details for tau2-bench airline

Success rate (pass^1): The share of the 50 airline customer-service tasks the agent completed, averaged over Sierra's repeated runs of each task.

  • Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Extended thinking on.
  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort none: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: none.

Source version: Sierra tau-bench submissions, claude-opus-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-flash_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-pro_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, claude-sonnet-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2-none_sierra_2026-02-26, downloaded 2026-10-23

Results posted 2026-02-26 to 2026-03-02

Run dates not published by the tester; the dates are when results were posted.

License: Display permitted by Sierra in writing, 2026-09-28. Checked 2026-10-23.

tau2-bench telecom

best 92.3%Sierra114 tasksNewest result 2026-03-02reliability figures showncaveat

tau2-bench, built by Sierra, puts an AI agent in a customer-service job: it talks with a customer, played by another AI model, and must follow the business's policy while using the business's tools to finish the task. This source is the telecom domain: troubleshooting a customer's phone service, where the customer also has to take actions on their own device.

Results

Each model's best setting in this test

  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2692.3%

    Completes it 4 tries out of 4: 78.1%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0291.2%

    Completes it 4 tries out of 4: 70.2%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0291.0%

    Completes it 4 tries out of 4: 74.6%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2689.7%

    Completes it 4 tries out of 4: 71.9%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2684.9%

    Completes it 4 tries out of 4: 64.0%

Show all 6 results (1 not shown above) from tau2-bench telecom
  • Claude Opus 4.5 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2692.3%

    Completes it 4 tries out of 4: 78.1%

  • Gemini 3 Flash (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0291.2%

    Completes it 4 tries out of 4: 70.2%

  • Gemini 3 Pro (preview) Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-03-0291.0%

    Completes it 4 tries out of 4: 74.6%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort high, Run date not published; posted 2026-02-2689.7%

    Completes it 4 tries out of 4: 71.9%

  • Claude Sonnet 4.5 Agent with the domain's tools, talking to an AI-played customer, extended thinking on, Run date not published; posted 2026-02-2684.9%

    Completes it 4 tries out of 4: 64.0%

  • GPT-5.2 Agent with the domain's tools, talking to an AI-played customer, effort none, Run date not published; posted 2026-02-2657.2%

    Completes it 4 tries out of 4: 30.7%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra does not publish when it ran each model; the date shown is when the results were posted. The newest models on Sierra's site were run only on a separate banking domain, shown as its own test on this page. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability: Sierra runs each task more than once. The reliability figure is pass^4: the chance the model completes the same task on 4 tries out of 4. A big gap between the two numbers means the model is inconsistent.

Sierra built tau-bench and runs every result shown here. Sierra sells AI customer-service agents for retail, airline and telecom, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 10 models

Claude Fable 5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Gemini 2.5 Pro, Gemini 3.1 Pro (preview), GPT-5.4, GPT-5.5, GPT-5.6 Sol.

More about this test: tau2-bench telecom

What's in the test

Each task is a customer conversation that ends with the business's records either matching the right outcome or not. Sierra runs each task more than once, so it can report both how often the agent succeeds and how often it succeeds every time.

114 tasks

  • Customer conversations with tool use
  • Policy following
Technical details for tau2-bench telecom

Success rate (pass^1): The share of the 114 telecom customer-service tasks the agent completed, averaged over Sierra's repeated runs of each task.

  • Claude Opus 4.5, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Flash (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Pro (preview), agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: high.
  • Claude Sonnet 4.5, agent with the domain's tools, talking to an AI-played customer, extended thinking on: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Extended thinking on.
  • GPT-5.2, agent with the domain's tools, talking to an AI-played customer, effort none: Standard tau2-bench setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A conversation with a customer played by an AI model. Effort: none.

Source version: Sierra tau-bench submissions, claude-opus-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-flash_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-pro_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, claude-sonnet-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2-none_sierra_2026-02-26, downloaded 2026-10-23

Results posted 2026-02-26 to 2026-03-02

Run dates not published by the tester; the dates are when results were posted.

License: Display permitted by Sierra in writing, 2026-09-28. Checked 2026-10-23.

tau3-bench banking

best 48.7%Sierra97 tasksNewest result 2026-08-03reliability figures showncaveat

tau3-bench banking, built by Sierra and published as tau-knowledge, puts an AI agent in a bank's customer-service job. It talks with a customer, played by another AI model, and must find what it needs in a knowledge base of about 700 bank documents, follow the bank's policy and use the bank's tools to finish the task.

Results

Each model's best setting in this test

  • Claude Opus 5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-08-0348.7%

    Completes it 4 tries out of 4: 32.0%

  • GPT-5.6 Sol Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-07-2246.9%

    Completes it 4 tries out of 4: 27.8%

  • GPT-5.5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-07-2244.6%

    Completes it 4 tries out of 4: 29.9%

  • Claude Opus 4.7 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2340.2%

    Completes it 4 tries out of 4: 24.7%

  • Claude Fable 5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2339.7%

    Completes it 4 tries out of 4: 28.9%

  • Claude Opus 4.8 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2339.7%

    Completes it 4 tries out of 4: 22.7%

  • GPT-5.4 Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-05-0639.4%

    Completes it 4 tries out of 4: 21.6%

  • GPT-5.2 Agent with Sierra's standard search tools, talking to an AI-played customer, effort high, 2026-05-0532.2%

    Completes it 4 tries out of 4: 18.6%

Show all 16 results (8 not shown above) from tau3-bench banking
  • Claude Opus 5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-08-0348.7%

    Completes it 4 tries out of 4: 32.0%

  • GPT-5.6 Sol Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-07-2246.9%

    Completes it 4 tries out of 4: 27.8%

  • GPT-5.5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-07-2244.6%

    Completes it 4 tries out of 4: 29.9%

  • Claude Opus 4.7 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2340.2%

    Completes it 4 tries out of 4: 24.7%

  • Claude Fable 5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2339.7%

    Completes it 4 tries out of 4: 28.9%

  • Claude Opus 4.8 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-07-2339.7%

    Completes it 4 tries out of 4: 22.7%

  • GPT-5.4 Agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh, 2026-05-0639.4%

    Completes it 4 tries out of 4: 21.6%

  • GPT-5.2 Agent with Sierra's standard search tools, talking to an AI-played customer, effort high, 2026-05-0532.2%

    Completes it 4 tries out of 4: 18.6%

  • Claude Opus 4.6 Agent with Sierra's standard search tools, talking to an AI-played customer, effort max, 2026-05-0627.3%

    Completes it 4 tries out of 4: 11.3%

  • Gemini 3 Flash (preview) Agent searching with a terminal only, talking to an AI-played customer, effort high, 2026-03-0227.3%

    Completes it 4 tries out of 4: 7.2%

  • Gemini 3.1 Pro (preview) Agent with Sierra's standard search tools, talking to an AI-played customer, effort high, 2026-05-0526.0%

    Completes it 4 tries out of 4: 9.3%

  • Claude Sonnet 4.5 Agent searching with a terminal only, talking to an AI-played customer, extended thinking on, 2026-02-2425.3%

    Completes it 4 tries out of 4: 10.3%

  • Claude Opus 4.5 Agent with Sierra's standard search tools, talking to an AI-played customer, effort high, 2026-05-0524.7%

    Completes it 4 tries out of 4: 11.3%

  • Gemini 3 Pro (preview) Agent searching with a terminal only, talking to an AI-played customer, effort high, 2026-03-0218.0%

    Completes it 4 tries out of 4: 4.1%

  • Gemini 2.5 Pro Agent with Sierra's standard search tools, talking to an AI-played customer, effort high, 2026-05-0513.7%

    Completes it 4 tries out of 4: 1.0%

  • GPT-5.2 Agent searching with Qwen embeddings only, talking to an AI-played customer, effort none, 2026-05-2612.6%

    Completes it 4 tries out of 4: 4.1%

Watch out

Sierra, which runs this test, sells AI customer-service agents. The date shown is the day Sierra ran each model. Most rows let the agent search the documents with Sierra's standard set of search tools; four rows used a different search method, named in each row's setup, so compare those with care. Sierra re-scored its older banking runs on 2026-07-15 after fixing its grading and says earlier scores are not comparable; only results scored with the fixed grading are shown. For Gemini 3 Flash and Gemini 3 Pro, Sierra reports that 52 and 42 runs were stopped at the step limit and counted as failures. Banking is a different set of tasks from retail, airline and telecom, so compare models within one test, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability: Sierra runs each task more than once. The reliability figure is pass^4: the chance the model completes the same task on 4 tries out of 4. A big gap between the two numbers means the model is inconsistent.

Sierra built tau-bench and runs every result shown here. Sierra sells AI customer-service agents, the same kind of work this test covers. It does not make any of the models shown.

See Sierra's results
More about this test: tau3-bench banking

What's in the test

Each task is a customer conversation that ends with the bank's records either matching the right outcome or not. The agent has to search the bank's documents to know what to do. Sierra runs each task four times, so it can report both how often the agent succeeds and how often it succeeds every time.

97 tasks

  • Customer conversations with tool use
  • Searching a knowledge base
  • Policy following
Technical details for tau3-bench banking

Success rate (pass^1): The share of the 97 banking customer-service tasks the agent completed, averaged over Sierra's repeated runs of each task.

  • Claude Opus 5, agent with Sierra's standard search tools, talking to an AI-played customer, effort max: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: max.
  • GPT-5.6 Sol, agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: xhigh.
  • GPT-5.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: xhigh.
  • Claude Opus 4.7, agent with Sierra's standard search tools, talking to an AI-played customer, effort max: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: max.
  • Claude Fable 5, agent with Sierra's standard search tools, talking to an AI-played customer, effort max: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: max.
  • Claude Opus 4.8, agent with Sierra's standard search tools, talking to an AI-played customer, effort max: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: max.
  • GPT-5.4, agent with Sierra's standard search tools, talking to an AI-played customer, effort xhigh: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: xhigh.
  • GPT-5.2, agent with Sierra's standard search tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • Claude Opus 4.6, agent with Sierra's standard search tools, talking to an AI-played customer, effort max: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: max.
  • Gemini 3 Flash (preview), agent searching with a terminal only, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: A terminal to search the bank's documents (not Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3.1 Pro (preview), agent with Sierra's standard search tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • Claude Sonnet 4.5, agent searching with a terminal only, talking to an AI-played customer, extended thinking on: Standard tau2-bench setup, run by Sierra. Tools: A terminal to search the bank's documents (not Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Extended thinking on.
  • Claude Opus 4.5, agent with Sierra's standard search tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 3 Pro (preview), agent searching with a terminal only, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: A terminal to search the bank's documents (not Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • Gemini 2.5 Pro, agent with Sierra's standard search tools, talking to an AI-played customer, effort high: Standard tau2-bench setup, run by Sierra. Tools: Keyword search, meaning search and a sandboxed shell over the bank's documents (Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: high.
  • GPT-5.2, agent searching with Qwen embeddings only, talking to an AI-played customer, effort none: Standard tau2-bench setup, run by Sierra. Tools: Meaning search with Qwen embeddings over the bank's documents (not Sierra's standard setting), plus the bank's tools and policy. Input: A conversation with a customer played by an AI model. Effort: none.

Source version: Sierra tau-bench submissions, claude-opus-5_sierra_2026-08-04, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-6-sol_sierra_2026-08-04, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-5_sierra_2026-05-05, downloaded 2026-10-23; Sierra tau-bench submissions, claude-opus-4-7_sierra_2026-05-05, downloaded 2026-10-23; Sierra tau-bench submissions, claude-fable-5_sierra_2026-08-04, downloaded 2026-10-23; Sierra tau-bench submissions, claude-opus-4-8_sierra_2026-08-04, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-4_sierra_2026-03-25, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, claude-opus-4-6_sierra_2026-05-05, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-flash_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-1-pro-preview_sierra_2026-05-05, downloaded 2026-10-23; Sierra tau-bench submissions, claude-sonnet-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, claude-opus-4-5_sierra_2026-02-26, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-3-pro_sierra_2026-03-02, downloaded 2026-10-23; Sierra tau-bench submissions, gemini-2-5-pro_sierra_2026-05-05, downloaded 2026-10-23; Sierra tau-bench submissions, gpt-5-2-none_sierra_2026-02-26, downloaded 2026-10-23

Run 2026-02-24 to 2026-08-03

License: Display permitted by Sierra in writing, 2026-09-28. Checked 2026-10-23.

No figures on this page for: Claude Fable 5.1, Claude Haiku 4.5, Claude Haiku 5.5, Claude Opus 5.5, Claude Sonnet 4.6, Claude Sonnet 5, Claude Sonnet 5.5, Gemini 3.1 Flash-Lite (preview), Gemini 3.5 Flash, Gemini 3.5 Flash-Lite, Gemini 3.6 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 4 Argon, GPT-4.1, GPT-5 mini, GPT-5 nano, GPT-5.1, GPT-5.1 Codex, GPT-5.2 Codex, GPT-5.4 Mini, GPT-5.4 Nano, GPT-5.6 Luna, GPT-5.6 Terra, GPT-6 Astra, GPT-6 Luna, GPT-6 Sol, GPT-6.1 Sol.

Also tested by

What this doesn't tell you

Data version 2026-10-28+91ac3fc9d071 · Terms of use