Assurance

Voice agents

A voice agent on a spoken call with a customer: handling a request with business tools, such as changing a booking or fixing phone service, or calling a customer back to collect a detail exactly, such as a name, a date or a phone number. Good means the job is done correctly by the end of the call.

Sources last checked 2026-10-23.

Key evidence rings earned, out of 4Agent works through the task with tools, as the words after it sayFilled: best score in that test. Dashed: tested, figures not shown here.

At a glance

independent tester: An independent group tested this, earned scores shown here, from the last year: We show their numbers, published in the last year, earned two or more testers: Two or more groups tested it, not yet repeat runs, some or all models: Results checked by repeat runs, for some or all models, not yet

Vendor claims have not been collected for any category yet.

TVR 78.9%TVR 78.9%TVA 82.0%TVA 82.0%TVT 84.2%TVT 84.2%TEI 60.3%TEI 60.3%
ⓘ More about this category
  • Reliability not yet measured
  • 4 sources, 7 models with figures
  • Sources last checked 2026-10-30
  • Ring 4: needs rings 1 to 3 first
  • TVR = tau2-bench voice, retail, TVA = tau2-bench voice, airline, TVT = tau2-bench voice, telecom, TEI = τ-Entity intake calls, six languages

Who does well

TestBestBest scoreNextNext score
tau2-bench voice, retailGPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort78.9%GPT Realtime 2 Voice agent with the business's tools, effort xhigh47.4%
tau2-bench voice, airlineGPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort82.0%Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high64.0%
tau2-bench voice, telecomGPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort84.2%Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high21.9%
τ-Entity intake calls, six languagesGPT Live 1 Voice agent calling the customer back, GPT-6 Astra behind it at medium effort60.3%Gemini 3.8 Live Extended Thinking Voice agent calling the customer back, thinking high43.8%
Place within each test. Places are never added up across tests.
Modeltau2-bench voice, retailtau2-bench voice, airlinetau2-bench voice, telecomτ-Entity intake calls, six languages
Gemini 3.1 Flash Live3 of 6Voice agent with the business's tools, thinking high2 of 6Voice agent with the business's tools, thinking high2 of 6Voice agent with the business's tools, thinking high
Gemini Live 2.5 Flash6 of 6Voice agent with the business's tools, effort not published6 of 6Voice agent with the business's tools, effort not published6 of 6Voice agent with the business's tools, effort not published
GPT Live 11 of 6Voice agent with the business's tools, GPT-6 Astra behind it at medium effort1 of 6Voice agent with the business's tools, GPT-6 Astra behind it at medium effort1 of 6Voice agent with the business's tools, GPT-6 Astra behind it at medium effort1 of 2Voice agent calling the customer back, GPT-6 Astra behind it at medium effort
GPT Realtime 1.05 of 6Voice agent with the business's tools, effort not published5 of 6Voice agent with the business's tools, effort not published5 of 6Voice agent with the business's tools, effort not published
GPT Realtime 1.54 of 6Voice agent with the business's tools, effort not published4 of 6Voice agent with the business's tools, effort not published4 of 6Voice agent with the business's tools, effort not published
GPT Realtime 22 of 6Voice agent with the business's tools, effort xhigh3 of 6Voice agent with the business's tools, effort xhigh2 of 6Voice agent with the business's tools, effort xhigh

Findings for this category aren't written yet.

The tests behind this

tau2-bench voice, retail

best 78.9%Sierra114 tasksNewest result 2026-09-09reliability not measuredcaveat

tau2-bench voice, built by Sierra, puts an AI voice agent on a spoken call with a customer whose part is spoken by another AI model. The agent must follow the business's policy and use the business's tools to finish the task. This source is the retail domain: orders, returns and exchanges.

Results

Each model's best setting in this test

  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0978.9%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2447.4%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0245.6%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0344.7%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2736.0%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0329.8%
Show all 8 results (2 not shown above) from tau2-bench voice, retail
  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0978.9%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2447.4%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0245.6%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0344.7%
  • GPT Realtime 2 Voice agent with the business's tools, effort minimal, 2026-07-0139.5%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2736.0%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0329.8%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking minimal, 2026-04-1326.3%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra publishes no repeat-run figure for these voice runs, so there is no measure of how consistently a model succeeds; where its files say how many tries it ran, it is one per task. The customer is an AI model with a generated voice, not a person. Results from before and after Sierra's July 2026 update (version 1.0.1) are shown together because Sierra says that update changed only its banking tasks. GPT Live 1 was run the day before OpenAI announced it, on a build named gpt-live-1-diamond-alpha. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability not measured by this source.

Sierra built tau-bench and ran every result shown here. Sierra sells AI customer-service agents, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 1 model

Gemini 3.8 Live Extended Thinking.

More about this test: tau2-bench voice, retail

What's in the test

Each task is a spoken customer call that ends with the business's records either matching the right outcome or not. Both sides can speak at once, so the customer can interrupt the agent.

114 tasks

  • Spoken customer calls with tool use
  • Policy following
Technical details for tau2-bench voice, retail

Success rate (pass^1): The share of the 114 retail tasks the voice agent completed in Sierra's run.

  • GPT Live 1, voice agent with the business's tools, GPT-6 Astra behind it at medium effort: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Voice model build gpt-live-1-diamond-alpha, handing work to GPT-6 Astra at medium effort.
  • GPT Realtime 2, voice agent with the business's tools, effort xhigh: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: xhigh.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking high: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: high.
  • GPT Realtime 1.5, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • GPT Realtime 2, voice agent with the business's tools, effort minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: minimal.
  • GPT Realtime 1.0, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • Gemini Live 2.5 Flash, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: minimal.

Source version: Sierra tau-bench voice submissions, gpt-live-1-diamond-alpha_openai_2026-09-09, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2_openai_2026-06-29, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-high_google_2026-04-02, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1.5_sierra_2026-03-03, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2-minimal_openai_2026-07-01, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1-0_openai_2026-04-13, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-live-2.5-flash_sierra_2026-03-03, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-minimal_google_2026-04-13, downloaded 2026-10-30

Run 2026-03-03 to 2026-09-09

License: Display permitted by Sierra in writing, 2026-10-28. Checked 2026-10-30.

tau2-bench voice, airline

best 82.0%Sierra50 tasksNewest result 2026-09-09reliability not measuredcaveat

tau2-bench voice, built by Sierra, puts an AI voice agent on a spoken call with a customer whose part is spoken by another AI model. The agent must follow the business's policy and use the business's tools to finish the task. This source is the airline domain: bookings, changes and cancellations.

Results

Each model's best setting in this test

  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0982.0%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0264.0%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2458.0%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0340.0%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2736.0%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0330.0%
Show all 8 results (2 not shown above) from tau2-bench voice, airline
  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0982.0%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0264.0%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2458.0%
  • GPT Realtime 2 Voice agent with the business's tools, effort minimal, 2026-07-0156.0%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking minimal, 2026-04-1342.0%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0340.0%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2736.0%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0330.0%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra publishes no repeat-run figure for these voice runs, so there is no measure of how consistently a model succeeds; where its files say how many tries it ran, it is one per task. The customer is an AI model with a generated voice, not a person. Results from before and after Sierra's July 2026 update (version 1.0.1) are shown together because Sierra says that update changed only its banking tasks. GPT Live 1 was run the day before OpenAI announced it, on a build named gpt-live-1-diamond-alpha. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability not measured by this source.

Sierra built tau-bench and ran every result shown here. Sierra sells AI customer-service agents, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 1 model

Gemini 3.8 Live Extended Thinking.

More about this test: tau2-bench voice, airline

What's in the test

Each task is a spoken customer call that ends with the business's records either matching the right outcome or not. Both sides can speak at once, so the customer can interrupt the agent.

50 tasks

  • Spoken customer calls with tool use
  • Policy following
Technical details for tau2-bench voice, airline

Success rate (pass^1): The share of the 50 airline tasks the voice agent completed in Sierra's run.

  • GPT Live 1, voice agent with the business's tools, GPT-6 Astra behind it at medium effort: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Voice model build gpt-live-1-diamond-alpha, handing work to GPT-6 Astra at medium effort.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking high: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: high.
  • GPT Realtime 2, voice agent with the business's tools, effort xhigh: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: xhigh.
  • GPT Realtime 2, voice agent with the business's tools, effort minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: minimal.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: minimal.
  • GPT Realtime 1.5, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • GPT Realtime 1.0, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • Gemini Live 2.5 Flash, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.

Source version: Sierra tau-bench voice submissions, gpt-live-1-diamond-alpha_openai_2026-09-09, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-high_google_2026-04-02, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2_openai_2026-06-29, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2-minimal_openai_2026-07-01, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-minimal_google_2026-04-13, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1.5_sierra_2026-03-03, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1-0_openai_2026-04-13, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-live-2.5-flash_sierra_2026-03-03, downloaded 2026-10-30

Run 2026-03-03 to 2026-09-09

License: Display permitted by Sierra in writing, 2026-10-28. Checked 2026-10-30.

tau2-bench voice, telecom

best 84.2%Sierra114 tasksNewest result 2026-09-09reliability not measuredcaveat

tau2-bench voice, built by Sierra, puts an AI voice agent on a spoken call with a customer whose part is spoken by another AI model. The agent must follow the business's policy and use the business's tools to finish the task. This source is the telecom domain: troubleshooting a customer's phone service, where the customer also has to take actions on their own device.

Results

Each model's best setting in this test

  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0984.2%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0221.9%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2421.9%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0321.1%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2719.3%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0317.5%
Show all 8 results (2 not shown above) from tau2-bench voice, telecom
  • GPT Live 1 Voice agent with the business's tools, GPT-6 Astra behind it at medium effort, 2026-09-0984.2%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking high, 2026-04-0221.9%
  • GPT Realtime 2 Voice agent with the business's tools, effort xhigh, 2026-06-2421.9%
  • GPT Realtime 1.5 Voice agent with the business's tools, effort not published, 2026-03-0321.1%
  • GPT Realtime 2 Voice agent with the business's tools, effort minimal, 2026-07-0120.2%
  • GPT Realtime 1.0 Voice agent with the business's tools, effort not published, 2026-03-2719.3%
  • Gemini 3.1 Flash Live Voice agent with the business's tools, thinking minimal, 2026-04-1317.5%
  • Gemini Live 2.5 Flash Voice agent with the business's tools, effort not published, 2026-03-0317.5%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Sierra publishes no repeat-run figure for these voice runs, so there is no measure of how consistently a model succeeds; where its files say how many tries it ran, it is one per task. The customer is an AI model with a generated voice, not a person. Results from before and after Sierra's July 2026 update (version 1.0.1) are shown together because Sierra says that update changed only its banking tasks. GPT Live 1 was run the day before OpenAI announced it, on a build named gpt-live-1-diamond-alpha. Each domain (retail, airline, telecom) is a different set of tasks, so compare models within one domain, never across them. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability not measured by this source.

Sierra built tau-bench and ran every result shown here. Sierra sells AI customer-service agents, the same kinds of tasks this test covers. It does not make any of the models shown.

See Sierra's results
Not tested here: 1 model

Gemini 3.8 Live Extended Thinking.

More about this test: tau2-bench voice, telecom

What's in the test

Each task is a spoken customer call that ends with the business's records either matching the right outcome or not. Both sides can speak at once, so the customer can interrupt the agent.

114 tasks

  • Spoken customer calls with tool use
  • Policy following
Technical details for tau2-bench voice, telecom

Success rate (pass^1): The share of the 114 telecom tasks the voice agent completed in Sierra's run.

  • GPT Live 1, voice agent with the business's tools, GPT-6 Astra behind it at medium effort: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Voice model build gpt-live-1-diamond-alpha, handing work to GPT-6 Astra at medium effort.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking high: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: high.
  • GPT Realtime 2, voice agent with the business's tools, effort xhigh: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: xhigh.
  • GPT Realtime 1.5, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • GPT Realtime 2, voice agent with the business's tools, effort minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: minimal.
  • GPT Realtime 1.0, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.
  • Gemini 3.1 Flash Live, voice agent with the business's tools, thinking minimal: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Thinking: minimal.
  • Gemini Live 2.5 Flash, voice agent with the business's tools, effort not published: Standard tau2-bench voice setup, run by Sierra. Tools: The domain's tools and policy guidelines. Input: A spoken conversation with a customer voiced by an AI model. Effort: not published.

Source version: Sierra tau-bench voice submissions, gpt-live-1-diamond-alpha_openai_2026-09-09, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-high_google_2026-04-02, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2_openai_2026-06-29, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1.5_sierra_2026-03-03, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-2-minimal_openai_2026-07-01, downloaded 2026-10-30; Sierra tau-bench voice submissions, gpt-realtime-1-0_openai_2026-04-13, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-3-1-flash-live-preview-thinking-minimal_google_2026-04-13, downloaded 2026-10-30; Sierra tau-bench voice submissions, gemini-live-2.5-flash_sierra_2026-03-03, downloaded 2026-10-30

Run 2026-03-03 to 2026-09-09

License: Display permitted by Sierra in writing, 2026-10-28. Checked 2026-10-30.

τ-Entity intake calls, six languages

best 60.3%Sierra200 tasks per languageNewest result 2026-10-20reliability not measuredcaveat

τ-Entity, part of Sierra's tau-bench, tests whether a voice agent calling a customer back can collect one exact missing detail and save it in the business's record. The calls are for a clinic, an auto service and a hotel, and the customer's part is spoken by another AI model.

Results

  • GPT Live 1 Voice agent calling the customer back, GPT-6 Astra behind it at medium effort, Run date not published; posted 2026-10-2060.3%
  • Gemini 3.8 Live Extended Thinking Voice agent calling the customer back, thinking high, Run date not published; posted 2026-10-0743.8%

Watch out

Sierra, which runs this test, sells AI customer-service agents. Each task was tried once per language, so there is no measure of how consistently a model succeeds. The customer is an AI model with a generated voice, not a person. The date shown is the date Sierra gives each entry. Sierra does not give the day each model was run, and it has updated some entries since that date. Sierra's notes say the Gemini entry includes calls recovered after infrastructure problems. Each language's score is shown under More about this test. Data: Sierra, tau-bench leaderboard, shown with Sierra's permission.

Reliability not measured by this source.

Sierra built tau-bench and ran every result shown here. Sierra sells AI customer-service agents. It does not make any of the models shown.

See Sierra's results
Not tested here: 5 models

Gemini 3.1 Flash Live, Gemini Live 2.5 Flash, GPT Realtime 1.0, GPT Realtime 1.5, GPT Realtime 2.

More about this test: τ-Entity intake calls, six languages

What's in the test

200 tasks in each of six languages: English, Spanish, Portuguese, Hindi, Chinese and Korean. Each task asks for one missing detail, such as a person's name, a phone number, a date, a time, an address, an email address, a code or a medication. A wrong value, a missing value or a value in the wrong format fails the task.

200 tasks per language

  • Spoken callbacks
  • Exact details: names, numbers, dates, addresses

Where results change

By language

200 tasks in each language, scored separately.

Model and setupEnglishSpanishPortugueseHindiChineseKorean
GPT Live 1Voice agent calling the customer back, GPT-6 Astra behind it at medium effort74.0%73.5%69.0%56.5%57.5%31.5%
Gemini 3.8 Live Extended ThinkingVoice agent calling the customer back, thinking high50.5%54.0%45.5%51.0%43.0%18.5%
Technical details for τ-Entity intake calls, six languages

Success rate: The share of 1,200 calls (200 tasks in each of six languages) where the agent saved the right value in the business's record, in the format the record requires.

  • GPT Live 1, voice agent calling the customer back, GPT-6 Astra behind it at medium effort: Standard tau-bench 2.0 voice setup (τ-Entity), run by Sierra. Tools: A tool to save each value in the business's record. Input: A spoken callback to a customer voiced by an AI model, in one of six languages. Voice model gpt-live-1, handing work to GPT-6 Astra at medium effort.
  • Gemini 3.8 Live Extended Thinking, voice agent calling the customer back, thinking high: Standard tau-bench 2.0 voice setup (τ-Entity), run by Sierra. Tools: A tool to save each value in the business's record. Input: A spoken callback to a customer voiced by an AI model, in one of six languages. Thinking: high.

Source version: Sierra tau-bench voice-v2-entity packages, gpt-live-1_openai_2026-10-20_entity, downloaded 2026-10-30; Sierra tau-bench voice-v2-entity packages, gemini-3-8-live-extended-thinking_google_2026-10-07_entity, downloaded 2026-10-30

Results posted 2026-10-07 to 2026-10-20

Run dates not published by the tester; the dates are when results were posted.

License: Display permitted by Sierra in writing, 2026-10-28. Checked 2026-10-30.

What this doesn't tell you

Data version 2026-10-30+e615fc456b6e · Terms of use