Browser Gyms
Summary Failures Tasks Environment Reward hacking Provenance
// Browser Gyms · Cross-gym scored batch · Data package overview

Ten cross-gym tasks: failure patterns and pass@5 against GPT 5.6 sol

Ten real cross-gym agent tasks evaluated with DB expectation verifiers across five simulated SaaS gyms. This report leads with where and why runs failed; pass@5 is reported per task throughout.
Deccan AI · 10 tasks · 50 iterations · 39 Fail · 11 Pass · 40% failed pass@5 · 26 Aug 2026

Browser Gyms is a ten-task cross-gym batch spanning XSN, Xalendly, Xubspot, Xmail, and Xira. Each task runs five independent iterations; outcomes are scored by atomic DB expectation verifiers after browser execution.

39 of 50 iterations failed (78%). 4/10 tasks didn't have a single pass in pass@5. Section 2 summarizes failure categories; Section 3 lists failed expectations per task.

1 · Batch summary 2 · Failure landscape 3 · Task findings 4 · Environment and gyms 5 · Reward hacking mitigation 6 · Provenance

1 · Batch summary

78% iteration failure rate (39/50) for GPT 5.6 Sol evaluation runs only; Claude Opus 5 solvability proofs are excluded. pass@5: 4/10 tasks didn't have a single pass (40% task-level failure rate).

39
Fail iterations (GPT 5.6 Sol)
78%
Failure rate (GPT 5.6 Sol)
11
Pass iterations (GPT 5.6 Sol)
10
Cross-gym tasks
50
Iterations (GPT 5.6 Sol)

pass@5 by task

Bar length: passing runs (of 5). Value: pass@5 count.

Taskpass@5
CROSS-01
2/5
CROSS-02
2/5
CROSS-03
2/5
CROSS-04
1/5
CROSS-05
0/5
CROSS-06
0/5
CROSS-07
2/5
CROSS-08
2/5
CROSS-09
0/5
CROSS-10
0/5

Failed outcomes by gym

GymFailure
Xmail
39
Xira
30
Xubspot
20
Xalendly
17

Failure insight distribution

Distraction between elements
66
Unaddressed components
20
Lack of self evaluation
14
Navigation reasoning
3
Lookalike element confusion
3

2 · Failure landscape

Failure-first read of this batch (GPT 5.6 Sol runs only; solvability proofs excluded). pass@5 per task is in Section 3.

39
Fail iterations (GPT 5.6 Sol)
78%
Failure rate (GPT 5.6 Sol)
11
Pass iterations (GPT 5.6 Sol)
10
Total tasks
Self-declared completion. 14 failed expectation instances map to lack of self evaluation; models declared task completion while verifiers still reported Fail.

Failure insights

CategoryInsightCountShareExample tasksExample gyms
Distraction between elementsRequired input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (66 instances).6662%CROSS-01, CROSS-02, CROSS-05Xalendly, Xira, Xmail
Unaddressed componentsThe model struggled to complete a required operation, so the expected result never appeared in Xira (20 instances; Xira most affected).2019%CROSS-01, CROSS-04, CROSS-07Xira, Xubspot
Lack of self evaluationThe model declared completion while verifiers still reported Fail.1413%CROSS-01, CROSS-02, CROSS-07Xalendly, Xira, Xmail
Navigation reasoningThe agent did not reliably find or open the correct Xalendly surface before acting (3 failed checks across 3 runs).33%CROSS-03Xalendly
Lookalike element confusionA similar but incorrect flow or record type was used instead of the intended one (3 instances; Xira).33%CROSS-04, CROSS-07, CROSS-09Xira, Xmail

3 · Task findings

One card per task: click a task title for per-iteration metrics. Each card also shows prompt, designed difficulty, failed expectations, and failure insight sentences.

Q3 enterprise outreach

pass@5 · 2/5
CROSS-01
XSNXalendlyXubspotXmailXiraCross-gym scopeNavigation prerequisiteCross-connect
Failure rate
0.60
pass@5
2/5
Steps min
410
Steps mean
429.6
Steps max
464
Expectations
19

Designed challenge

Enterprise Sales Kickoff: Eric Jackson and Green Energy Solutions. The agent operates as Alex Morgan across 5 gyms (XSN, Xalendly, Xubspot, Xmail, Xira), executing 5 distinct cross-surface actions in a single episode.

Several steps depend on finding the correct thread, contact, or record before any write occurs.

Actions in Xalendly are expected to propagate to eric.jackson@crestbetamotors.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Alex Morgan (alex.morgan@d-ai.com). Your team is kicking off Q3 enterprise outreach across two key accounts: Crest Beta Motors through Eric Jackson, their VP of Product (eric.jackson@crestbetamotors.com), and Green Energy Solutions through Grace Chen (grace.chen@globalind.com). In XSN, create a lead list called "Q3 Enterprise Outreach Targets" with the description "Priority contacts supporting Crest Beta Motors and Green Energy Solutions expansion", and add Karen Black as a supporting contact for the outreach plan. Add a private note on Karen Black saying "Supporting contact for Q3 Crest Beta and Green Energy outreach planning." Bookmark the Summit Beta Energy alert that mentions "Summit Beta Energy has a new update" and add the comment "Energy-sector signal tracked alongside the Green Energy Solutions kickoff." on that alert.
You are Alex Morgan (alex.morgan@d-ai.com). Your team is kicking off Q3 enterprise outreach across two key accounts: Crest Beta Motors through Eric Jackson, their VP of Product (eric.jackson@crestbetamotors.com), and Green Energy Solutions through Grace Chen (grace.chen@globalind.com). In XSN, create a lead list called "Q3 Enterprise Outreach Targets" with the description "Priority contacts supporting Crest Beta Motors and Green Energy Solutions expansion", and add Karen Black as a supporting contact for the outreach plan. Add a private note on Karen Black saying "Supporting contact for Q3 Crest Beta and Green Energy outreach planning." Bookmark the Summit Beta Energy alert that mentions "Summit Beta Energy has a new update" and add the comment "Energy-sector signal tracked alongside the Green Energy Solutions kickoff." on that alert. In Xalendly, book Support Team Round Robin with Eric Jackson on Tuesday September 22, 2026 at 10:30 PM Central Time (30 min). Add meeting note "Prep enterprise support session for Crest Beta Motors." In Xubspot, link Grace Chen to Green Energy Solutions. Create deal "Renewable Platform Expansion" worth $250,000 with High priority, owned by Alex Morgan, associated with Green Energy Solutions and Grace Chen. Create CRM task "Send Green Energy platform overview" as a To-do with High priority, assigned to Alex Morgan, associated with Green Energy Solutions and Grace Chen. In Xmail, send Eric Jackson (eric.jackson@crestbetamotors.com) an email with subject "Next Steps: Crest Beta Motors Enterprise Review" whose body states the date of the Xalendly session you booked for him. Send Grace Chen (grace.chen@globalind.com) an email with subject "Green Energy Solutions deal kickoff" whose body states the value of the Renewable Platform Expansion deal you created. In Xira, add a high-priority task in the Xira Project Management project titled "Prepare onboarding materials for Green Energy Solutions". Assign it to Sofia Petrov with a due date of September 30, 2026, add the comment "Include migration timeline and stakeholder checklist for Green Energy onboarding." Create a subtask under it titled "Collect Green Energy stakeholder checklist" assigned to Sofia Petrov, move the onboarding task to In Progress, then log 2h of work with the description "Prepared Green Energy onboarding timeline and stakeholder checklist."

Failed expectations

Counts are failed-check instances across 3 Fail iteration(s).

ExpectationGymTableFail runs
Green Energy stakeholder checklist subtaskXiratickets2/3
Manual follow-up from Alex Morgan to EricXmailemails2/3
Deal kickoff email to Grace ChenXmailemails1/3
Onboarding prep task for Green EnergyXiratickets1/3

Pass vs fail gap

GPT 5.6 Sol passed 2/5; successful runs completed the full five-gym chain in ~420 steps, while failures stalled in Xira and Xmail. Fail runs most often skipped the Green Energy stakeholder checklist subtask and Alex Morgan's follow-up to Eric Jackson, and sometimes the Grace Chen deal kickoff or onboarding prep task. Passing trajectories closed every handoff; failures stopped after partial CRM or scheduling work without finishing the email and ticket steps.

Failure insights

  • Lack of self evaluation: The model declared completion while verifiers still reported Fail (3/3 Fail runs).
  • Unaddressed components: The model struggled to complete a required operation, so the expected result never appeared in Xira (2 instances; Xira most affected).
  • Distraction between elements: Required input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (1 instances).
Finding. pass@5 is 2/5; the dominant failure pattern is lack of self evaluation across this task's Fail runs.

Pioneer Delta import escalation

pass@5 · 2/5
CROSS-02
XmailXiraXSNCross-gym scopeFlow variantNavigation prerequisite
Failure rate
0.60
pass@5
2/5
Steps min
194
Steps mean
237.0
Steps max
316
Expectations
20

Designed challenge

Import Bug Triage and Account Monitoring. The agent operates as Sofia Petrov across 3 gyms (Xmail, Xira, XSN), executing 10 distinct cross-surface actions in a single episode.

Cross-surface sequencing and verifier traps combine to stress end-to-end reasoning.

Actions in Xira are expected to propagate to elena.rodriguez@d-ai.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Sofia Petrov (sofia.petrov@d-ai.com). A profile import defect is blocking Pioneer Delta Industries account onboarding and needs engineering escalation. In Xira, in the API Development project, open the ticket titled exactly "Error when update profile". Assign it to yourself (Sofia Petrov), change status to "In Progress", raise priority to "High", add the label "api", and add the comment "Escalating import profile failure blocking Pioneer Delta Industries onboarding." Then log 1h of work with the description "Triaged Pioneer Delta profile import failure." Set the team to "Backend Team", set the due date to September 22, 2026, and create a subtask titled "User import via CSV" assigned to yourself.
You are Sofia Petrov (sofia.petrov@d-ai.com). A profile import defect is blocking Pioneer Delta Industries account onboarding and needs engineering escalation. In Xira, in the API Development project, open the ticket titled exactly "Error when update profile". Assign it to yourself (Sofia Petrov), change status to "In Progress", raise priority to "High", add the label "api", and add the comment "Escalating import profile failure blocking Pioneer Delta Industries onboarding." Then log 1h of work with the description "Triaged Pioneer Delta profile import failure." Set the team to "Backend Team", set the due date to September 22, 2026, and create a subtask titled "User import via CSV" assigned to yourself. In Xmail, send Alex Morgan (alex.morgan@d-ai.com) an email with subject "Escalation: Pioneer Delta import profile failure". Mark the escalation thread Important and apply the "High Priority" label to that thread, then star that escalation email you sent. Search mail for subject:"Help needed: import" and open that thread (not your escalation thread). Apply the "High Priority" label to the import thread, then snooze it until September 22, 2026. Star the import thread, then reply with exactly "Need the CSV template and error log for Pioneer Delta profile import." In XSN, bookmark the alert for Pioneer Delta Industries that mentions "Pioneer Delta Industries has a new update". Add the comment "Import failure may be blocking Pioneer Delta profile sync. Monitor during triage." Open Pioneer Delta Industries and add the public company note "Pioneer Delta profile import blocked. Watch during Sofia triage."

Failed expectations

Counts are failed-check instances across 3 Fail iteration(s).

ExpectationGymTableFail runs
Sent reply on Help needed import threadXmailemails2/3
Import reply sent to Marcus ChenXmailemail_recipients1/3
Import reply inbox copyXmailemails1/3
High Priority label on import help threadXmailthread_labels1/3

Pass vs fail gap

GPT 5.6 Sol passed 2/5; passes finished Pioneer Delta escalation in under 200 steps, while failures averaged 260 steps and broke on Xmail. The consistent miss was the exact reply on the "Help needed: import" thread, wording, High Priority label, and inbox routing, even when Xira triage and the XSN note were done. Successful runs sent that reply as the final cross-gym handoff; failures never matched what the verifier expected.

Failure insights

  • Distraction between elements: Required input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (3 instances).
  • Lack of self evaluation: The model declared completion while verifiers still reported Fail (2/3 Fail runs).
Finding. pass@5 is 2/5; the dominant failure pattern is distraction between elements across this task's Fail runs.

HelioWind Energy onboarding

pass@5 · 2/5
CROSS-03
XalendlyXubspotXmailCross-gym scopeNavigation prerequisiteVerifier trap
Failure rate
0.60
pass@5
2/5
Steps min
186
Steps mean
276.2
Steps max
332
Expectations
15

Designed challenge

New Customer Onboarding for HelioWind Energy. The agent operates as Priya Sharma across 3 gyms (Xalendly, Xubspot, Xmail), executing 5 distinct cross-surface actions in a single episode.

Several steps depend on finding the correct thread, contact, or record before any write occurs.

Actions in Xalendly are expected to propagate to katie.smith@terraretail.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Priya Sharma (priya.sharma@d-ai.com). HelioWind Energy is a new prospect and you are setting up their onboarding path. In Xalendly, book Customer Onboarding Session with Katie Smith (katie.smith@terraretail.com) on Tuesday September 22, 2026 at 9:00 AM Eastern European Time (60 min). Make yourself the host of the meeting. Add meeting note "HelioWind onboarding goals and implementation owners confirmed."
You are Priya Sharma (priya.sharma@d-ai.com). HelioWind Energy is a new prospect and you are setting up their onboarding path. In Xalendly, book Customer Onboarding Session with Katie Smith (katie.smith@terraretail.com) on Tuesday September 22, 2026 at 9:00 AM Eastern European Time (60 min). Make yourself the host of the meeting. Add meeting note "HelioWind onboarding goals and implementation owners confirmed." In Xubspot, create company "HelioWind Energy" with domain "heliowind.co", owner Priya Sharma, city "Austin", state "Texas", and industry "Environmental Services". Add Alice Johnson (alice.johnson@acme.com) to HelioWind Energy. Create deal "HelioWind Onboarding Package" worth $180,000 with High priority, owned by Priya Sharma, associated with HelioWind Energy. Create CRM task "HelioWind onboarding checklist review" as a To-do with High priority, associated with HelioWind Energy. In Xubspot, under CRM Message Templates, create template "HelioWind Welcome Notice" with subject "Welcome to HelioWind onboarding" and body "Hi {{first_name}}, welcome to the HelioWind Energy onboarding program." In Xmail, send Katie Smith (katie.smith@terraretail.com) an email with subject "Welcome to HelioWind onboarding" using the HelioWind Welcome Notice template body. Star that welcome email you sent and apply the "High Priority" label to the welcome thread.

Failed expectations

Counts are failed-check instances across 3 Fail iteration(s).

ExpectationGymTableFail runs
Customer Onboarding Session with Katie SmithXalendlymeetings3/3

Pass vs fail gap

GPT 5.6 Sol passed 2/5; passes booked the Customer Onboarding Session with Katie Smith in Xalendly in 186–280 steps, while three fails never created the meeting. Failures clustered in Xalendly despite progress in Xubspot and Xmail. Passing runs anchored the workflow on a valid calendar event; fail runs declared progress without it.

Failure insights

  • Navigation reasoning: The agent did not reliably find or open the correct Xalendly surface before acting (3 failed checks across 3 runs).
Finding. pass@5 is 2/5; the dominant failure pattern is navigation reasoning across this task's Fail runs.

Global Industries pipeline build

pass@5 · 1/5
CROSS-04
XSNXubspotXiraCross-gym scopeNavigation prerequisiteFlow variant
Failure rate
0.80
pass@5
1/5
Steps min
369
Steps mean
416.2
Steps max
451
Expectations
24

Designed challenge

Global Industries Pipeline Build. The agent operates as Nina Patel across 3 gyms (XSN, Xubspot, Xira), executing 9 distinct cross-surface actions in a single episode.

Several steps depend on finding the correct thread, contact, or record before any write occurs.

Actions in Xira are expected to propagate to nina.patel@d-ai.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Nina Patel (nina.patel@d-ai.com). You are advancing the Global Industries opportunity and lining up delivery coordination. In XSN, create a lead list called "Global Industries Q3 Targets" with the description "Champions and influencers for the Global Industries platform upgrade", and add both Aaron Le and Karen Black to that list. Add a private note on Aaron Le saying "Global Industries expansion champion for Q3 pipeline." and a private note on Karen Black saying "Supporting influencer for the Global Industries platform upgrade." Open your conversation with Aaron Le and send the message "Hi Aaron, excited to partner on the Global Industries platform upgrade this quarter."
You are Nina Patel (nina.patel@d-ai.com). You are advancing the Global Industries opportunity and lining up delivery coordination. In XSN, create a lead list called "Global Industries Q3 Targets" with the description "Champions and influencers for the Global Industries platform upgrade", and add both Aaron Le and Karen Black to that list. Add a private note on Aaron Le saying "Global Industries expansion champion for Q3 pipeline." and a private note on Karen Black saying "Supporting influencer for the Global Industries platform upgrade." Open your conversation with Aaron Le and send the message "Hi Aaron, excited to partner on the Global Industries platform upgrade this quarter." In Xubspot, on Global Industries, update phone number to "+1-555-4104", city to "Chicago", and state to "Illinois", and add the existing deal "Global Industries Enterprise License" to the company. Create deal "Global Industries Platform Upgrade" worth $320,000 with High priority, owned by Nina Patel, stage set to "Qualified To Buy", associated with Global Industries and Alice Johnson. Create CRM task "Schedule Global Industries stakeholder workshop" as a To-do with High priority, due date October 15, 2026 at 9:00 AM, assigned to Nina Patel, notes "Confirm workshop attendees include Aaron Le and Alice Johnson.", associated with Global Industries, Alice Johnson, and the Platform Upgrade deal. In Xira, in the API Development project, create a task titled "Coordinate Global Industries rollout plan". Assign it to yourself (Nina Patel) with High priority and due date October 15, 2026, Set the task description to "Coordinate rollout milestones aligned to the Global Industries deal close timeline." and add the comment "Align rollout milestones with the Global Industries deal close timeline." Add the label "rollout" to the rollout task, link it to "Error when update profile" using a Relates to link, and create a subtask under it titled "Draft Global Industries rollout milestones" assigned to Nina Patel. Change the rollout task status to In Progress, then log 1h of work with the description "Coordinated Global Industries rollout milestones with deal timeline."

Failed expectations

Counts are failed-check instances across 4 Fail iteration(s).

ExpectationGymTableFail runs
Rollout milestones subtaskXiratickets4/4
Global Industries rollout coordination taskXiratickets1/4
Rollout milestone commentXiracomments1/4
rollout label on rollout coordination taskXiraticket_labels1/4
Rollout coordination worklogXiraworklogs1/4
Rollout task linked forward to profileXiraticket_links1/4
Rollout task linked backward from profileXiraticket_links1/4

Pass vs fail gap

GPT 5.6 Sol passed only 1/5; the lone pass completed Global Industries rollout coordination in Xira including the milestones subtask, while four fails broke on the same Xira work. Fail runs repeatedly missed the milestones subtask and its comment, label, and worklog even when earlier gyms succeeded. The gap is finishing Xira subtask depth, not starting the rollout.

Failure insights

  • Unaddressed components: The model struggled to complete a required operation, so the expected result never appeared in Xira (9 instances; Xira most affected).
  • Lookalike element confusion: A similar but incorrect flow or record type was used instead of the intended one (1 instances; Xira).
Finding. pass@5 is 1/5; the dominant failure pattern is unaddressed components across this task's Fail runs.

Global Industries login blocker and renewal

pass@5 · 0/5
CROSS-05
XiraXmailXalendlyXubspotCross-gym scopeVerifier trapFlow variant
Failure rate
1.00
pass@5
0/5
Steps min
376
Steps mean
434.6
Steps max
500
Expectations
20

Solvability proof

PassClaude Opus 5 · 839 steps · 400 msgs

Independent passing run demonstrating the task is solvable under the golden verifier.

Designed challenge

Global Industries Renewal Blocker. The agent operates as Elena Rodriguez across 4 gyms (Xira, Xalendly, Xmail, Xubspot), executing 12 distinct cross-surface actions in a single episode.

Several steps depend on finding the correct bug, thread, company, or existing deal before any write occurs.

The episode includes negative constraints the agent must respect: do not book a meeting in Xalendly (offer exactly two windows instead), and do not send the Katie Smith email (save as draft).

The September 22 date must stay consistent across the Xira due date, the first availability window, the quoted text in the Xmail reply, and the Xubspot task; the agent must keep these cross-application consequences aligned.

Task prompt Show full promptShow less
You are Elena Rodriguez (elena.rodriguez@d-ai.com). Global Industries is blocked by a login issue that is holding up renewal payment terms and a scheduled account consultation. In Xira, open the bug "Login throws IP Banned Error" in the Xira Project Management project. Assign it to yourself, change the priority to "Critical", move the status to "In Progress", and set the due date to September 22, 2026. Add the label "security", and add the comment "Global Industries users blocked from vendor portal, so the renewal workflow is on hold." Log 1 hour of work on this bug with the work description "IP ban triage and vendor coordination." Then create a subtask titled "Whitelist Global Industries IP range", assigned to yourself.
You are Elena Rodriguez (elena.rodriguez@d-ai.com). Global Industries is blocked by a login issue that is holding up renewal payment terms and a scheduled account consultation. In Xira, open the bug "Login throws IP Banned Error" in the Xira Project Management project. Assign it to yourself, change the priority to "Critical", move the status to "In Progress", and set the due date to September 22, 2026. Add the label "security", and add the comment "Global Industries users blocked from vendor portal, so the renewal workflow is on hold." Log 1 hour of work on this bug with the work description "IP ban triage and vendor coordination." Then create a subtask titled "Whitelist Global Industries IP range", assigned to yourself. In Xalendly, do not book a meeting. The consultation cannot be confirmed until the login blocker clears, so offer Katie Smith slots instead. Create a one-off meeting titled "Global Industries Renewal Consultation", hosted by yourself, 1 hour long, held on Google Meet, in Central Time, offering exactly two availability windows: Tuesday September 22, 2026 of 09:00-10:00 and Thursday September 24, 2026 of 14:00-15:00. Set its description to "Consultation slot held open until the IP Banned login blocker is cleared." In Xmail, reply in the "Payment Terms Discussion" thread with text mentioning "Global Industries Q4 renewal" and the first consultation window, quoting "September 22" and "09:00" in the body. Apply the "High Priority" label to the Payment Terms Discussion thread, star that reply email, and mark the thread Important. Then compose a new email to Katie Smith (katie.smith@terraretail.com) with the subject "Global Industries renewal status" and body text mentioning "login blocker", and save it as a draft. Do not send it. In Xubspot, open the "Global Industries" company record and update its phone number to "+1-555-6120". Then create a CRM task titled "Global Industries renewal blocker follow-up", of type "To-do", priority "High", assigned to yourself, due September 22, 2026, with the note "Hold renewal paperwork until the IP ban is lifted." Associate this task with the "Global Industries Platform Upgrade" deal.

Failed expectations

Counts are failed-check instances across 5 Fail iteration(s).

ExpectationGymTableFail runs
One-off Global Industries renewalXalendlyevents5/5
Follow-up taskXubspottask_deal_association5/5
Sent reply to Payment Terms DiscussionXmailemails4/5
Renewal status draft to Katie Smith savedXmailemails4/5
Renewal blocker follow-up CRM taskXubspotcrm_tasks4/5
Payment Terms reply inbox copyXmailemails2/5
Payment Terms thread picks up SentXmailthread_labels2/5
Global Industries phone updatedXubspotcompanies1/5
Elena hosts the one-off consultationXalendlyevent_hosts1/5
One-off consultation runs for an hourXalendlyevent_durations1/5
First window: 09:00-10:00 on Tuesday 22Xalendlyevent_segments1/5
Second window: 14:00-15:00 on Thursday 24Xalendlyevent_segments1/5

Pass vs fail gap

Claude Opus 5 passed all verifiers in 839 steps; GPT 5.6 Sol failed 5/5 (mean 455 steps). GPT runs almost never created the one-off renewal consultation, the CRM follow-up on the Platform Upgrade deal, or the Payment Terms reply and renewal-status draft. Opus completed the full renewal-blocker chain; GPT runs often triaged the Xira bug but did not carry scheduling, CRM, and email handoffs through.

Failure insights

  • Distraction between elements: Required input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (31 instances).
Finding. pass@5 is 0/5; the dominant failure pattern is distraction between elements across this task's Fail runs.

Green Energy Solutions renewal alignment

pass@5 · 0/5
CROSS-06
XubspotXSNXmailXiraCross-gym scopeFlow variantVerifier trap
Failure rate
1.00
pass@5
0/5
Steps min
382
Steps mean
419.0
Steps max
464
Expectations
30

Solvability proof

PassClaude Opus 5 · 861 steps · 439 msgs

Independent passing run demonstrating the task is solvable under the golden verifier.

Designed challenge

Green Energy Solutions Renewal Alignment. The agent operates as James Okonkwo across 4 gyms (Xubspot, Xmail, XSN, Xira), executing 15 distinct cross-surface actions in a single episode.

The mail leg requires deliberate compose behaviour: draft vs send distinctions and overlay handling are part of the intended difficulty.

Task prompt Show full promptShow less
You are James Okonkwo (james.okonkwo@d-ai.com). Green Energy Solutions is up for renewal and you want the CRM record, provisioning work, competitive intelligence, and client communication aligned. In Xubspot, update Green Energy Solutions phone number to "+1-555-3377", city to "Portland", and state to "Oregon", and link Grace Chen to Green Energy Solutions. Create order "Green Energy Solutions - Annual Renewal 2026" on the Order Pipeline in Open stage with total price $45,000, owned by James Okonkwo, associated with Grace Chen and Green Energy Solutions. Create message template "Renewal Confirmation Notice" with subject "Your Green Energy renewal is confirmed" and body "Hi {{first_name}}, your annual renewal order has been recorded." Create CRM task "Confirm Green Energy renewal paperwork" as a To-do with High priority, due date September 30, 2026, notes "Countersigned renewal paperwork due back before the order closes.", associated with Green Energy Solutions and Grace Chen. Then create deal "Green Energy Solutions Renewal Expansion" with amount $60,000, priority High, stage "Qualified To Buy", owned by James Okonkwo, associated with Green Energy Solutions and Grace Chen. After the deal exists, open the "Confirm Green Energy renewal paperwork" task again and also associate it with the "Green Energy Solutions Renewal Expansion" deal.
You are James Okonkwo (james.okonkwo@d-ai.com). Green Energy Solutions is up for renewal and you want the CRM record, provisioning work, competitive intelligence, and client communication aligned. In Xubspot, update Green Energy Solutions phone number to "+1-555-3377", city to "Portland", and state to "Oregon", and link Grace Chen to Green Energy Solutions. Create order "Green Energy Solutions - Annual Renewal 2026" on the Order Pipeline in Open stage with total price $45,000, owned by James Okonkwo, associated with Grace Chen and Green Energy Solutions. Create message template "Renewal Confirmation Notice" with subject "Your Green Energy renewal is confirmed" and body "Hi {{first_name}}, your annual renewal order has been recorded." Create CRM task "Confirm Green Energy renewal paperwork" as a To-do with High priority, due date September 30, 2026, notes "Countersigned renewal paperwork due back before the order closes.", associated with Green Energy Solutions and Grace Chen. Then create deal "Green Energy Solutions Renewal Expansion" with amount $60,000, priority High, stage "Qualified To Buy", owned by James Okonkwo, associated with Green Energy Solutions and Grace Chen. After the deal exists, open the "Confirm Green Energy renewal paperwork" task again and also associate it with the "Green Energy Solutions Renewal Expansion" deal. In Xira, create a task in the Xira Project Management project titled "Provision Green Energy Solutions renewal contract" with the description "Provisioning checklist for the Green Energy Solutions renewal contract.", priority "High", due date September 30, 2026, assigned to yourself. Move that task to "In Progress". Log 1 hour of work on it with the work description "Renewal contract provisioning kickoff." In XSN, bookmark the Summit Beta Energy alert that mentions "Summit Beta Energy has a new update". Add the comment "Monitoring energy-sector peers during Green Energy Solutions renewal cycle." on that Summit Beta Energy alert. Create a lead list named "Energy Sector Renewal Watch" with the description "Peers and contacts to monitor during the Green Energy Solutions renewal." Add Karen Black to that list. Add a private note on Karen Black that says "Track for renewal-cycle outreach after the Green Energy Solutions order closes." Add a public note on the Summit Beta Energy company that says "Energy-sector peer to watch during the Green Energy Solutions renewal cycle." In Xmail, send Grace Chen (grace.chen@globalind.com) an email with subject "Green Energy Solutions renewal confirmed" and a body that includes the order name "Green Energy Solutions - Annual Renewal 2026" and the order total "$45,000". Star that client confirmation email you sent, apply the "High Priority" label to the confirmation thread, and mark that thread Important. Send Alex Morgan (alex.morgan@d-ai.com) an email with subject "Green Energy renewal logged" and a body that mentions the paperwork due date "September 30, 2026", then mark that internal thread Important and apply the "High Priority" label to it as well. Finally, compose a new email to Grace Chen with the subject "Green Energy renewal receipt" and body text mentioning "receipt to follow", and save it as a draft. Do not send it.

Failed expectations

Counts are failed-check instances across 5 Fail iteration(s).

ExpectationGymTableFail runs
Renewal confirmation templateXubspotmessage_templates5/5
Renewal contract provisioning taskXiratickets4/5
Provisioning kickoff worklogXiraworklogs4/5
Renewal receipt draft to Grace Chen savedXmailemails2/5
Draft thread picks up DraftsXmailthread_labels2/5
Paperwork task linked to renewal expansionXubspottask_deal_association2/5
Renewal orderXubspotorder_company_association1/5

Pass vs fail gap

Claude Opus 5 passed all verifiers in 861 steps; GPT 5.6 Sol failed 5/5 (mean 419 steps). GPT runs consistently missed the renewal confirmation template, the contract provisioning task with worklog, and often the Grace Chen receipt draft in Xmail. Opus chained XSN, CRM, ticket, and draft steps; GPT runs stopped after partial social and CRM work.

Failure insights

  • Distraction between elements: Required input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (20 instances).
Finding. pass@5 is 0/5; the dominant failure pattern is distraction between elements across this task's Fail runs.

Vendor portal support escalation

pass@5 · 2/5
CROSS-07
XmailXiraXubspotCross-gym scopeFlow variantCross-connect
Failure rate
0.60
pass@5
2/5
Steps min
228
Steps mean
248.0
Steps max
273
Expectations
10

Designed challenge

Vendor Portal Support Escalation. The agent operates as Marcus Chen across 3 gyms (Xmail, Xira, Xubspot), executing 3 distinct cross-surface actions in a single episode.

Cross-surface sequencing and verifier traps combine to stress end-to-end reasoning.

Actions in Xira are expected to propagate to elena.rodriguez@d-ai.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Marcus Chen (marcus.chen@d-ai.com). A vendor portal issue reported over email needs engineering tracking and a CRM follow-up task. In Xmail, open the "mobile app not working" thread and reply with "Please share the exact settings page and steps to reproduce the vendor portal configuration error." Apply the "High Priority" label to that thread and star that reply email.
You are Marcus Chen (marcus.chen@d-ai.com). A vendor portal issue reported over email needs engineering tracking and a CRM follow-up task. In Xmail, open the "mobile app not working" thread and reply with "Please share the exact settings page and steps to reproduce the vendor portal configuration error." Apply the "High Priority" label to that thread and star that reply email. In Xira, create a bug in the API Development project titled "Vendor portal timeout on login" with High priority. Assign it to yourself (Marcus Chen) as assignee and reporter, add the comment "Repro steps requested from mobile app not working email thread." Then log 1h of work with the description "Triaged vendor portal timeout report." In Xubspot, create CRM task "Follow up on vendor portal incident" with task type "To-do", High priority, due date September 22, 2026, assigned to Marcus Chen, associated with Tech Innovators and Carol Smith (carol.smith@techinnovators.io). Create message template "Vendor Portal Incident Follow-up" with subject "Update on your portal configuration issue" and body "Hi {{first_name}}, we are investigating the vendor portal incident and will share next steps shortly."

Failed expectations

Counts are failed-check instances across 3 Fail iteration(s).

ExpectationGymTableFail runs
Vendor portal timeout bugXiratickets2/3
Vendor portal incident follow-up templateXubspotmessage_templates2/3
Repro steps comment on vendor portal bugXiracomments1/3
Vendor incident triage worklogXiraworklogs1/3

Pass vs fail gap

GPT 5.6 Sol passed 2/5; passes cleared Xmail, Xira, and Xubspot in ~245 steps, while three fails broke on Xira and Xubspot. Fail runs replied on the mobile-app thread but often never filed the vendor portal timeout bug or the follow-up message template. Successful passes treated email, bug, and CRM template as one incident workflow.

Failure insights

  • Unaddressed components: The model struggled to complete a required operation, so the expected result never appeared in Xira (4 instances; Xira most affected).
  • Lack of self evaluation: The model declared completion while verifiers still reported Fail (1/3 Fail runs).
  • Lookalike element confusion: A similar but incorrect flow or record type was used instead of the intended one (1 instances; Xira).
Finding. pass@5 is 2/5; the dominant failure pattern is unaddressed components across this task's Fail runs.

Pioneer Delta outreach and invoice triage

pass@5 · 2/5
CROSS-08
XSNXmailXiraCross-gym scopeVerifier trapFlow variant
Failure rate
0.60
pass@5
2/5
Steps min
142
Steps mean
179.2
Steps max
233
Expectations
17

Designed challenge

Prospect Outreach to Pioneer Delta Holdings. The agent operates as Daniel Kim across 3 gyms (XSN, Xmail, Xira), executing 11 distinct cross-surface actions in a single episode.

Verification is aligned to a known failure mode: a specific UI trap.

Actions in XSN are expected to propagate to clifford.torres@pioneerdeltaholdings.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Daniel Kim (daniel.kim@d-ai.com). You are warming up outreach to Pioneer Delta Holdings while finance needs a quick internal note on an open invoice ticket before your prospect meeting. In XSN, create a Smart Link titled "Pioneer Delta Engineering Trends Brief" with a website item labeled "Engineering Leadership Benchmark" and URL "https://hbr.org/topic/subject/technology". Then open your conversation with Clifford Torres and send the message "Hi Clifford, I have a short benchmark on engineering leadership trends for Pioneer Delta Holdings and would value your perspective."
You are Daniel Kim (daniel.kim@d-ai.com). You are warming up outreach to Pioneer Delta Holdings while finance needs a quick internal note on an open invoice ticket before your prospect meeting. In XSN, create a Smart Link titled "Pioneer Delta Engineering Trends Brief" with a website item labeled "Engineering Leadership Benchmark" and URL "https://hbr.org/topic/subject/technology". Then open your conversation with Clifford Torres and send the message "Hi Clifford, I have a short benchmark on engineering leadership trends for Pioneer Delta Holdings and would value your perspective." In Xmail, send clifford.torres@pioneerdeltaholdings.com an email with subject "Engineering trends brief for Pioneer Delta Holdings" and a body that includes the benchmark URL "https://hbr.org/topic/subject/technology" (the Engineering Leadership Benchmark link from your Smart Link) and echoes your XSN message with the phrase "value your perspective". Mark the outreach thread Important and apply the "High Priority" label to that outreach thread, then star that outreach email you sent. Search mail for subject:"Help needed: billing" and open that thread (not your outreach thread). Star the billing thread, apply the "High Priority" label to it, then snooze it until September 22, 2026. In Xira, open ticket "Issue with invoice". Add the comment "Looping in finance for vendor invoice clarification before the Pioneer Delta Holdings prospect meeting.", add the label "needs-review", then link "Issue with invoice" to "Help needed: API" using the "relates to" relationship.

Failed expectations

Counts are failed-check instances across 3 Fail iteration(s).

ExpectationGymTableFail runs
High Priority label on Help needed billingXmailthread_labels3/3

Pass vs fail gap

GPT 5.6 Sol passed 2/5; passes finished sprint triage in 168–174 steps, while three fails broke on one Xmail step. Every failure missed the High Priority label on the Help needed billing thread despite correct Xira and Xubspot work. The gap is a single labeling action on the same thread passes handled.

Failure insights

  • Lack of self evaluation: The model declared completion while verifiers still reported Fail (3/3 Fail runs).
Finding. pass@5 is 2/5; the dominant failure pattern is lack of self evaluation across this task's Fail runs.

Vertex Fusion Bio triage

pass@5 · 0/5
CROSS-09
XiraXmailXSNCross-gym scopeFlow variant
Failure rate
1.00
pass@5
0/5
Steps min
157
Steps mean
211.6
Steps max
266
Expectations
14

Solvability proof

PassClaude Opus 5 · 328 steps · 162 msgs

Independent passing run demonstrating the task is solvable under the golden verifier.

Designed challenge

Vertex Fusion Bio Coordinated Triage. The agent operates as Amira Hassan across 3 gyms (Xira, Xmail, XSN), executing 3 distinct cross-surface actions in a single episode.

Cross-surface sequencing and verifier traps combine to stress end-to-end reasoning.

Task prompt Show full promptShow less
You are Amira Hassan (amira.hassan@d-ai.com). Vertex Fusion Bio has surfaced across support, email, and account intelligence channels and needs coordinated triage before handoff to operations. In Xira, open the ticket "Help needed: invoice" in the Xira Project Management project and add the comment "Vertex Fusion Bio vendor invoice needs billing code confirmation before payment processing." Link "Help needed: invoice" to "Request: billing" using a Relates to link. Under "Help needed: invoice", create a subtask titled "Confirm Vertex Fusion Bio billing code" assigned to Sofia Petrov. Then log 1h of work on "Help needed: invoice" with the description "Reviewed Vertex Fusion Bio invoice and opened the billing code confirmation subtask."
You are Amira Hassan (amira.hassan@d-ai.com). Vertex Fusion Bio has surfaced across support, email, and account intelligence channels and needs coordinated triage before handoff to operations. In Xira, open the ticket "Help needed: invoice" in the Xira Project Management project and add the comment "Vertex Fusion Bio vendor invoice needs billing code confirmation before payment processing." Link "Help needed: invoice" to "Request: billing" using a Relates to link. Under "Help needed: invoice", create a subtask titled "Confirm Vertex Fusion Bio billing code" assigned to Sofia Petrov. Then log 1h of work on "Help needed: invoice" with the description "Reviewed Vertex Fusion Bio invoice and opened the billing code confirmation subtask." In Xmail, open the email thread whose subject contains "Error when configure settings". Reply with "Need verification details on the Vertex Fusion Bio settings export format" and save the reply as a draft. Apply the "High Priority" label to the Error when configure settings thread. In XSN, bookmark the Vertex Fusion Bio account alert. Add a public note on the Vertex Fusion Bio company account saying "Vertex Fusion Bio flagged across invoice, settings, and account signals. Route to operations."

Failed expectations

Counts are failed-check instances across 5 Fail iteration(s).

ExpectationGymTableFail runs
Billing code confirmation subtaskXiratickets5/5
Subtask assignment notification lands inXmailthread_labels5/5
Subtask assignment notification also picksXmailthread_labels5/5
Draft reply on settings error threadXmailemails1/5
Settings thread picks up DraftsXmailthread_labels1/5

Pass vs fail gap

Claude Opus 5 passed all verifiers in 328 steps; GPT 5.6 Sol failed 5/5 (mean 212 steps). No GPT run created the billing code confirmation subtask or sent the assignment notifications to Sofia's inbox and All Mail. Opus completed the Vertex Fusion Bio invoice triage, settings-thread draft, and full subtask plus notify chain; GPT runs stopped after partial mail or XSN work.

Failure insights

  • Distraction between elements: Required input was lost to adjacent UI controls or overlays (e.g. autocomplete covering a field) (11 instances).
  • Unaddressed components: The model struggled to complete a required operation, so the expected result never appeared in Xira (5 instances; Xira most affected).
  • Lookalike element confusion: A similar but incorrect flow or record type was used instead of the intended one (1 instances; Xmail).
Finding. pass@5 is 0/5; the dominant failure pattern is distraction between elements across this task's Fail runs.

Vertex Fusion Bio research and project review

pass@5 · 0/5
CROSS-10
XSNXalendlyXmailCross-gym scopeVerifier trapNavigation prerequisite
Failure rate
1.00
pass@5
0/5
Steps min
201
Steps mean
250.8
Steps max
285
Expectations
15

Solvability proof

PassClaude Opus 5 · 324 steps · 167 msgs

Independent passing run demonstrating the task is solvable under the golden verifier.

Designed challenge

Vertex Fusion Bio Research and Project Review. The agent operates as Marcus Chen across 3 gyms (XSN, Xalendly, Xmail), executing 7 distinct cross-surface actions in a single episode.

Verification is aligned to a known failure mode: a specific UI trap.

Actions in Xalendly are expected to propagate to katie.smith@terraretail.com; the agent must keep cross-application consequences consistent.

Task prompt Show full promptShow less
You are Marcus Chen (marcus.chen@d-ai.com). You are preparing Vertex Fusion Bio account research and scheduling a project review with Katie Smith to walk through findings. In XSN, bookmark the Vertex Fusion Bio account alert. Create a Smart Link titled "Vertex Fusion Bio Research Pack" with a website item labeled "Vertex Bio Market Brief - Expansion Trends" and URL "https://www.vertexbiobrief.example.com/industries/life-sciences". Add a public note on the Vertex Fusion Bio company record saying "Research pack prepared for October project review with Katie Smith."
You are Marcus Chen (marcus.chen@d-ai.com). You are preparing Vertex Fusion Bio account research and scheduling a project review with Katie Smith to walk through findings. In XSN, bookmark the Vertex Fusion Bio account alert. Create a Smart Link titled "Vertex Fusion Bio Research Pack" with a website item labeled "Vertex Bio Market Brief - Expansion Trends" and URL "https://www.vertexbiobrief.example.com/industries/life-sciences". Add a public note on the Vertex Fusion Bio company record saying "Research pack prepared for October project review with Katie Smith." In Xalendly, book Project Review with Katie Smith (katie.smith@terraretail.com) on Monday October 5, 2026 (Greenwich Mean Time) 5:00pm - 6:00pm. Make yourself the host of the meeting. Add meeting note "Vertex Fusion Bio research pack ready for project review." In Xmail, send Katie Smith (katie.smith@terraretail.com) an email with subject "Project Review scheduled for October 5, 2026". The body must include "Vertex Fusion Bio", "https://www.vertexbiobrief.example.com/industries/life-sciences", and the phrase "Project Review". Draft (do not send) a follow-up to Katie with subject "Follow-up: Vertex Fusion Bio research pack" and the same research URL in the body. Mark the confirmation thread Important and apply the "High Priority" label to that confirmation thread, then star that confirmation email you sent.

Failed expectations

Counts are failed-check instances across 5 Fail iteration(s).

ExpectationGymTableFail runs
Project Review with Katie SmithXalendlymeetings5/5

Pass vs fail gap

Claude Opus 5 passed all verifiers in 324 steps; GPT 5.6 Sol failed 5/5 (mean 251 steps). Every GPT failure missed booking the Project Review with Katie Smith in Xalendly despite often completing XSN and Xmail prep. Opus linked outreach to a concrete calendar event; GPT runs reached scheduling UI but never produced the required meeting.

Failure insights

  • Lack of self evaluation: The model declared completion while verifiers still reported Fail (5/5 Fail runs).
Finding. pass@5 is 0/5; the dominant failure pattern is lack of self evaluation across this task's Fail runs.

4 · Environment and gym flows

Five gyms with equal visual weight: same card structure and balanced documented flow lists.

Prompt

Multi-step instructions spanning 3–5 gyms with field-level requirements.

Execution

The agent navigates each gym via browser automation, mutating backend databases.

Verification

DB expectation verifiers compare post-run state against frozen expectations per gym.

XSN

Social CRM: contacts, lists, alerts

Open gym ↗
  • Sales User Builds a Target Account List
  • Sales User Discovers and Saves New Leads
  • Sales User Explores Accounts Dashboard and Insights

Xalendly

Scheduling: meetings and availability

Open gym ↗
  • Host (Primary User)
  • Invitee (External Participant)
  • Admin or Team Owner

Xubspot

CRM: deals, tasks, companies

Open gym ↗
  • Creating a Deal
  • Creating a Message Template
  • Creating a Task

Xmail

Email: threads, labels, compose

Open gym ↗
  • End-to-end email processing and organization
  • Managing multiple inputs via Compose overlay
  • Search and Retrieval

Xira

Project management: tickets, worklogs

Open gym ↗
  • Admin Manages Users & Permissions
  • Applying a Workflow Transition
  • Contributor Triages the Issue

5 · Reward hacking mitigation

How we keep evaluation honest across all models in this batch.

Every model in this batch, GPT 5.6 Sol and Claude Opus 5, is evaluated against the same deterministic expectation verifiers. Verifier definitions and run-time pass/fail results are not exposed to the agent; the model receives only the task prompt and browser access. During execution it may recheck outcomes through the UI, relying on visible data persistence the way a human operator would, but it has no access to verifier logic or hidden inspection APIs. Scoring is applied after the run by comparing application state changes to frozen golden expectations the model never sees, which mitigates reward hacking against the evaluation harness.

6 · Provenance

Each task exposes the agent to cross-gym scenarios in a Playwright-sandboxed browser environment. The agent was allowed to perform the actions and workflows covered in well-defined prompts only; the model received no additional context, hints, or ground-truth APIs. Results reflect the model's own navigation, reasoning, and self-evaluation during the run, scored after execution by DB expectation verifiers. Run metrics in this report are computed from Tasks/ zip archives at generation time (26 Aug 2026).

Models had no restrictions on steps or API calls. They were free to perform the task and were not stopped until they acknowledged task completion or gave up.

Batch folder layout

Each task in a batch export is a self-contained folder tree: one folder per task, one subfolder per model, one subfolder per iteration.

CROSS-NN/ # one folder per task in the batch {Model Name}/ # model display name used for that task iteration-1-fail/ # one run; -pass or -fail from the verifier conversation/ conversation.json # full model chat transcript for this iteration db_diff/ db_diff.json # summary of database rows that changed during the run logs/ execution.log # raw execution logs from the worker for this run replay/ replay.html # standalone html you can open offline to replay the run screenshots/ before.png, step-N-{before,after}.png # before/after pngs captured at each browser step task_responses/ response.json # the model's final answer for the task timeline/ action_timeline.json # json of every action, url, and step in order verification/ verification.json # verifier output, scores, and pass/fail details iteration-2-pass/ ... ...
conversation/conversation.json: Full model chat transcript for this iteration
db_diff/db_diff.json: Summary of database rows that changed during the run
logs/execution.log: Raw execution logs from the worker for this run
replay/replay.html: Standalone HTML you can open offline to replay the run
screenshots/before.png, step-N-{before,after}.png: Before/after PNGs captured at each browser step
task_responses/response.json: The model's final answer for the task
timeline/action_timeline.json: JSON of every action, URL, and step in order
verification/verification.json: Verifier output, scores, and pass/fail details

Headline stats in this HTML are aggregated from verification/, timeline/, and task_responses/. Full per-iteration detail (raw expectation identifiers, verifier JSON, trajectories, transcripts, logs, screenshots) is in browser_gyms.json and the audit XLSX export.