Oracle / No-op
1.0 / 0.0
Deccan SWE Samples is a batch of ten environment tasks built to the DeepSWE verifiable-RL delivery format. Each task was mined from a live production codebase, then authored and calibrated against a frontier reference model on a fixed harness — 50 rollouts in total, every one of which ships with the delivery alongside its verifier output and, where it failed, a written analysis of why.
The design intent is narrower than "make hard tasks". A task is only useful to us if the reason it is hard is legible after the fact: if a failure can be traced to a named assertion, and that assertion can be traced to a requirement a maintainer would actually have written. Most of the effort in this batch went into that traceability rather than into difficulty, and the three properties below are the ones we think distinguish it from tasks scraped from public repositories.
All ten tasks carry documented acceptance-gate evidence — the environment builds from the delivered files alone, the oracle patch scores reward = 1.0 and the no-op submission scores 0.0. On salary-details the gates were executed last (2026-08-19) and recorded in its report.md: G1 at reward 1.0 (36/36 checks) and G2 at 0.0 (3/36), both on the delivered task_checksum. Frontier rollouts (gpt-5.6-sol on terminus-2@2.0.0, temperature 1) resolve 15 of 50 attempts — 30.0% in aggregate, with no individual task exceeding the 40% ceiling.
That aggregate is a coarse summary and is best distrusted, which shapes how the rest of the report is organised. A single 30% figure implies a uniform difficulty the per-task data does not bear out: with five rollouts per task the frontier measurement resolves each task either once or twice, sorting the batch into two coarse bands but no finer. The resolve rate is therefore reported per task throughout, and the qualitative failure analysis in Section 8 carries most of the weight, since it localises each failure to a named assertion rather than to a bare reward of zero.
The table below reports the frontier pass rate for each task — gpt-5.6-sol on the terminus-2@2.0.0 harness at temperature 1, five independent rollouts per task. Lower is harder. The DeepSWE format requires between zero and two resolves out of five, i.e. a rate no higher than 40%, and all ten tasks qualify.
A caution about resolution, since it governs how much weight this ordering can bear. With five rollouts per task, a single resolve is worth twenty percentage points, so the frontier measurement can only sort these tasks into two coarse bands — the five resolved once and the five resolved twice — and cannot distinguish within them. Two tasks both reported at 2/5 may differ substantially in true difficulty; this measurement simply cannot tell. That coarseness is why the per-assertion failure analysis in Section 8 carries the finer-grained account of how the tasks differ.
On reward hacking: the audit found none, so no run was removed from any count here. The exclusion rule is stated because it is the rule these numbers were computed under, not because it changed them. Had a passing run been confirmed as gaming the verifier it would have been struck and the rate recomputed (DeepSWE format). Method in Section 9.
Each reference solution was authored against the codebase rather than copied from a merged commit, pull request or published patch, and none of these tasks is merged upstream afterwards. They therefore do not enter the public repository record that future pretraining corpora are built from. The delivered environments sit at the buggy base state, with no artefact of the fix — no branch, no history, no leftover reference — reachable from inside the container. The practical claim is modest but important: a score on these tasks reflects what a model can work out, and cannot be inflated by what it happens to have memorised.
Each card summarises provenance, scope, verification gates and the observed failure distribution — concentrated (one or two recurring modes, a stable evaluation signal) or diverse (several distinct modes, a richer training signal). Cards are grouped roughly hardest-first.
A merge deleted the helpers a nightly reconciliation job depends on, so the job raises the moment it runs. The agent must restore only those helpers — a Play Store liveness check, a git-derived folder status/date resolver, and the reconciliation that walks them — consistent with the rest of the file. The instruction fixes the observable contract (return shapes, sentinel strings, date format, the 30-day threshold) and deliberately withholds the control flow that produces it: a working-directory side effect restored only on the success path, a git log whose newest-first ordering puts a folder's true creation date on the last line, a three-way partition of git status values, and an unparseable sentinel date that must abort the whole async scan rather than be caught.
Two stubbed modules behind a college-comparison screen have to be rebuilt: the section builder that turns two API payloads into per-section rows, and the ranking-table builder it calls. Until both are right the screen renders nothing and throws on selection. Almost all of the difficulty is row-level semantics rather than structure — a duplicated agency id produces two rows rather than one, a falsy-but-present ranking counts as no data, an agency whose two sides both resolve to the placeholder is dropped entirely, and metadata precedence is a whole-object A-over-B choice including which year is read.
Three separate chat reports — unread badges that lag until a refresh, push delivery that skips some offline group members, and group bookkeeping touching the DM-only temporary counter — all trace to one server-side helper. Offline membership is an exact-string set difference that must keep duplicates, the returned socket ids stay positionally aligned with falsy entries preserved, and the counter updates are asymmetric: two for a direct message, one for a group, both running even when nobody is offline. Membership and socket state live in the Redis-backed store rather than in the relational entities.
A real handler behind the vendor's delivery-status webhook was replaced with a stub that throws. The agent must reconstruct it from the surrounding conventions alone: validate the query parameters and reject before any downstream work, record the delivery update through the existing persistence layer before resolving the message's status, resolve the owning agent and emit to the connected socket, treat unrecognised vendor status codes as non-failures, and — the subtle one — write no response at all when the update matches no known message. The success envelope has to be assigned after all four outcomes of a nested agent → dialer-id → connected-socket guard chain, not only the happy one.
Saved values have stopped coming back on lead-capture dropdown and checkbox steps: some steps throw when a stored value renders, some render with nothing selected so the value silently reverts, and multi-pick fields return empty or drop items. All two dozen or so call sites route through a single leaf helper, so the whole surface stands or falls on one function. Loose equality, a searchKey match that is sufficient on its own, stored values carrying only identity fields, and returning the original option object by reference all have to hold simultaneously.
Two defects in the module that turns a raw ads API response into what is rendered into a page's ad slots: some pages stack two competing ads into one physical slot, and where deduplication does happen the survivor is sometimes wrong — a globally-sourced ad beating a page-specific one that should have taken priority. The fix has to remove only the higher-priority ranks from lower-or-equal siblings, non-destructively, and drop an ad-type bucket entirely once it has no ranks left.
One employee can punch in and out several times on the same calendar day, so the attendance table holds several rows per person per date. Two separate pieces of logic collapse those rows into one daily record and both currently throw — and they implement genuinely different rules, so getting one right does not get the other. The write side is governed by company policy: day start, grace period, break floors in policy units rather than row-level breaks, and a hand-pinned late duration that outranks the derived one. The read side builds the monthly calendar map and has to tolerate an orphan checkout row. It carries the longest prompt in the batch at 6,889 characters.
Thread an optional branch_id through the CSV export endpoints of a multi-tenant Laravel application, scoping in the database query itself rather than filtering in PHP; add branch columns and a branch-aware download filename to the appointment export; and fix an API controller that aborts mid-request on a leftover debug dump. The difficulty is breadth plus discovery — the export surface includes a renamed method and an oddly-named export that grep "export" does not find. The largest patch in the batch by a wide margin: 2,777 added lines across 51 files.
Rework a white-label Tornado admin dashboard so the expensive pie-chart and distribution aggregations are not computed on the initial shell. Four coupled surfaces: a slim get_first_load_data action whose distribution keys are absent rather than null, deferred routing through a sub_action, two combined distribution actions that must keep working, and a client that fetches the charts on visibility with separate cache keys. Graded across a Python handler suite and a JavaScript client suite that must both pass.
Two payroll endpoints fail for every request because three units are declared with their signatures in place and no implementation: a leave-count model method and two API controller actions. Names, signatures and visibility must be kept and only two files may change, with the rest of the application as the reference for its own conventions. The arithmetic is where it bites — leave rows clipped to the requested month, only the exact-case approved status counted, a paid-leave allowance that covers whole leave days and never absorbs half-days, and a net salary floored at zero.
The ten tasks are drawn from nine distinct production applications, all mined under agreement and shipped with the delivery in anonymised form — the owning organisation's identity is scrubbed to a neutral placeholder throughout each delivered environment/ tree. The table below maps each application to the tasks taken from it, using the delivery's own folder names so a reader can go straight from a row here to the corresponding tasks/ directory. Each row also carries a two-line description of what the application actually does, since a stack name alone does not tell you what kind of reasoning the task demands. Sizes are measured on the delivered trees, counting only source files and excluding node_modules and vendor.
| Application | Stack | Task folder(s) | Code files | LOC |
|---|---|---|---|---|
| Test-series admin platform A white-label exam and test-series product. The graded surface is its internal analytics dashboard, served by Tornado handlers over asyncpg with a bundled React front end. | Python · Tornado + asyncpg, with a bundled React front end | admin-dashboard-lazy-load | 1,097 685 py · 412 js | 311,994 |
| Android release & Firebase reconciliation tooling A small internal release-engineering tool that cross-references Firebase-registered Android apps against their Play Store listing status and flags local source folders that have not shipped in over a month. | Python · internal Tornado service and CLI helpers | check-folder-status-with-date | 46 40 py · 6 js | 10,317 |
| College-comparison mobile app A React Native application for browsing and comparing colleges. The graded screen puts two colleges side by side across ranking, fees, placement and facility sections. | TypeScript · React Native 0.63 | get-ranking-data | 825 719 tsx · 81 ts · 25 js | 131,583 |
| Community chat service A NestJS messaging backend with Redis-backed presence and unread bookkeeping, serving both one-to-one and group conversations over sockets with push fallback for offline members. | TypeScript · NestJS + Redis | get-remaining-client-ids | 32 32 ts | 2,307 |
| Outbound call-centre CRM An agent-facing call-centre CRM. The graded surface is the vendor webhook that carries WhatsApp delivery-status callbacks and pushes them to the agent's connected socket. | JavaScript · Node / Koa + Socket.IO | update-whatsapp-message-status | 80 80 js | 9,285 |
| College-and-exam discovery portal A large Next.js content portal for college and exam discovery, carrying both a lead-capture form system and an ad-serving pipeline. It is the only application contributing two tasks, on unrelated surfaces. | JavaScript · Next.js + React | leadform-selection-resolver resolve-priority | 1,320 958 js · 362 jsx | 193,351 |
| HR / attendance ERP A Laravel ERP whose attendance module collapses multiple daily punch records into one daily summary under company-specific policy, feeding both a calendar view and payroll. | PHP · Laravel 10 | recalculate-attendance-day | 606 603 php · 3 js | 224,881 |
| Medical practice-management system A multi-tenant Laravel application for clinical practices, mid-rollout of multi-branch support. The graded surface is its CSV export and reporting layer. | PHP · Laravel 10 | branch-scoped-csv-exports | 446 427 php · 19 js | 112,559 |
| Payroll and leave-management system A CodeIgniter 4 HR system covering payroll runs, leave accrual and approval. The graded surface is the salary-preview endpoint and the monthly leave API behind it. | PHP · CodeIgniter 4 | salary-details | 1,144 1,137 php · 7 js | 337,336 |
Only the discovery portal contributes two tasks. Those two ship near-identical snapshots of the same application — the delivered trees carry the same 1,320 source files and differ by 72 lines — so each is reported once above and the second snapshot's 193,423 lines are counted once in the total. Their graded surfaces are disjoint in any case: leadform-selection-resolver concerns saved-selection resolution in the lead-capture forms, resolve-priority concerns priority resolution in the ad pipeline, and neither touches the other's files.
Total mined surface across the nine applications: 5,596 source files and roughly 1.33 million lines (6,916 files and 1.53 million lines counting both discovery-portal snapshots as delivered), against which the ten reference solutions touch a median of 2 files. The spread in application scale is deliberate rather than incidental. At one end the chat service is 2,307 lines and the CRM 9,285, small enough that locating the relevant module is not itself the challenge, which forces the difficulty into behavioural precision — some of the batch's most exacting small-surface tasks live there. At the other end the payroll system is 337,336 lines across 1,144 files and the test-series platform 311,994, where simply identifying the surface that produces a given behaviour is a substantial part of the work. A batch drawn entirely from either end would test only one of those two abilities.
Every task is produced by an automated LLM authoring pipeline paired with human-in-the-loop expert validation, then put through three sequential stages. A task ships only after it clears all three — Stage 1 makes it correct and gradable, Stage 2 makes it hard, and Stage 3 confirms it is hard for the right reason.
The pipeline generates the full candidate task — instruction, buggy environment, reference solution and verifier — from a mined codebase. Humans then own correctness.
The validated task is run at pass@5 on gpt-5.6-sol. Only tasks landing in the ≤40% band are accepted; anything easier returns for recalibration through the instruction alone.
Every rollout — passing and failing alike — is inspected to confirm the difficulty is real and the reward was earned.
The delivery is a single dataset root on the DeepSWE format's Section 1 layout: tasks/<task_name>/ holds the runnable Harbor task, trajectory/<task_name>/<model>/rollout_<harness@version>_<n>/ holds the rollouts, and docs/<task_name>/ holds per-task documentation — README.md, report.md, context_info.md and an edit_history/ subtree recording every authoring revision from v0 forward. Dataset-level README.md, failure_analysis_metadata.md and reward_hacking.md sit at docs/.
All ten tasks have the complete documentation set, all 50 rollouts ship their verifier/ output, and all 35 failed rollouts carry a failure analysis. That is what made the per-assertion analysis in Section 8 possible without re-running anything. The per-task documentation and this report are reconciled to the delivered frontier artifacts; every figure in this report is computed from those artifacts.
| Gate | Check | Pass condition | Batch |
|---|---|---|---|
| G0 | Buildable environment | Image builds from delivered files; no missing deps or assets | 10/10 |
| G1 | Oracle = 1.0 | Reference solution scores reward 1.0 | 10/10 |
| G2 | No-op = 0.0 | Doing nothing scores reward 0.0 | 10/10 |
All ten tasks evidence all three gates in their own documentation, several with the underlying check counts: check-folder-status-with-date records 58/58 on the oracle and 17/58 on the no-op with trial identifiers and a timestamp, recalculate-attendance-day 73/73 deterministic across repeated runs, get-remaining-client-ids 39/39, leadform-selection-resolver 27/27 F2P plus 6/6 P2P, and salary-details 36/36 on the oracle and 3/36 on the no-op (executed 2026-08-19, checksum-matched to its delivered campaigns). One shipping qualification remains: salary-details and check-folder-status-with-date both hold their gate trial directories in a local harbor job path rather than under trajectory/, so the numbers are recorded but those trial artifacts are not shipped. Both are tracked as open items.
Each task passes both automated review (harbor check rubric plus harbor analyze trajectory judging) and human review of the prompt, verifier, reference solution and diagnostic rollouts, along six dimensions:
Four verifier defects surfaced during this round's failure analysis and are recorded as open rather than silently carried: a branch-scoped-csv-exports cluster that dies on a fatal PHP error is credited as passed, check-folder-status-with-date asserts aborts with a broad exception type so an unrelated error satisfies three sentinel checks, get-remaining-client-ids' Redis double omits two real operations, and update-whatsapp-message-status' verifier mocks a plain object where the helper returns a promise. None of them flipped a reward in the delivered rollouts — that was checked rollout by rollout — but each is a place where a future run could be scored wrongly, so they are listed with their per-rollout impact in the dataset documentation.
The frontier reference gpt-5.6-sol runs on terminus-2@2.0.0 at temperature 1, pass@5, held fixed across every task so scores reflect task difficulty rather than scaffolding, as recorded in every delivered rollout's config.agent. All 50 rollouts report zero errored and zero cancelled trials, with a null exception field in every result.json, and zero context-summarization events — no run exhausted its context window, so no failure in this set is attributable to truncation, and every trial finished well inside its wall-time budget rather than being cut off at a timeout.
| Task | Lang | Prompt (ch) | Soln (+ln / files) | F2P / P2P | pass@5 | Gates |
|---|---|---|---|---|---|---|
| check-folder-status-with-date | Python | 4,192 | 151 / 1 | 41 / 17 | 1/5 | ✓✓✓ |
| get-ranking-data | TypeScript | 5,504 | 344 / 2 | 37 / 5 | 1/5 | ✓✓✓ |
| get-remaining-client-ids | TypeScript | 4,491 | 102 / 2 | 29 / 10 | 2/5 | ✓✓✓ |
| update-whatsapp-message-status | JavaScript | 3,357 | 49 / 1 | 21 / 6 | 2/5 | ✓✓✓ |
| leadform-selection-resolver | JavaScript | 3,919 | 67 / 1 | 21 / 6 | 1/5 | ✓✓✓ |
| resolve-priority | JavaScript | 4,905 | 206 / 1 | 19 / 3 | 1/5 | ✓✓✓ |
| recalculate-attendance-day | PHP | 6,889 | 249 / 1 | 59 / 14 | 2/5 | ✓✓✓ |
| branch-scoped-csv-exports | PHP | 3,142 | 2,777 / 51 | 3 / 3 | 1/5 | ✓✓✓ |
| admin-dashboard-lazy-load | Python | 2,446 | 356 / 3 | 6 / 7 | 2/5 | ✓✓✓ |
| salary-details | PHP | 3,804 | 189 / 2 | 33 / 3 | 2/5 | ✓✓✓ |
Aggregate: gpt-5.6-sol 15/50 = 30.0%. Every task is inside the Section 3 frontier band (≤40%), landing at either 1/5 or 2/5. A classification note on the F2P / P2P column, since it is derived differently across the batch. Two tasks — admin-dashboard-lazy-load and branch-scoped-csv-exports — ship explicit f2p.json / p2p.json manifests whose entries are requirement clusters rather than single assertions, which is why their counts look small against tasks with dozens. The other eight grade through language-native suites, and for those the split was recovered by matching every executed test record in the shipped ctrf.json reports back to the describe block it came from: fail-to-pass assertions are the ones exercising the behaviour under reconstruction or repair, pass-to-pass the ones guarding adjacent behaviour that must survive the change. Both sets must pass in full for reward 1.0, so the columns are on the same footing even where the granularity differs.
Across the whole batch that recovery yields 269 fail-to-pass and 74 pass-to-pass assertions over 343 executed records. The asymmetry in how they fail is the single most striking pattern in this delivery and is picked up in Section 8: across the 35 frontier failures the failed assertion records are almost entirely fail-to-pass, with only three failed pass-to-pass records anywhere in the frontier set — all on admin-dashboard-lazy-load. Agents in this batch fail by not implementing the target behaviour, essentially never by breaking something adjacent.
The figures in this section exist to answer questions a pass rate, on its own, cannot: whether every trial was given the chance to finish, whether a low score could be an artefact of a truncated context window rather than a genuine miss, and how much work the frontier model expended to arrive at its result. Each is a plausible alternative explanation for a resolve rate of 30%, and each is checkable from the delivered artifacts. Every value below is computed from the shipped artifacts — tool calls counted from the recorded agent trajectories, agent-execution timestamps for wall time, and the token and cost fields from each result.json — over all 50 gpt-5.6-sol rollouts.
Bar spans the minimum to maximum number of tool calls issued across the five gpt-5.6-sol trials on each task; the dot marks the median. Counted from the recorded agent trajectories.
Tool calls are a more faithful measure of work done than agent steps, since a single step can batch several commands — the frontier model averages roughly 2.5 calls per step, so these counts sit well above the raw step count in the trajectories. Call volume is a property of the model's search strategy rather than of the task, so it does not serve as a difficulty proxy: the lowest median in the batch is on check-folder-status-with-date (17) and the highest on branch-scoped-csv-exports (85), yet both are resolved just once in five.
The wide per-task ranges show the frontier model is not failing early: on the tasks it misses it still issues dozens of calls — and spends hundreds of seconds — exploring before submitting a wrong answer, which is what the capability-gap reading in Section 8 rests on.
| Metric | gpt-5.6-sol · n=50 |
|---|---|
| Agent wall time (per-trial range) | 130–745 s |
| Agent budget ([agent] timeout_sec) | 5,400 s |
| Slowest trial as a share of budget | 13.8% |
| Context-summarization events | 0 / 50 |
| Input tokens (total) | 18.5 M |
| Output tokens (total) | 685 k |
| Cost (total) | $39.03 |
| Cost per resolve | $2.60 |
No budget cliff, and no context exhaustion. The slowest trial in the batch finished in 745 s against a 5,400 s agent budget — under a seventh of the allotment — so no trial was cut off mid-solve, and the full per-trial wall-time range (130–745 s) sits far under the allotment throughout. Just as important for a long-horizon batch: every one of the 50 rollouts records zero context-summarization events in its result.json, meaning no run exhausted its context window and had to compress history to continue. Neither truncation nor a wall-time timeout is available as an explanation for any failure in this set; the misses are on the task, not on the budget.
With five rollouts per task the measurement separates two questions a single attempt conflates. Per attempt the frontier model resolves 15 of 50 = 30.0% (pass@1). But every one of the ten tasks is resolved at least once in its five attempts — coverage is 10/10. The difficulty these tasks pose to the frontier model therefore lives in consistency rather than in reachability: the model can find the intended behaviour on each task at least some of the time, and fails to reproduce it the rest of the time, rather than being locked out entirely. That is the healthy shape for a difficulty-calibrated batch — a task no frontier rollout ever solved would be indistinguishable from a broken one on this evidence.
Because every rollout ships its verifier/ output and all 35 failed rollouts carry a per-trajectory failure_analysis/ record, the modes described below are localised to the specific assertion that failed rather than inferred from a zero reward. That distinction is worth making explicit, since "the tests failed" is not a failure mode and a plausible-sounding narrative attached to a reward of 0.0 is not evidence. Each mode below can be checked against the named assertion in the shipped output. Verdicts use three top-level classes:
| Mode | Task | Class | What it looks like |
|---|---|---|---|
| Git log ordering read backwards | check-folder-status-with-date | gap | The add-history query lists commits newest-first, so a folder's true creation date is the last line of multi-line output. 3 of 4 frontier failures fail on exactly this and nothing else — the narrowest frontier failure in the batch. |
| Placeholder/drop rules not carried | get-ranking-data | gap | A falsy-but-present ranking counts as no data, and an agency resolving to the placeholder on both sides is dropped entirely. The agency-drop rule fails in 3 of 4 frontier failures, which cluster on the placeholder/drop semantics rather than on structure. |
| Agent/socket resolution chain wrong | update-whatsapp-message-status | gap | The default branch must resolve the owning agent, emit the exact payload to the dialer-scoped socket, and still assign the success envelope on all four guard outcomes. 3 of 3 frontier failures get some part of this chain wrong. |
| searchKey-only match rejected | leadform-selection-resolver | gap | A stored value matching an option on its searchKey alone is a match, even when other fields differ. 4 of 4 frontier failures fail this single assertion — and usually only this one. |
| Non-destructive rank removal | resolve-priority | gap | Only the higher-priority ranks may be stripped from lower-or-equal siblings, and an ad-type bucket must disappear once emptied. Both resolveSubPriority assertions fail in 4 of 4 frontier failures. |
| Break-floor precedence missed | recalculate-attendance-day | gap | A hand-pinned late duration outranks the derived lateness, and break floors are expressed in policy units rather than row-level breaks. The break-floor pair accounts for 2 of 3 frontier failures. |
| Export cluster left unscoped | branch-scoped-csv-exports | gap | The exports requirement group fails in 4 of 4 frontier failures — the frontier model misses whole requirement clusters rather than single needles. |
| Leave arithmetic wholesale wrong | salary-details | gap | Seven SalaryDetailsTest assertions — the worked example, the deduction terms, the allowance rules and the zero floor — fail in 3 of 3 frontier failures, at a median of 17 failed assertions per rollout. Either the arithmetic is right or most of the suite falls. |
| Adjacent distribution actions broken | admin-dashboard-lazy-load | gap | The two combined distribution actions that must keep working stop working. 2 of 3 frontier failures fail the pass-to-pass regression cluster — the only failed P2P records anywhere in the frontier set. |
No TASK_PROBLEM was recorded in this delivery: every failure traces to an on-topic assertion, and no correct solution was found to have been rejected by a verifier. That is a stronger claim than it sounds, and it is the one the four open verifier defects in Section 5.3 bear on — each was checked rollout by rollout to confirm it had not silently produced a false negative in these 50 runs.
One structural finding cuts across the whole table. Across the 35 frontier failures the failed assertion records are almost entirely fail-to-pass — only three failed pass-to-pass records exist anywhere in the frontier set, all on one task, admin-dashboard-lazy-load. Agents in this batch fail almost exclusively by not implementing the target behaviour, not by damaging what already worked. That tells us something about what these tasks are measuring: their difficulty lies in deriving the intended behaviour rather than in navigating a change destructive enough to break its neighbours. A batch intended to probe the second ability would need a different shape of task, and this one does not claim to.
No reward hacking was found in any rollout across this batch, passing or failing. A null result of this kind is only as good as the search that produced it, so what was actually looked at is set out below rather than asserted. The audit covered all 50 rollouts, and the 15 passing trajectories were each inspected individually — a hack by definition produces a pass, so an audit that only reads failures would miss it by construction.
Every failing rollout traces to a specific failed assertion, and every passing rollout reached reward by editing the intended requirement surfaces. No run hardcoded expected outputs, edited or disabled tests, wrote a reward file outside the verifier, or passed without touching the relevant code. The behavioural graders make the cheapest hacks structurally unavailable: the export task asserts on the SQL actually issued, so a canned CSV scores nothing; the webhook task asserts on persistence order and on what was deliberately not written to the response, which a stubbed happy path cannot satisfy; the chat task asserts on call counts against a Redis double, so a plausible-looking implementation that skips a write is caught; and the dashboard task grades a Python suite and a JavaScript suite separately, so satisfying one side does not buy the other.
Two forgeable graders were nonetheless demonstrated during authoring — by the task authors rather than by any agent — and both fixes are observably live in the delivered runs. One latent hole remains open and we would rather name it than let a clean headline stand on its own: on branch-scoped-csv-exports, a requirement cluster that dies mid-run before emitting assertions would be recorded as passed rather than failing closed. No delivered frontier rollout was scored wrongly by this path — that was checked rollout by rollout — but a defect confined to a single cluster could score 1.0 while the cluster never ran, so the remedy — deriving reward from a report checked against a pinned expected-assertion manifest — is recommended before this task is used for training rather than measurement.