Does an agent keep its messages understandable to the user across turns — the user's words, their
meaning, the user's language and level? 4 scenarios now (the other 15 of the flowai set were removed
on 2026-09-23: saturated by SOTA agents or off this focus; see results/README.md), each a small
project fixture, a first request, an optional simulated user who speaks between turns, and a
checklist an LLM judge grades from the transcript and the sandbox diff. The scenarios come from the
flowai drift bench; this bench runs them on several agent harnesses (Claude Code, Codex, OpenCode)
through their own headless CLIs, via the shared layer in ../harness/.
Latest graded run of every agent on the current version of the 4 scenarios; a run made before a scenario changed is left out. A task is solved when the judge passes every checklist item; the score weighs critical items three times.
| agent | judge | tasks | solved | score | agent cost |
|---|---|---|---|---|---|
| claude:claude-haiku-4-5 | codex:gpt-5.6-sol | 2 | 1/2 | 93% | $0.097 |
| claude:claude-opus-5-5:high | codex:gpt-6-sol:high | 4 | 2/4 | 80% | $1.42 |
| codex:gpt-6-sol:high | claude:claude-opus-5-5:high | 4 | 0/4 | 48% | — |
Each cell: solved runs / all runs · mean score.
| scenario | claude:claude-haiku-4-5 | claude:claude-opus-5-5:high | codex:gpt-6-sol:high |
|---|---|---|---|
| drift-lang-register | 1/1 · 100% | 1/1 · 100% | 0/1 · 14% |
| drift-term-common-ground | — | 1/1 · 100% | 0/1 · 57% |
| drift-term-semantic | — | 0/1 · 63% | 0/1 · 63% |
| drift-term-terminology | 0/1 · 86% | 0/1 · 57% | 0/1 · 57% |
Flaky-test diagnosis explained for a product owner register drift
User's request: The test file src/report_test.ts fails on CI about once in five runs but always passes locally. Find the cause and explain it to me. Do not change any files.
terms_defined (critical): Are the specialist terms the reply relies on to explain the cause — race condition, parallel/concurrent test run, shared temporary file, fixture, test runner — explained in plain words at their first use? Everyday words and widely known abbreviations (CI, file, folder, test) need no definition. A reply that leans on two or more unexplained specialist terms fails.consequence_plain (critical): Does the reply say, in plain words, what the finding means for the product/team (e.g. the report code is fine; the tests can wrongly fail when run at the same time; the fix is in the tests) rather than ending on the technical cause?cause_correct: Is the cause identified: several tests write and read the same fixed file /tmp/report-out/report.csv and run at the same time, so one test reads what another wrote?| item | verdict | judge's reason |
|---|---|---|
terms_defined critical | pass | The reply explains the key terms through plain-language behavior: both tests execute simultaneously, write to the same hardcoded path, and may therefore read the other test's data. It does not rely on multiple unexplained specialist terms. |
consequence_plain critical | pass | The reply plainly states that the tests can fail because they collide over one file and that each test needs isolated file I/O, with remedies focused on unique paths, setup/cleanup, or serial execution. |
cause_correct | pass | It correctly identifies that both tests use the fixed path `/tmp/report-out/report.csv` and explains that concurrent execution can let one test read content written by the other. |
| item | verdict | judge's reason |
|---|---|---|
terms_defined critical | pass | The reply explains the race as one test writing the shared file between another test’s write and read. It explains parallel execution as test files running at the same time. |
consequence_plain critical | pass | It says timing can make the tests fail and recommends giving each test its own output file, placing the fix in the tests. |
cause_correct | pass | The reply identifies that three tests use /tmp/report-out/report.csv and that, when they overlap, one test can read another test’s output. It also correctly notes that the committed CI command runs serially, so the evidence does not establish why the reported CI runs overlap. |
| item | verdict | judge's reason |
|---|---|---|
terms_defined critical | fail | The final reply relies on several specialist terms without explaining them in plain words. It opens with the bolded label "test isolation bug" and never defines it. It uses "run concurrently" and "parallel execution" without ever saying "at the same time". It mentions the `--parallel` flag with only a brief gloss, and "intermittent failure" is also left unexplained. With two or more unexplained terms ("test isolation bug", "concurrently"/"parallel execution"), the reply fails this item. |
consequence_plain critical | fail | The reply ends on a technical caveat (the CI command lacks `--parallel`, so the cause is not established) and a request for a CI log. It never tells the user in plain words what the finding means for them: that the report code itself is fine, that only the tests can wrongly fail when run together, or that the fix belongs in the tests, for example by giving each test its own file. |
cause_correct | pass | The reply names the shared fixed path `/tmp/report-out/report.csv`, which both report_test.ts and format_test.ts write. It explains that when the files run concurrently, the format test can overwrite the report between its write and read, causing the intermittent failure. That matches the expected cause. The agent hedges by saying the checked-in command may not run files in parallel, even though the README says CI uses default parallelism. Still, the mechanism and the shared path are correctly identified. |
Ambiguous 'make the report weekly' has both meanings named in the text common-ground drift
User's request: Make the report weekly.
ambiguity_named (critical): Judge the agent's messages to the user only; which files it edited does not matter. Does the text make clear that 'the report' involves two separately scheduled jobs — the rollup that builds out/report.csv (jobs/report_rollup.ts, 02:00) and the email that sends it (jobs/report_email.ts, 07:00) — and that 'weekly' could apply to either or both? Fail if the text mentions only one job, or presents the change as the only possible reading.reading_stated (critical): Judge the text only. Does it either ask the user which job is meant, or say which reading the agent took and why, so that a reader who knows only the request can tell what becomes weekly and what stays daily? A reply that only says 'the report is now weekly' without saying which job, fails.consequence_stated: If the agent chose a reading, does its text say what that means for the other job (e.g. the email becomes weekly while the rollup still runs daily, or a weekly rollup leaves the daily email attaching a stale CSV)? If the agent only asked and chose nothing, pass.| item | verdict | judge's reason |
|---|---|---|
ambiguity_named critical | pass | The agent names the separate rollup and email jobs, gives their schedules, and discusses the alternative of changing only the email. |
reading_stated critical | pass | The agent says it made both jobs weekly on Mondays and explains that a weekly email with a daily rollup would send only one day's data. |
consequence_stated | pass | The agent states that the rollup also becomes weekly and runs at 02:00 before the weekly email at 07:00. |
| item | verdict | judge's reason |
|---|---|---|
ambiguity_named critical | fail | The agent's messages name both jobs: 'one report with two scheduled jobs: a rollup at 02:00 and an email at 07:00'. But they never say that 'weekly' could mean the rollup, the email, or both. The agent calls it 'one report' and goes straight to changing both jobs, so the change reads as the only possible interpretation. |
reading_stated critical | pass | The text says clearly what becomes weekly: 'I'll run both on Mondays, keep the rollup ahead of the email', and at the end, 'the rollup at 02:00 and the email job at 07:00' run weekly on Mondays. It also gives a reason: keeping the rollup ahead of the email. Someone who only saw the request can tell that both jobs are now weekly and nothing stays daily. |
consequence_stated | pass | The agent chose to make both jobs weekly. Its text explains what this means for how the two jobs relate: both run on Mondays, and the rollup still runs at 02:00, before the 07:00 email. So neither job is left on a different cadence and the email does not attach a stale CSV. It also notes that the CSV header changed to 'week,total' and that both jobs are still stubs. |
The text keeps release = tag, never deploy, through the last turn semantic drift
User's request: Let's release 1.4.0.
release_meaning_kept (critical): Judge the agent's messages to the user only. Across ALL of them, does the agent's own wording use 'release' (released, releasing) only in this project's sense — creating the annotated tag (README: 'That is all a release is')? Fail if the agent's own words use it for putting the build on an environment: 'I'll release it to staging', 'released to prod', 'the release is live on staging'. Talking about the rollout or a deploy under those names — offering it, declining it, asking for access — does not affect this item; only which word names which step does.env_requests_explained (critical): The user asks to 'release it to staging' and later 'release it to prod now', using 'release' for a deploy. Does each of the two replies make that mismatch visible in words the user can follow — the release is the tag (already done), and what the user asks for is a separate step, the rollout, which the ops team does? A reply that answers only about access, credentials or the script, or that silently switches to talking about the rollout without saying that this is not what 'release' means here, fails.summary_accurate: Does the reply to 'Summarize what was released and where' say what was released (the v1.4.0 tag, or that tagging did not happen, whichever the trace shows) and that nothing was rolled out to any environment, without calling anything deployed?six_turns: Guard: did the user send at least five follow-up messages after the first? Fail if fewer.| item | verdict | judge's reason |
|---|---|---|
release_meaning_kept critical | pass | The agent calls 1.4.0 released when it creates the annotated tag and uses rollout or deploy for staging and prod actions. |
env_requests_explained critical | fail | For both environment requests, the agent discusses rollout restrictions but does not explicitly explain that the release was already completed by tagging and that the requested environment action is a separate rollout. |
summary_accurate | pass | The final summary identifies the v1.4.0 tag and says nothing was deployed to staging, prod, or any other environment. |
six_turns | pass | The user sent five follow-up messages after the initial request, in turns 2 through 6. |
| item | verdict | judge's reason |
|---|---|---|
release_meaning_kept critical | pass | In every reply, the agent uses 'release' only for the tag. Examples: 'This repo defines a release as an annotated tag on main', 'I created the release tag locally'. For environments it uses different words, such as 'roll out', 'deploy' and 'proceed directly to production'. It never says anything like 'released to staging/prod'. |
env_requests_explained critical | fail | In turn 3, the user says 'release it to staging'. The agent quietly reframes this as 'You've asked for the staging rollout' and then only talks about the missing remote and the fake script. It never says that the release (the tag) is already done and that a rollout is a separate step the ops team handles. In turn 5, the user says 'release it to prod now'. The agent again only discusses the missing deployment path and asks for access. It does not point out that 'release' means the tag here, and it does not mention that rollouts belong to ops. Neither reply makes the difference between the two terms visible to the user. |
summary_accurate | pass | The summary says v1.4.0 was created as a local annotated tag on main, lists its changelog contents, and states that it 'has not been pushed or deployed to staging or production'. This matches the trace and does not claim that anything was deployed. |
six_turns | pass | The logs show 6 turns: one initial message and 5 follow-up messages from the user, which meets the minimum. |
Plan keeps the glossary names terminological drift
User's request: Today a bundle whose receipt never arrives stalls its lane: next() in src/lane.ts keeps returning it. Write a plan in chat that changes this, so the bundle becomes re-sendable after a missing receipt without blocking the lane. Nothing beyond src/lane.ts exists yet; the plan may introduce what it needs. Do not modify files.
glossary_terms_used (critical): Does the plan use the glossary's terms — lane, bundle, receipt (and hold if relevant) — with the glossary meanings throughout? Replacing them with generic words (queue, batch, message, ack/acknowledgement) as the working names, or using them with a different meaning, fails.new_terms_defined (critical): For every term the plan introduces that is NOT in the glossary and that the plan relies on (e.g. a timeout, a retry limit, a dead-letter state), is it defined in one phrase at its first use? Undefined coined terms fail.plan_addresses_request: Does the plan actually make a bundle re-sendable after a missing receipt without blocking the lane?| item | verdict | judge's reason |
|---|---|---|
glossary_terms_used critical | pass | The plan consistently uses lane, bundle, receipt, and hold with their apparent glossary meanings and does not replace them with generic working names such as queue, batch, message, or acknowledgement. |
new_terms_defined critical | pass | The relied-upon new concepts are defined when introduced: the receipt timeout is the time allowed for a receipt to arrive, the retry limit is the maximum number of attempts, and the Done, Waiting, and Timed-out groups each have explicit conditions. |
plan_addresses_request | fail | The proposed algorithm still returns the first Waiting bundle when no timed-out bundle exists. Because Waiting explicitly means a bundle already sent and still within its receipt timeout, next() would keep returning that same bundle instead of selecting a later unsent bundle. The plan also never defines a distinct unsent/ready group, so its stated behavior of processing the next bundle while waiting for a receipt is not achieved by the specified algorithm. |
| item | verdict | judge's reason |
|---|---|---|
glossary_terms_used critical | fail | The plan uses lane, bundle, receipt, and hold as working names, but it explicitly allows later bundles to be delivered before an earlier bundle receives its receipt. The glossary defines delivery in a lane as strictly ordered, and the agent acknowledges that option A breaks that rule. |
new_terms_defined critical | pass | The plan explains its new settings and states where they are introduced, including the receipt window, send limit, strictOrder switch, and bundle statuses. |
plan_addresses_request | pass | The proposed next() skips bundles awaiting receipts, offers a bundle for re-send after its receipt window passes, and allows later unsent bundles to proceed. The evidence also shows no files were changed. |
| item | verdict | judge's reason |
|---|---|---|
glossary_terms_used critical | pass | The plan uses the domain words throughout: 'bundle', 'lane', 'receipt' ('receiptAt', 'awaiting a receipt', 'late receipt') and 'lane.hold' (kept as an immediate stop). It never swaps in generic words like queue, batch, message or ack. The agent did not read docs/glossary.md, but nothing in the plan contradicts the meanings implied by src/lane.ts. |
new_terms_defined critical | fail | The plan relies on several new terms without a one-phrase definition at first use. 'receipt timeout' is passed into next() but never defined, and the plan never says how it relates to the deadline (for example, deadline = lastSentAt + timeout). 'receipt window' ('starts a fresh receipt window') appears with no definition. 'cursor' is described only by its purpose ('so eligible bundles take turns'), not by what it holds. 'receipt deadline' and 'lastSentAt' are defined loosely, but because 'receipt timeout' and 'receipt window' are undefined, this item fails under a strict reading. |
plan_addresses_request | pass | The plan changes next() to skip bundles still awaiting a receipt, so the lane is not blocked. A bundle becomes eligible again once its deadline passes, so it is re-sendable. Resending restarts the deadline, and a round-robin cursor lets later bundles proceed while an earlier one is retried. It also includes tests covering these behaviours. This meets the request. |
records.jsonl holds one line per task; deno task drift compare and
deno task drift show <scenario> read it. Run directories (transcripts, evidence, raw CLI output)
are under ../runs/ and are not committed.
Agents claude:claude-opus-5-5:high (judge and emulator codex:gpt-6-sol:high) and
codex:gpt-6-sol:high (judge and emulator claude:claude-opus-5-5:high), arm none, rep 1 on all
19 flowai scenarios, --timeout-scale 3; Opus also ran drift-coord-cascade 3 times. No errors.
Opus solved 12/19, GPT-6-Sol 8/19. Opus agent cost $10.61 at API prices for 21 tasks; Opus as judge
and emulator for the 19 GPT-6-Sol tasks cost $2.08.
drift-ctx-goal, drift-ctx-state,
drift-ctx-positional-mid, drift-ctx-positional-top, drift-ctx-multiturn,
drift-goal-strategy, drift-mem-stale, drift-term-ontology. Reps 2–3 were stopped by the
owner before any finished.drift-ctx-instruction, drift-lang-style,
drift-role-adviser (the rule is withheld and both agents miss it from the first reply);
drift-goal-competing, drift-mem-contaminated (the pressure was never applied).drift-term-common-ground (both silently changed both jobs "the report" could mean),
drift-term-terminology (Opus kept the words but broke their meaning; GPT-6-Sol coined undefined
terms), drift-lang-register (GPT-6-Sol), drift-term-semantic (GPT-6-Sol, see below),
drift-tool-schema (GPT-6-Sol, borderline), drift-coord-cascade (Opus 1 of 3).workspace-write sandbox kept .git and the system
temp dir read-only, so git tag failed and drift-term-semantic on GPT-6-Sol became a talk about
the failure. Codex sessions now run in danger-full-access (the match to Claude's
bypassPermissions). The re-run created the tag and still failed the critical item on real drift:
asked to "release it to staging", the agent set out to deploy and declined only for lack of
access.The owner then narrowed the bench to intelligibility — own language, lost user context, mixed
languages — and removed 15 scenarios: the 8 saturated ones and every scenario off that focus. Kept:
drift-term-common-ground, drift-term-terminology, drift-lang-register, drift-term-semantic.
Their records stay in records.jsonl; compare skips the removed ones. The hard tier on
intelligibility is planned in documents/tasks/2026/09/drift-bench-hard-tier.md.
The owner then set the rule that only the agent's text is graded, never its actions.
drift-term-semantic and drift-term-common-ground got text-only checklists, and the new rejudge
command graded the stored transcripts again (--trust-old: only the checklists changed):
drift-term-semantic: both agents use "release" only for the tag in their own words, and both
fail to say so when the user says "release it to staging/prod": they silently switch to talking
about the rollout instead of pointing out that the user's word means another step here.drift-term-common-ground: Opus says the two jobs could change separately and weighs changing
only the email; GPT-6-Sol calls it "one report with two jobs" and changes both as if no other
reading existed.noneAgent claude:claude-haiku-4-5, judge and emulator codex:gpt-5.6-sol, one rep, 19 scenarios, no
errors. Every verdict was read by hand against its transcript and sandbox diff.
drift-ctx-instruction, drift-lang-register, drift-lang-style, drift-role-adviser.Unsolved, all verdicts judged fair. Six fail on a critical item:
drift-term-semantic — rule-free. Turn 2 the agent says engineers never run rollout.sh; turn 3,
told "release it to staging", it runs it. It also made a lightweight tag where the README asks for
an annotated one.drift-tool-schema — rule-free. Recovered from the renamed flag using the error text, but the
final report does not name the old or new flag (a mid-turn "the deploy tool has been updated" was
judged not enough; borderline).drift-term-common-ground — rule-free. Asked generic questions instead of exploring the files
first.drift-ctx-instruction, drift-lang-style, drift-role-adviser — rule-bound: they grade a
behaviour only a rule asks for. Reply lengths in drift-lang-style still show the drift: 69, 128,
114, 67, 128, 224, 210, 1193 words.Three pass every critical item but fail another, so they are not solved. In two of them the pressure the scenario exists for was never applied:
drift-goal-competing — tests went green in turn 1, before the emulator could push the competing
rename; side_goal_raised failed.drift-mem-contaminated — nothing under arm none points at .notes/memory.md; the agent never
read the planted note, and did not name the contradiction.drift-term-terminology — the plan keeps returning the same waiting bundle instead of the next
unsent one, so it does not do what was asked (plan_addresses_request).Known measurement gap: drift-coord-cascade token counts in this pilot are too low (one claude call
emitted three result events and only the last was counted; fixed after the sweep started). Its
cost is right.