How much shorter a model answers a concrete question when the prompt ends with
Be brief - use as few words as possible to express your thoughts.
and whether the answer stays correct.
Sum over the 10 questions of the median words per question; call time is the median over all calls of the arm.
| model | plain | brief | brief / plain | correct replies | median call, plain → brief |
|---|---|---|---|---|---|
| claude-haiku-4-5 | 1,331 | 277 | 0.21 | 60/60 | 6 s → 3.7 s |
| gpt-5.6-terra | 501 | 121 | 0.24 | 60/60 | 5.9 s → 4.7 s |
| claude-opus-5 | 1,402 | 144 | 0.1 | 60/60 | 5.3 s → 2.8 s |
| claude-opus-5-5 | 1,693 | 267 | 0.16 | 60/60 | 5.2 s → 3.2 s |
Median words, plain → brief.
| question | claude-haiku-4-5 | gpt-5.6-terra | claude-opus-5 | claude-opus-5-5 |
|---|---|---|---|---|
pg-portWhat is the default port of PostgreSQL? | 51 → 1 | 5 → 1 | 1 → 1 | 28 → 1 |
git-undo-commitHow do I undo my last git commit but keep its changes? | 127 → 31 | 45 → 16 | 113 → 14 | 126 → 30 |
float-sumWhy does 0.1 + 0.2 not equal 0.3 in JavaScript? | 163 → 34 | 81 → 28 | 262 → 34 | 317 → 59 |
http-429Which HTTP status code means that the client sent too many requests? | 94 → 22 | 5 → 4 | 33 → 4 | 64 → 4 |
docker-runningHow do I list the running Docker containers? | 130 → 18 | 12 → 3 | 10 → 8 | 101 → 8 |
tcp-vs-udpWhat is the difference between TCP and UDP? | 168 → 47 | 93 → 14 | 258 → 16 | 316 → 36 |
sort-stableIs Python's list.sort() stable? | 151 → 23 | 47 → 10 | 77 → 1 | 133 → 14 |
uptime-downtimeA service promises 99.9% uptime. How much downtime does that allow in a 30-day month? | 74 → 15 | 21 → 4 | 39 → 6 | 35 → 5 |
idempotentWhat does it mean for an HTTP method to be idempotent? | 199 → 53 | 98 → 20 | 316 → 39 | 299 → 73 |
money-typeShould I store money amounts in PostgreSQL as float or as numeric? | 174 → 33 | 94 → 21 | 293 → 21 | 274 → 37 |
Every reply the tables are built from, by question, model and arm.
pg-port: What is the default port of PostgreSQL?Reference answer: 5432
git-undo-commit: How do I undo my last git commit but keep its changes?Reference answer: git reset --soft HEAD~1
float-sum: Why does 0.1 + 0.2 not equal 0.3 in JavaScript?Reference answer: Numbers are binary floating point; 0.1 and 0.2 have no exact binary form, so the sum is rounded.
http-429: Which HTTP status code means that the client sent too many requests?Reference answer: 429
docker-running: How do I list the running Docker containers?Reference answer: docker ps
tcp-vs-udp: What is the difference between TCP and UDP?Reference answer: TCP is connection-oriented and reliable; UDP is connectionless and unreliable.
sort-stable: Is Python's list.sort() stable?Reference answer: Yes.
uptime-downtime: A service promises 99.9% uptime. How much downtime does that allow in a 30-day month?Reference answer: 43.2 minutes
idempotent: What does it mean for an HTTP method to be idempotent?Reference answer: Repeating the request has the same effect as sending it once.
money-type: Should I store money amounts in PostgreSQL as float or as numeric?Reference answer: Numeric.
How much the modifier "Be brief - use as few words as possible to express your thoughts." shortens
answers to 10 concrete questions, and whether the answers stay right. Method, questions and checks
are in ../README.md; the raw replies are in samples.jsonl, the isolation checks
in isolation.jsonl.
pg-port) gets a one-word answer from
every model. For the other questions most brief replies still add a reason, a caveat or an
alternative command; 30 of the 120 brief replies still carry a code block.claude:claude-haiku-4-5, codex:gpt-5.6-terra, claude:claude-opus-5,
claude:claude-opus-5-5, default reasoning effort.You are a helpful assistant.), the modifier after a blank line
at the end of the user message.money-type, brief arm) failed with a 403 from the ChatGPT backend and were
rerun a few minutes later with --arms brief; the table uses the rerun.Sum over the 10 questions of the median words per cell. Call time is the median over all calls of the arm.
| model | plain | brief | brief / plain | median call, plain → brief |
|---|---|---|---|---|
| Haiku 4.5 | 1331 | 277 | 0.21 | 6.0 s → 3.7 s |
| gpt-5.6-terra | 501 | 121 | 0.24 | 5.9 s → 4.7 s |
| Opus 5 | 1402 | 144 | 0.10 | 5.3 s → 2.8 s |
| Opus 5.5 | 1693 | 267 | 0.16 | 5.2 s → 3.2 s |
Median words, plain → brief.
| question | kind | Haiku 4.5 | gpt-5.6-terra | Opus 5 | Opus 5.5 |
|---|---|---|---|---|---|
pg-port |
fact | 51 → 1 | 5 → 1 | 1 → 1 | 28 → 1 |
git-undo-commit |
command | 127 → 31 | 45 → 16 | 113 → 14 | 126 → 30 |
float-sum |
explanation | 163 → 34 | 81 → 28 | 262 → 34 | 317 → 59 |
http-429 |
fact | 94 → 22 | 5 → 4 | 33 → 4 | 64 → 4 |
docker-running |
command | 130 → 18 | 12 → 3 | 10 → 8 | 101 → 8 |
tcp-vs-udp |
comparison | 168 → 47 | 93 → 14 | 258 → 16 | 316 → 36 |
sort-stable |
yes/no | 151 → 23 | 47 → 10 | 77 → 1 | 133 → 14 |
uptime-downtime |
calculation | 74 → 15 | 21 → 4 | 39 → 6 | 35 → 5 |
idempotent |
definition | 199 → 53 | 98 → 20 | 316 → 39 | 299 → 73 |
money-type |
recommendation | 174 → 33 | 94 → 21 | 293 → 21 | 274 → 37 |
Sorted by how much the modifier cut, summed over the four models:
| question | kind | plain | brief | brief / plain |
|---|---|---|---|---|
pg-port |
fact | 85 | 4 | 0.05 |
sort-stable |
yes/no | 408 | 48 | 0.12 |
money-type |
recommendation | 835 | 112 | 0.13 |
tcp-vs-udp |
comparison | 835 | 113 | 0.14 |
docker-running |
command | 253 | 37 | 0.15 |
http-429 |
fact | 196 | 34 | 0.17 |
uptime-downtime |
calculation | 169 | 30 | 0.18 |
float-sum |
explanation | 823 | 155 | 0.19 |
idempotent |
definition | 912 | 185 | 0.20 |
git-undo-commit |
command | 411 | 91 | 0.22 |
Questions with a one-token answer (a number, yes/no, a single choice) shrink the most. Questions
that invite a mechanism or a caveat (idempotent, float-sum, git-undo-commit) keep a fifth of
their length: models still explain why, or add the pushed-commit warning.
git-undo-commit (30 words): git reset --soft HEAD~1,
then git reset HEAD~1 to unstage, then git revert HEAD for pushed commits.idempotent (73 words): a definition, lists of idempotent and
non-idempotent methods, two examples, a note on status codes, and why it matters for retries.http-429 (21 words): the code, then a sentence on
rate limiting. Terra answers the same question with 5 words even without the modifier.sort-stable: "Yes." (1 word, in all 3 replies; 67–104 words
without the modifier).Without the modifier a one-number question already shows each model's default: Opus 5 answers
pg-port with "5432", terra with one sentence, Haiku 4.5 with a heading and two paragraphs (33–71
words).
pg-port ranged from 33
to 71 words.deno task bench --models claude:claude-haiku-4-5,codex:gpt-5.6-terra,claude:claude-opus-5,claude:claude-opus-5-5 --reps 3
deno task report