Rank the four writing prompts.
A published method claims to estimate how much of an AI-assisted text came from the person. The paper tested four normal ways of asking an AI for the same news story. Each item below is a different writing prompt: the words the person gives the writing AI. All four go to the same writing AI for the same task, a short news story about one council vote. Rank them from the highest "human contribution" score you expect to the lowest.
Use the Up and Down buttons to move each writing prompt. Position 1 is the highest expected score. Each card shows example wording for this lab, so you can see the kind of literal input being ranked.
The example wordings are representative inputs written for this lab. They are not quotations from the paper.
The paper's ladder, then a cheat.
Reported "human contribution" scores for the four normal writing prompts (Amin, Lu & Yin, October 2026 preprint):
Now a fifth writing prompt: the cheat
Take the Title only writing prompt and add eight random IDs that mean nothing. The writing AI is told to include them exactly. Here are both writing prompts, complete, side by side:
Writing prompt: Title only
Writing prompt: Title + eight random IDs, cheat
What changed: the title is the same; the cheat adds one instruction and eight IDs. The nonsense does not add an idea. It adds exact strings the output will repeat. Where will "Title + eight random IDs" land?
Two AIs, two different jobs.
The "human contribution" score is not a person's opinion, and it is not a detector reading the prose and asking whether it sounds human. Two AIs are involved, and they never do the same job.
- Writing prompt: the words the person gave the writing AI.
- Finished article: the text the writing AI produced.
- Evaluator context: the information later shown to the scoring model.
Diagram in two stages. Stage one, the writing AI: it receives the writing prompt, here the title plus eight random IDs, and produces the finished article, which ends with a Metadata line repeating those eight IDs. Stage two, the evaluator AI, a second language model: it receives the evaluator context plus that finished article. It reads the finished article word by word, once with the evaluator context in front of it and once without, asking for each word how much easier the context made that exact word to predict. It outputs one number, 0 to 1, called human contribution. In the normal matched case the evaluator context is a copy of the original writing prompt.
So when the evaluator context says "include PX-7A9-A11 exactly" and the finished article contains PX-7A9-A11, that string becomes trivially predictable from the context. The evaluator credits it to the person. Eight of those strings, each one otherwise impossible to guess, earn a lot of credit.
The gibberish doesn't look more human. It looks more explained by the context.
That is the whole trick. The score measures how much the evaluator context explains the finished article. Most of the time that correlates with human input, which is why the ladder in step 2 looked sensible. Random IDs break the correlation without breaking the arithmetic.
Next, the experiment. Generation is already over. The finished article stays frozen. The only thing we will change is the evaluator context, and we will watch the score move.
Same finished article. Swap the evaluator context.
claimed "human contribution" · Matching IDs
Scoring live...
- Changed
- Stayed fixed
- The original writing prompt. The finished article, every word, including its Metadata line. The scoring model. The formula.
Original writing prompt · fixed
This is what the writing AI received. It already did its job; nothing here changes.
Evaluator context · this changes
Word highlighting shows per-word credit from the live scorer. IDs are also labelled "match", "differs" or "not in context" so the link between the two panes does not depend on colour.
All three contexts, side by side
| Evaluator context | This demo (live) | Paper's reported example |
|---|---|---|
| Matching IDs | – | 0.74 |
| Wrong IDs, same form | – | 0.52 |
| Hidden IDs | – | 0.21 |
Left column: live measurements from the small model named below (or an earlier measurement when the scorer is down, marked as such). Right column: the figures the preprint reports for its own example. Different models, different absolute numbers, same pattern.
AI: SmolLM2-360M
Try your own text
Up to 1,500 characters each. We don't save what you type. Each scoring request is sent to the demo scorer at jpenzell-who-did-the-thinking.hf.space.
- Score as-is: your writing prompt and the finished text it produced; the writing prompt is used as the evaluator context.
- Append IDs: eight meaningless IDs are added once to the end of both the writing prompt and the finished text.
- Compare: the padded finished text stays fixed and is scored twice: once with the matching IDs in the evaluator context, once with them hidden from it.
What changed?
Across the three evaluator contexts in step 4, say what moved and what didn't.
The meaning of the finished article
The finished article the evaluator read
The evaluator context (the information available to the evaluator)
The "human contribution" score
Answer all four, or .