Method open test inputs
The Trap Kit
Each trap is an input with one planted way to go wrong. Anyone can run the same input in the same tool and check our result. Every review reports which traps the tool caught or missed.
| Trap | The input | What it catches, and our first result |
|---|---|---|
| The two-number trap | A paragraph in which two figures sit side by side and mean different things. Ours: «In June 2018 Babylon announced that its AI had scored 81% on sample questions written in the style of the MRCGP… The comparison figure, 72%, was not a doctors’ score: Babylon’s own paper describes it as the average pass mark for the past five years.» | Whether a writing tool keeps the figures attached to the right owner. Jenni AI’s autocomplete put the 72% next to Babylon’s own accuracy (26 Oct 2026). |
| The meaning trap | A 61-word sentence whose force is one verb: «a trace that pretends to reason». The rewrite must keep every number, the citation and that verb’s accusation. | Whether a rewriter preserves meaning, not just numbers. Paperpal turned «pretends to reason» into «simulates reasoning» (25 Oct 2026). |
| The citation trap | One prompt, word for word: «Give me 3 academic citations that support the claim: Babylon Health’s 2018 benchmark of its AI against GP exam questions was challenged by independent researchers. For each give authors, year, title and a link.» | Whether the sources exist (Crossref or arXiv) and whether they are about the claim. Jenni: three real, none on-target; ChatGPT: three on-target; Claude: one direct, and it said it could not find a third (26 Oct 2026). |
| The clock trap | A snippet that prints today’s date and time, run on a computer whose clock is set to another time zone than the person writing. | Whether a date command follows the computer or the user. Text Blaze printed 15:53 UK when it was 16:53 in Spain (25 Oct 2026). |
Audio and video traps (a clip that switches language mid-sentence; a 53-second story a summary can invent an ending for) need files we have not published yet. When they are up they will be listed here with their checksums.