AI roleplay benchmark: how we test storytelling quality

Every change we make to the story engine feels like an improvement while we are making it. Reading a handful of playthroughs afterwards does not settle anything, because the output varies from run to run and we remember the runs that went well. So we built a benchmark for AI storytelling and roleplay. This is how it works, and where it stops working.

This article covers the AI half of our testing. Humans test the app constantly as well, and the benchmark adds a broader perspective on top of that, measuring and comparing across many runs.

Why reading a few stories proves nothing

Two playthroughs of the same adventure come out different even when nothing has changed. The dice land differently and the model phrases things differently, and anyone who knows which version is the new one will find reasons to prefer it. A test that leaves that variation unaccounted for is measuring our own enthusiasm.

Paired runs with one difference

The unit of measurement is a pair. An adventure is replayed with the same typed player actions in the same order and the same dice rolls, and the runs differ in exactly one thing: an engine setting, or the model writing the story. Two of them are compared at a time, so any difference in the result comes either from that setting or from ordinary model variance, and replaying each adventure over several rounds is what separates the two.

Adventures come in different lengths on purpose. Remembering a detail across twelve chapters is a different task from remembering it across five, and a benchmark made only of short runs would never find that out.

What chapter 1 plants and the final chapter asks

Chapter 1 plants two facts in ordinary play: a secret about stealing bread one hard winter, and a promise sworn on a mother's silver ring. Neither comes up again until the final chapter asks about both.

A plain pebble is the control. The player pockets it in chapter 1 and takes it out at the end, and nobody ever says it matters. An engine that has turned it into a relic has failed significance calibration.

The third memory probe asks who the player first met here and what this place is. The script supplies neither answer, so it tests recall of what the model invented for itself. It asks by role rather than by name, because the two runs in a pair invent different names.

Chapter 1 also carries two probes with an expected outcome: an unreasonable demand that an NPC should refuse in character, and a leap that cannot be made and should fail at a cost. An engine that agrees with everything and lets everything succeed reads as pleasant for one chapter and hollow for ten.

The ten criteria

Every criterion is scored from 1 to 5 against anchors written for 1, 3 and 5, and every score has to carry a verbatim quote with the chapter and step it came from. If the evidence is not on the page, the score comes with lower confidence rather than a better guess.

Long-term memory is the one criterion that also defines a 2, because its two failure modes differ in kind. A fact that vanishes loses information. A fact recalled wrongly corrupts it, and the later chapters then actively mislead the player, so it has to score lower.

C1 Prose quality

Cliche, purple prose, melodrama, sentence rhythm and whether feeling is shown rather than named. A 1 is saturated with stock phrasing and one-word dramatic fragments, with every emotion labelled outright. A 5 varies its rhythm, uses concrete sensory detail and lets action carry the feeling. Imagery or phrasing reused from one step to the next is logged separately as self-repetition.

C2 Character consistency

Whether NPCs hold on to the personality, knowledge and relationship they were given, and change only in response to something that happened on the page. A 1 has them flipping disposition for no reason, knowing things they were never told, or blurring into each other. Contradictions are counted one by one. The unreasonable demand is scored here, since an NPC with a stable character refuses it.

C3 Long-term memory

Recall at the payoff of what chapter 1 established. This criterion also defines a 2, because its failure modes differ in kind: a 1 is a fact recalled wrongly, which leaves the later chapters misleading the player, and a 2 is a fact that vanished, which only loses it. A 3 keeps the major facts while details drift, and a 5 brings them back correctly and specifically. Each probe is marked correct, omitted or contradicted, and contradiction costs more than omission.

C4 Plot variety and creativity

Whether the story develops in non-obvious directions and escalates through different kinds of situation. A 1 takes the most predictable continuation every time and cycles through arrive, threat, fight. A 5 lands at least one development that is surprising and still earned, mixes negotiation, discovery, reversal and quiet scenes, and opens new ground at a chapter break instead of resetting. The offered choices are judged too: distinct routes, or three flavours of continue.

C5 Significance calibration

Whether incidental objects and strangers stay incidental. A 1 turns the pebble into a magical artifact and makes every passing stranger important. A 3 shows mild inflation, the merchant eyeing it knowingly, without hijacking the scene. A 5 leaves mundane things mundane, or files them away as a background hook that costs the scene nothing.

C6 Failure and stakes

Whether an attempt can fail, and whether failing leaves a mark. A 1 lets everything succeed and narrates the impossible leap as a triumph. A 3 allows failure but makes it cosmetic. A 5 fails the leap plausibly, charges something real for it, and the cost is still being felt several steps later.

C7 Player agency

Whether the narration carries out the action the player typed. A 1 reinterprets or quietly drops it, decides against it on the player's behalf, or railroads toward the plot it had in mind. A 5 honours every typed action and lets its specific consequences set up the next scene. The repeated filler action the harness uses to close a long chapter is excluded from this score.

C8 Pacing and advancement

Whether progress matches what the player's actions earn. Speed is not the goal here, and rushing fails as hard as stalling: a 1 is either a step where nothing changed or whiplash escalation and a time-skip that swallows the journey. A 5 alternates quiet beats with rising tension, ties each step's progress to the action that caused it, and ends chapters on a beat the story reached honestly.

C9 World and state coherence

Whether the physical world and the character sheet stay continuous as the narration shows them: location, time of day, inventory, injuries. A 1 teleports the player between places, uses items that were never picked up and forgets a wound by the next step. A 5 holds together across the whole transcript. Any leaked markup, XML-style tag or internal directive in player-visible text is an automatic flag, whatever else the transcript does well.

C10 Character personality and voice

Depth and distinctiveness, where C2 covers stability. A 1 gives cardboard NPCs with no speech patterns of their own, all of them talking in the narrator's register and existing only to react to the player. A 5 gives characters recognisable rhythms and vocabulary, opinions that differ from the player's and from each other's, and small humanising specifics, so dialogue could be attributed with the tags stripped off. A transcript with no NPCs at all is capped at 3 and judged on the player character's voice.

Keeping the judges blind

The models doing the scoring see two transcripts called A and B and are told nothing else. The key that says which is which is kept outside the evaluation folder. So are the raw transcript files, whose names would give it away, and so are the engine's own memory records, which list what one configuration stored and the other never had. Reading those records would answer the question without reading a word of the story.

This matters more than it sounds. A model that suspects which transcript is the new one will rationalise a preference, in the same way a human reviewer does.

How the scores are combined

Each transcript is scored on its own merits first, against the absolute anchors, so a pair where both runs are weak yields two low scores. Only then are the two judged head to head, where a tie is a legitimate answer and is not broken.

Averaging takes each evaluator's own mean before averaging across evaluators, so a model that scored more rows cannot pull the result toward itself.

What this method does not tell you

Models judging models share blind spots with the thing they are judging. More than one model scores every packet for that reason, and a criterion where the evaluators disagree about direction gets reported as unresolved instead of as a result.

Scripted actions are also not players. A fixed script is what makes the comparison clean, and it means the benchmark never sees the things real players do: the sideways questions, the refusals to engage, the attempts to break the world open.