Compare two prompts, side by side

Why comparing two prompts is harder than it looks

The obvious method — run both, read both answers, keep the better one — is the one that misleads most reliably.

Language models are nondeterministic. They select each next token probabilistically, so the same prompt sent twice produces different text. In our own testing across a 132-input corpus, quality scores for a single unchanged prompt varied by 12 to 16 points between runs. That spread is larger than most of the improvements people are trying to detect.

Which means: run version A once and version B once, and roughly half the time you will conclude that the worse prompt is better. You are measuring the sampler, not your edit. This is the single most common error in prompt iteration, and it feels like evidence, which is what makes it dangerous.

What this tool compares — and what it does not

It compares the prompts, not their output. Each version is scored across eleven dimensions that measurably change what a model returns — task clarity, audience, format, specificity, examples, constraints, reasoning, role, exclusions, length target and structure — and the table shows how each one moved.

The totals matter less than the breakdown. A rewrite that gains four points overall while losing six on context and gaining ten on format has done something specific, and the totals hide it entirely. That trade might be exactly what you wanted, or it might be an accident you introduced while fixing something else. You cannot tell from two numbers.

What it cannot tell you is whether the idea is good. A perfectly specified prompt asking the wrong question scores well and produces a well-formed answer to the wrong question. Structure is necessary and not sufficient.

How to run a prompt test that means something

1. Write down what "better" means first. Before you generate anything. "More specific", "shorter", "no invented statistics" — a criterion you set afterwards is a criterion you fitted to the result you liked.
2. Change one thing. If B adds an audience and a format and a role, you will not know which one mattered. The table below tells you how many dimensions moved; more than one or two and your test is compound.
3. Run each version at least three times. Five for anything you will depend on. This is the step people skip and it is the one that makes the rest worthwhile.
4. Take the median, not the best. The best result of five tells you what the prompt can do on a good day. The median tells you what it will do on Tuesday.
5. Judge blind where you can. Knowing which version produced which answer biases you toward the one you worked harder on.
6. Keep the losers. A record of what did not work is worth as much as the winner, and it stops you retrying the same idea in three months.

The eleven dimensions, and what they weigh

Weights reflect how much each one changes an answer in practice. The comparison table orders rows by how far each moved, so the biggest change appears first — which is usually not the change you thought you were making.

What actually moves the score

Across the prompts we have measured, the same few edits account for most real improvement — and they are not the ones people reach for first.

Adding an audience is the largest single available gain in most prompts, and the most commonly missing element. "For finance leads who have never used a data warehouse" tells a model what to explain and what to assume; without it, the model writes for everyone, which means no one.

Adding one example is worth ten points and is skipped almost universally. A single line showing the shape you want communicates more about tone and structure than a paragraph describing them.

Specifying format with a number beats specifying it vaguely by a wide margin. "Use sections" could mean two or twenty; "four sections of about 200 words each, then a five-row table" cannot be misread.

Conversely, adding a role is the edit people make first and it is weighted eighth of eleven. "You are a helpful assistant" changes nothing at all — it is what the model already believes it is.

A worked comparison

The example pair loaded by the button above, with the real numbers. Load it and you will get exactly these — the scoring is deterministic.

Version A: Write a blog post about email marketing for small businesses. Make it engaging and useful. — scores 17. Fifteen words, and almost every decision is left open.

Version B adds a role, an audience, a word count, a section count, a table specification, an example of the section pattern, and two exclusions — scoring 72.

Two things are worth noticing. The single biggest gain is audience, not the role — and the role is what most people add first. And B still scores 72 rather than 100: it never gives a worked example of the writing itself, and it does not ask the model to think before drafting. Those two gaps are visible in one glance at the table, which is the entire argument for looking at dimensions instead of totals.

Which dimension to fix next

Once you can see the breakdown, the next edit stops being guesswork. Work from the largest gap, not from the top of the list.

Make one of these, compare again, and confirm that the dimension you aimed at is the one that moved. If a different row changed instead, your edit did something other than what you intended — which is worth knowing before you run it against a model.

Five mistakes that make an A/B test worthless

• One run each. Covered above, and worth repeating because it is the mistake almost everyone makes.
• Changing several things at once. You learn that B is better, and nothing about why, so you cannot apply it to the next prompt.
• Judging on the first paragraph. Prompt changes often affect structure and completeness, which show up at the end.
• Comparing across models. Different model, different result — that is not your prompt change. Hold the model, the temperature and any system prompt constant.
• Testing on one input. A prompt template that works beautifully for one topic and falls apart on three others has not been tested, it has been demonstrated.

When the lower-scoring version is the right one

The score measures completeness of specification. That is usually what you want, and sometimes it is not.

Simple asks. "What is the capital of Peru" scores terribly and is a perfect prompt. Adding an audience, a format and three constraints would make it worse.

Creative latitude. If you want the model to surprise you, every constraint narrows the space it can surprise you from. Specify the things you genuinely care about and deliberately leave the rest open — that is a decision, not an omission.

Exploration. Early in a piece of work, an open prompt surfaces angles you had not considered. Tighten it once you know what you are looking for.

Temperature, and why it belongs in any comparison

Temperature controls how much randomness the sampler allows. At 0 the model takes the most likely token every time and output is near-identical between runs; at 1 it samples more freely and varies considerably. Most chat interfaces sit somewhere in between and do not show you the number.

This matters for testing in two ways. Low temperature makes comparison easier because it removes most of the run-to-run variance you would otherwise have to average out — if the API you use exposes the setting, turn it down while iterating and put it back afterwards. And a prompt tuned at one temperature may not hold at another: constraints that are respected at 0.2 start being interpreted loosely at 0.9.

The structural comparison above is unaffected by any of this, which is precisely why it is a useful first step. It gives you a fixed reference point that does not move while everything else does.

Common questions

Why do I sometimes get the same score for two different prompts?

Because the differences are in wording rather than in specification. If B rephrases A without adding an audience, a format or a constraint, nothing the scorer measures has changed — and quite possibly nothing the model sees has meaningfully changed either.

Can I compare more than two versions?

Compare them in pairs, keeping the winner each time. Comparing a field of six at once invites the compound-change mistake.

Does prompt length affect the comparison?

Only indirectly. A longer prompt has more room to specify things, but two 50-word prompts in our testing scored 43 and 100. What the words do matters; how many there are does not.

Should I compare before or after running the prompts?

Before. It costs nothing and takes a second, and it often shows that two versions are structurally identical — which saves you running an experiment that could not have produced a result.

Is my prompt stored?

No. Both versions are scored in memory and discarded — not logged, not saved, not used for training.

Glossary

• A/B test — comparing two versions that differ in one respect, to attribute a difference in outcome to that one change.
• Nondeterminism — the same input producing different output on different runs.
• Temperature — how much randomness the sampler allows. Lower is more repeatable.
• Median — the middle value of several runs. The honest summary of what a prompt does, where the maximum is the flattering one.
• Compound change — altering several things at once, so no single cause can be attributed.
• Regression — an edit that fixes one dimension while quietly breaking another. What the per-dimension table is for.
• Baseline — the version you are measuring against. Without one, "better" has no referent.