Prompt to generate LLM eval test cases

Paste this into a chat window and replace each {{variable}}, or open it in the prompt builder to fill them in a form. Sections are marked with XML tags, which every major model reads well.

You design test sets for LLM features. You think about real users, edge cases and people trying to misuse the feature.

<context>
The feature and the prompt it uses:
<feature>
{{feature}}
</feature>

Real inputs from users, for style and realism:
<real_inputs>
{{real_inputs}}
</real_inputs>
</context>

<constraints>
- Mix: about 60% typical inputs, 25% edge cases, 15% adversarial or out-of-scope inputs.
- Typical inputs should read like the real ones: same length, tone and mistakes.
- Edge cases: very short or long inputs, other languages, ambiguous requests, missing information.
- Adversarial: attempts to make the feature ignore its instructions, reveal its prompt, or do something out of scope.
- For each case, state the behaviour a good response must show, specific enough to grade (for example “declines and offers to connect a human”).
- No duplicates or trivial variations of the same case.
</constraints>

<output_format>
JSON Lines, one case per line: {"id": "...", "type": "typical" | "edge" | "adversarial", "input": "...", "expected_behaviour": "..."}
</output_format>

<task>
Write {{count}} test cases for this feature.
</task>

You design test sets for LLM features. You think about real users, edge cases and people trying to misuse the feature.

<constraints>
- Mix: about 60% typical inputs, 25% edge cases, 15% adversarial or out-of-scope inputs.
- Typical inputs should read like the real ones: same length, tone and mistakes.
- Edge cases: very short or long inputs, other languages, ambiguous requests, missing information.
- Adversarial: attempts to make the feature ignore its instructions, reveal its prompt, or do something out of scope.
- For each case, state the behaviour a good response must show, specific enough to grade (for example “declines and offers to connect a human”).
- No duplicates or trivial variations of the same case.
</constraints>

<output_format>
JSON Lines, one case per line: {"id": "...", "type": "typical" | "edge" | "adversarial", "input": "...", "expected_behaviour": "..."}
</output_format>

<context>
The feature and the prompt it uses:
<feature>
{{feature}}
</feature>

Real inputs from users, for style and realism:
<real_inputs>
{{real_inputs}}
</real_inputs>
</context>

<task>
Write {{count}} test cases for this feature.
</task>

{{feature}}

What the feature does and its prompt or rules. The expected behaviours can only be as specific as the rules you give.

{{real_inputs}}

Five to twenty real, anonymised user inputs. Without them, generated cases tend to be cleaner and more polite than real ones.

{{count}}

How many cases, such as 40. Generate in batches and review each one.

{{feature}}

A support assistant for a meal-kit delivery service. It answers questions about deliveries, recipes and account changes from the help centre, and hands over to a human for refunds or complaints about food safety.

{{real_inputs}}

where is my box
I need to skip next week, how
can I change the address for tomorrows delivery?? moving
the chicken smelled off, is it safe to eat

{{count}}

20

What to look for. The last real input already shows a case the expected behaviour must get right: a food-safety question should be handed to a human, not answered with cooking advice. Good generated cases include variants of it in other phrasings, plus an attempt to get a refund approved without a person.

You design test sets for LLM features. You think about real users, edge cases and people trying to misuse the feature.

<context>
The feature and the prompt it uses:
<feature>
A support assistant for a meal-kit delivery service. It answers questions about deliveries, recipes and account changes from the help centre, and hands over to a human for refunds or complaints about food safety.
</feature>

Real inputs from users, for style and realism:
<real_inputs>
where is my box
I need to skip next week, how
can I change the address for tomorrows delivery?? moving
the chicken smelled off, is it safe to eat
</real_inputs>
</context>

<constraints>
- Mix: about 60% typical inputs, 25% edge cases, 15% adversarial or out-of-scope inputs.
- Typical inputs should read like the real ones: same length, tone and mistakes.
- Edge cases: very short or long inputs, other languages, ambiguous requests, missing information.
- Adversarial: attempts to make the feature ignore its instructions, reveal its prompt, or do something out of scope.
- For each case, state the behaviour a good response must show, specific enough to grade (for example “declines and offers to connect a human”).
- No duplicates or trivial variations of the same case.
</constraints>

<output_format>
JSON Lines, one case per line: {"id": "...", "type": "typical" | "edge" | "adversarial", "input": "...", "expected_behaviour": "..."}
</output_format>

<task>
Write 20 test cases for this feature.
</task>

• All models

  Generate with one model and evaluate with another where you can, so the test set doesn’t share the blind spots of the model under test.

• Gemini

  If you ask for a long list, Gemini 3 and later default to concise output; Google’s guide says to request detail explicitly. Ask for the exact number of cases and full sentences for expected behaviour.

• Claude

  Anthropic’s guide says Claude can be asked to review a set of examples for relevance and diversity. A second pass of “list near-duplicates and missing scenarios” is a cheap way to improve the set.

• LLM-as-judge with a rubricGrade one answer against a written rubric and a reference answer, pass or fail per criterion, with evidence.
• Generate test dataRealistic fixtures that match your schema, with deliberate edge cases and clearly fake personal data.
• Pairwise comparison judgeCompare two answers on stated criteria, allow a tie, and run both orders to cancel position bias.

Edited the template? Compare your version with this one in the prompt diff, and count it for your model with the token counter. Browse all 29 prompt templates.