# Build a test set where the right ad is no ad

Use pairs of nearly identical prompts to find false positives before a publisher pilot.

Published: 2026-11-05T03:01:04Z

![Build a test set where the right ad is no ad](https://jolts.xyz/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MSwicHVyIjoiYmxvYl9pZCJ9fQ==--1612ea9c533fee5ae518f909d34d63f0e6fb2504/a-test-set-for-showing-nothing.webp)

Two matchers each recommend the right product for every shopping prompt in your test set. One also recommends buying a new pair of shoes when someone asks where to recycle their old ones. A shopping-only benchmark gives both matchers full marks.

There is a useful parallel in OpenAI’s analysis of hallucinations. On SimpleQA, OpenAI reported accuracy of 22% for gpt-5-thinking-mini and 24% for o4-mini. Their error rates were much farther apart: 26% and 75%, respectively. The first model abstained much more often. Those figures describe answers to factual questions. The lesson we can borrow for advertising is to record what happened when the system had the option to decline. [OpenAI’s analysis and results](https://openai.com/index/why-language-models-hallucinate/).

## A small evaluation you can inspect by hand

Keep inventory fixed: running shoes, replacement laces and one $65 daypack. Label each prompt before looking at the matcher’s output.

| Prompt | Acceptable outcome | Reason |
| --- | --- | --- |
| Find shoes for road running | Running shoes | An explicit product search |
| Where can I recycle old running shoes? | Empty | Disposal is the task |
| Find replacement laces | Laces | The user needs a part |
| How do I tie the laces I have? | Empty | The user already has the part |
| Find a daypack under $80 | $65 pack | Within budget |
| Find a daypack under $40 | Empty | No affordable item |

A rule that always shows the nearest product can pass all three shopping cases while failing all three empty cases. Report both counts. With only three examples per group, a single changed decision moves the group score by a third. The actual prompts tell a reviewer more than a percentage shown to two decimal places.

To try the product decision itself, switch between the offer and empty states in [the Jolts publisher preview](https://jolts.xyz/publishers#placement-tab-ad). Then add an unseen pair from your own app. A useful next case is one where two reviewers initially disagree about whether showing an offer would help.


## Related terms
- [Fill rate](https://jolts.xyz/resources/glossary/fill-rate)

## Next step

[Explore advertising](https://jolts.xyz/advertisers) or [monetize your AI app](https://jolts.xyz/publishers).