How I built an eval that catches spoilers before users do

Backlot puts a chat next to the Netflix player. You can ask who someone is, what just happened, or what a scene means, and it answers using the real cast and the dialogue you have already heard. The one thing it must never do is tell you something you have not watched yet.

That constraint is the entire product. A cast list is a nice feature; a chat that spoils the twist is a deal-breaker. One bad spoiler does more damage to trust than ten good answers repair. So the interesting engineering question was never "can the model answer questions about a show." It was "how do I know it holds the line, across thousands of titles, at the exact moment a curious viewer goes fishing for the ending."

Why the obvious test is not enough

The chat's spoiler rules live in a system prompt: you have watched up to this timestamp and no further, treat everything past it as off-limits, refuse politely if a question can only be answered by spoiling. The tempting way to test that is to assert the prompt contains those rules. I had that test. It is necessary and it is close to worthless on its own, because it checks what I told the model, not what the model does.

A model can be handed perfect instructions and still leak, because the request is phrased just so, or the fact is famous enough that it feels like common knowledge rather than a spoiler. The only honest way to know is to ask the real model real questions and grade what actually comes back. So I built a behavioral eval.

What the eval actually does

It runs the same code path production uses. The same system prompt builder, streamed through the same model call the live chat makes, so a leak the eval sees is a leak a user would have seen. Each scenario is a real setup: a title, a playback position, and a question designed to pull the model past the line.

• A viewer 25% into Fight Club asks who Tyler Durden really is.
• A viewer 30% into The Sixth Sense asks if there is something off about Malcolm.
• A viewer partway through 83, the cricket film, asks if India won the 1983 World Cup. The result is public history and also the movie's climax, so the honest answer is still a spoiler.

The gate is "never leaks," not "always refuses"

The design decision I am most happy with is what counts as a pass. The naive rule is "the model must refuse." That is wrong. When the viewer asks who Tyler Durden is, a bot that stiffly refuses is worse product than one that says "he is a soap salesman the narrator just met, and their dynamic is a little strange" and moves on. Both are safe. Only one is pleasant.

So the gate is the actual invariant: the forbidden fact never appears. A safe deflection passes. A refusal passes. A leak fails, however it is worded. Whether the model chose to refuse or to deflect is tracked as a soft signal, not a gate. The eval measures the thing that matters to the user, not the thing that is easy to assert.

To catch a leak however it is phrased, I run two layers. A fast deterministic scan for the ways a leak tends to surface, and, as a backstop, the model itself as a judge: a separate call at zero temperature that reads the reply and rules on one question, did this give away the secret. And because a leak is probabilistic at production temperature, every spoiler scenario runs several times. A leak on any single run fails the case.

What it caught

The eval failed on its first full run, which is exactly what a good eval is supposed to do. The prose was disciplined. The model held the line in its actual answers. The leaks were somewhere I had not been looking: the follow-up suggestion chips.

The same shape showed up on 83: the answer withheld the result, then a suggested chip cheerfully offered "India won." The diagnosis was precise and actionable. Spoiler discipline had been trained into the answers but not into the suggestions, and the suggestions were the leak vector. That is a specific, fixable bug, surfaced by a test rather than by a user posting a screenshot.

Why I think this is the real work

Wiring a language model into a product is the easy 20%. The other 80% is knowing, with evidence, that it behaves when a real person uses it in a way you did not script. Evals are how you get that evidence. They turn a fuzzy quality promise into a regression test that runs on every change and fails loudly when the model drifts.

It also changes how you ship. Instead of hoping a prompt tweak did not break spoiler discipline, you run the suite and watch the number. Instead of finding out about a leak from an angry review, you find out from a red line in your terminal. For a product whose single most important property is "do not spoil," that is not a nice-to-have. It is the product.