How do you keep DeepEval's synthetic goldens from being too easy?

## The failure this question is about You run `generate_goldens_from_docs` over the corpus, get 300 goldens, run your application, and score in the nineties on everything. That is not good news. It usually means the questions are extraction tasks whose answers sit verbatim in a single chunk, phrased in the same vocabulary as the source. Your retriever cannot miss, because the question was written from the chunk. Your generator cannot fail, because the task is restatement. The suite will report success right through a regression. Making synthetic data useful is therefore a deliberate act, not a default. ## Lever 1 — where the contexts come from The synthesizer's own splitting is a convenience, not a model of your system. If production retrieval returns 400-token chunks with heavy overlap, and the synthesizer generated from 1,024-token contexts, the goldens are anchored to text your retriever never surfaces as one unit. Passing your retriever's actual chunk groups into `generate_goldens_from_contexts` closes that gap. This is the single highest-leverage change for a RAG suite, and it costs nothing extra at generation time. ## Lever 2 — difficulty via evolutions Weighting `EvolutionConfig` toward `Evolution.REASONING` and `Evolution.MULTICONTEXT` turns lookups into synthesis. Raise `num_evolutions` cautiously: past two or three passes on most corpora, questions start leaving their grounding behind, and an unanswerable question is a false failure, not a hard test. ## Lever 3 — register, via StylingConfig This one is routinely skipped and it matters more than people expect. LLM-generated questions are well-formed, complete sentences with full context and correct spelling. Real users write "refund window??" and "can i cancel + also change address". Your application may handle the first perfectly and fall apart on the second — and a suite made only of the first will never tell you. `StylingConfig` takes plain-English descriptions: `task` (what the application does), `scenario` (who is asking and why), `input_format` (how the input should look), `expected_output_format` (how a good answer should look). Describing the input as "a short, lowercase, sometimes misspelled message typed by a support agent mid-call" changes the generated distribution substantially. ## Lever 4 — filtering and human review `FiltrationConfig` takes a `critic_model` and a quality bar, with retries when a generated input scores below it. Use a stronger model here than for generation: filtering is a judgment task and generation is a volume task, so the money is better spent on the critic. But filtering is a model checking a model. The irreducible step is a human sample-read: pull twenty goldens, read each question beside its `context`, and answer three things — could I answer this from this context? Is this a question a real user would ask? Would a mediocre system get it wrong? A golden that fails the third question is filler. The same scrutiny applies harder to generated `expected_output`. It was written by a model from the same context, so it inherits that model's blind spots — and once it is stored, it becomes the standard your application is judged against. Review references before you trust any metric that compares against them, or keep the goldens reference-free and rely on metrics that do not need one. ## The check that settles it: discriminating power Whatever you do to the generation config, the dataset is only worth gating on if it can tell good from bad. Build a deliberately degraded variant of your application — return three random chunks instead of the top-k, or swap in a much weaker generator — and run both against the suite. If the scores barely move, the suite is measuring nothing regardless of how sophisticated the generation config was. If the degraded build drops clearly and the good build holds, you have a dataset worth keeping. Run the same check when you regenerate. A new synthetic batch is a new measuring instrument, and instruments need calibrating against a known-bad reference before you trust their readings. ## Mixing in the real thing The strongest suites are not purely synthetic. Seed them with real inputs — questions from support tickets, logged production queries, the specific cases behind past incidents — and let synthetic generation provide breadth around that spine. Every real failure you have shipped deserves a permanent golden, and that one is curated by hand, forever, no matter what the synthesizer produces.