How do you run a LangSmith experiment over only one split of a dataset?

## Splits are labels, not datasets The mental model that trips people up is the machine-learning one, where train/validation/test are three separate files. In LangSmith a split is a **label on an example inside one dataset**. The dataset remains the unit of identity — the thing experiments attach to, the thing that gets versioned — and splits are a way to slice it. That has a pleasant consequence: an example can be in a split and still be part of the full set, so a small smoke split is a *view* on the big dataset, not a copy of some of its rows. When you fix a reference answer, you fix it once and both views see the fix. Copying rows into a second dataset gives you two references that drift apart. ## Assigning a split The usual path is the UI: select examples in the dataset view and put them in a split. In the SDK, splits are set through the example-update call, `client.update_examples(...)` with the `splits` argument, which is how you script the assignment from a rule — for instance, labelling every example whose metadata says it came from an incident. ## Running an experiment on one There is no `split=` parameter on `evaluate`. The mechanism is the one `data` already offers: it accepts a dataset name, a dataset id, **or an iterable of examples**. So you fetch the subset and hand it over: ``` evaluate( target, data=client.list_examples(dataset_name="support-qa", splits=["smoke"]), experiment_prefix="pr-1421-smoke", ) ``` `list_examples` returns an iterator over the matching examples, and the same trick generalises: any filtered listing you can express — by split, by metadata, by version — becomes the population of an experiment. ## Why teams reach for this The common shape is two tiers. A small, fast split runs often, because a suite that takes twenty minutes and eighty dollars will not survive contact with a pull-request workflow. A larger set runs less often, on a schedule or before a release. Splits let both tiers read from one curated source of truth rather than two datasets that slowly diverge. Other uses are just as ordinary: a split per failure category so you can see whether the change that helped summarisation hurt refusals; a split of examples with hand-verified references, separated from ones auto-promoted from traces; a split of the newest examples that no prompt has been tuned against yet, kept as a held-out set so you can tell tuning from overfitting. ## What splits do not do They do not make cross-split experiments comparable. An experiment over 40 smoke examples and an experiment over 800 full examples produce two averages that mean different things; putting them side by side is a category error even though both live under the same dataset. Compare like with like, and label the experiment prefix with the split so nobody misreads the list. They also do not protect you from membership churn. If someone adds twelve rows to the smoke split on Tuesday, Wednesday's number is not measuring the same thing as Monday's. If a split is a tracked gate, treat its membership as change-controlled, and when you do extend it, re-run the previous configuration so you have a fresh baseline on the new membership rather than comparing across a moved goalpost. Finally, splits do not by themselves reduce cost when you are careless with what you call: a 40-example smoke run with three LLM-judge evaluators is still 160 model calls. The split shrinks the row count; the per-row call count is your evaluator choice. ## Naming discipline Split names show up in filters and in nothing else — they carry no semantics for the tool. So they are worth naming for humans: `smoke`, `held-out`, `refusals`, `verified` all say what they are. Names like `split-2` mean that a year from now nobody can tell whether it is safe to run against.