How do you run GitHub Models prompt evaluations from a GitHub Actions workflow?

## The problem being solved Prompts are production logic that traditionally escapes every engineering control: no diff review, no tests, no regression signal. GitHub Models addresses that by making a prompt a file in the repository and giving CI a way to execute and score it. ## Prompt files A `.prompt.yml` file describes one prompt as data. It names the model to run against, the model parameters, and the messages — system and user, with placeholders for inputs. Alongside that it can carry test data (the sample inputs to run) and evaluators (the checks applied to each response). Because it is YAML in the repo: - a prompt change appears in a pull request diff and gets reviewed; - the model id and parameters are versioned with the code that uses them; - the same file drives the browser editor, the CLI and CI, so there is one definition rather than three drifting copies. ## The CLI `gh models` is an extension to the GitHub CLI, installed with `gh extension install github/gh-models`. The commands you will actually name in an interview: - `gh models list` — what the catalog offers. - `gh models run <model> "<prompt>"` — a one-off call from the terminal, useful for a quick sanity check or piping output into a script. - `gh models eval <file.prompt.yml>` — execute the prompt over its test data and apply its evaluators, reporting per-case results. The CLI authenticates with your `gh` login locally and with the workflow token in CI, so the same command works in both places. ## Wiring it into Actions Three requirements: 1. **Permissions.** Add `models: read` to the workflow or job `permissions` block. Remember the block is exhaustive for the scope it appears on — if you write it on the job, also list `contents: read` or the checkout step loses access. 2. **The token in the environment.** Expose `GITHUB_TOKEN` to the step (commonly as `GH_TOKEN`, which the GitHub CLI reads). No repository secret is needed, and the credential expires with the run. 3. **The trigger.** `pull_request` is the useful one: you want the signal while the change is reviewable. Some teams add a scheduled run to catch drift when nothing changed on their side. Gate the job on the evaluation outcome if you want a hard signal, or run it informationally and read the output in the job log while the practice beds in. ## Evaluation is comparison, not proof The honest framing for an interview: automated prompt evaluation gives you a *relative* signal — this prompt versus the previous one, this model versus that one, on your own cases. It is not proof of correctness. Model outputs vary between runs, so a small test set can flip on noise. Practical mitigations: keep a decent number of cases per behaviour you care about, prefer deterministic checks where the task allows one, treat borderline score movement as a prompt to look rather than an automatic block, and keep human review for the cases that matter most. ## Cost and quota discipline Every evaluation case is inference, and CI inference draws on the same account-scoped rate limits as every developer and every other job. A large test matrix across several flagship models, running on every push, will exhaust a prototyping allowance and throttle your colleagues. Sensible defaults: run on pull requests rather than every push, keep the routine matrix on small models and reserve larger ones for a scheduled or manual deeper run, and cap the case count. If evaluation becomes central to your process, that is the moment to consider paid usage or a dedicated endpoint so CI stops competing with humans. ## What this does not replace Evaluation in CI checks prompt behaviour on curated inputs. It does not replace application tests around the code that builds the prompt and parses the response, and it does not replace production monitoring of real traffic. Say that out loud: the strongest answer positions prompt evals as one layer, sitting between unit tests and observability.