roadmap
- 01
- 02
- 03
- 04
- 05
- 06
- 07
- 08
Prepare for promptfoo interview questions on evaluating and red-teaming LLM apps from one config. Covers assertions and model graders, attack plugins and strategies, scoring and CI gates.
162 of 162 questions in your plan: 0 known, 0 in progress, 0 due, 0 skipped, 162 new. 0 cut, outside your plan.
promptfoo comes up in two kinds of loop. AI engineers meet it inside an evaluation or LLMOps round, where the question is how a prompt or model change gets tested before it ships and what the CI job does when a score drops. AI red teamers and AI security engineers meet it in a tooling conversation or a take-home, often a promptfooconfig.yaml to read or extend against a deliberately weak target. In both, interviewers seldom quiz flags. They hand you a result and ask what it proves.
The probes are specific. A suite passed on llm-rubric assertions: was the output safe, or was the grading model lenient? A red-team report shows a low attack success rate over three plugins: which harm classes were never attempted at all? Last month's run and this week's disagree: did the model change, did the suite regenerate in between, or did a warm cache replay old answers? Knowing what each key in the file does and what promptfoo redteam run produces is the floor. From middle level up, candidates are expected to treat every number as a measurement with a denominator, a grader and a sample size, and to say which of the three moved.
Evaluation in general comes before any promptfoo syntax. Offline and online evals, judge validity and regression gating are the ideas the tool's config file encodes, and they decide whether its output means anything. Writing an eval then puts them into that file: which providers and prompt versions to cross, which assertions grade each answer, and how to aim the run at a live application over HTTP. Red-team mode is the same file with a generator added, which is why it comes second.
Before any generation, the plan stops on what is about to be generated. The OWASP LLM Top 10 is one of the framework presets promptfoo selects plugins by, and the jailbreak families, from fiction as cover to consent built up over several turns, are the framings its strategies wrap around each case. Red-team generation follows: plugins, strategies, the purpose text the generator reads and the number of cases per plugin. The attacker-model loops behind the iterative and tree-based strategies sit right after it, because a search that refines a prompt against a judge inherits that judge's errors.
From there the plan turns from producing attacks to reading what came back. What a success rate measures, including over-refusal and sampling error, comes first; measuring a guard adds bypass rate against over-blocking for the filter in front of the model; and published frameworks are how a finding gets mapped for a reader outside the team. Time comes last. Continuous evaluation asks when two runs are comparable at all, and safety evaluation is the release decision those runs feed.
An AI engineer who expects promptfoo inside an evaluation round can cut level down to the seniority on the posting and spend most of the hours on the first two modules and the last one: assertions, graders and CI comparisons carry those interviews. For an AI red team or AI security role, study by topics from the attack taxonomy onward; with only days left, raise importance to must know. Senior candidates in either track should expect the scoring module to decide the conversation, since a reported rate has to survive a question about its denominator. Whichever track you are on, run one small eval and one generated red-team suite against a toy app first: questions about cost per run, caching and regenerated suites are easier to answer about a suite you have paid for. The plan treats promptfoo as the instrument, not the engagement. Scoping and running an engagement, prompt injection and data exfiltration in depth, other scanners such as PyRIT and garak, and the wider attack surface of AI systems belong to the [AI Red Teaming roadmap](/roadmaps/ai-red-teaming). Building the retrieval and agent systems that get tested is covered by the [AI Engineer roadmap](/roadmaps/ai-engineer).
An AI engineer who expects promptfoo inside an evaluation round can cut level down to the seniority on the posting and spend most of the hours on the first two modules and the last one: assertions, graders and CI comparisons carry those interviews. For an AI red team or AI security role, study by topics from the attack taxonomy onward; with only days left, raise importance to must know. Senior candidates in either track should expect the scoring module to decide the conversation, since a reported rate has to survive a question about its denominator. Whichever track you are on, run one small eval and one generated red-team suite against a toy app first: questions about cost per run, caching and regenerated suites are easier to answer about a suite you have paid for.
The plan treats promptfoo as the instrument, not the engagement. Scoping and running an engagement, prompt injection and data exfiltration in depth, other scanners such as PyRIT and garak, and the wider attack surface of AI systems belong to the AI Red Teaming roadmap. Building the retrieval and agent systems that get tested is covered by the AI Engineer roadmap.