In a KFP v2 pipeline, how do you run the deploy step only when the new model beats the current one, and what replaced dsl.Condition?
answer
- the backend evaluates the branch
- compare parameter outputs, not artifacts
- with dsl.If(...): deploy inside
- Elif and Else right after If
- dsl.Condition warns, points to dsl.If
basics
~20 sHave the evaluate step return both scores as parameter outputs and call deploy inside a with dsl.If(...) block that compares them; the backend evaluates the condition at run time. dsl.If, with dsl.Elif and dsl.Else, replaced the deprecated dsl.Condition.
solid answer
~50 sA pipeline function only builds a graph, so the decision has to be a graph construct that the backend evaluates at run time. Make `evaluate_model` return parameters, for example a `NamedTuple` with `new_score: float` and `current_score: float`. Then put the deploy call inside `with dsl.If(eval_task.outputs['new_score'] > eval_task.outputs['current_score']):`. The comparison builds a condition on parameter channels, at least one operand must be a task output or a pipeline parameter, and artifacts cannot be compared, so read the score out of a `Metrics` artifact into a returned float. `dsl.Elif` and `dsl.Else` must follow immediately, and an `Else` can run a notify step. `dsl.Condition` still compiles but is deprecated and warns `dsl.Condition is deprecated. Please use dsl.If instead.` A plain Python `if` on a task output never reaches the backend: the comparison object is truthy at compile time, so deploy is compiled in unconditionally.
code
python · 21 linesfrom typing import NamedTuple
from kfp import dsl
from kfp.dsl import Dataset, Input, Model
@dsl.component(base_image='python:3.11')
def evaluate_model(model: Input[Model], test: Input[Dataset]
) -> NamedTuple('scores', new_score=float, current_score=float):
scores = NamedTuple('scores', new_score=float, current_score=float)
return scores(0.912, 0.897) # stand-in for scoring both models on test
@dsl.pipeline(name='article-topics-weekly')
def article_topics():
split_task = split_dataset(articles=ingest_articles().outputs['articles'])
train_task = train_classifier(train=split_task.outputs['train'])
eval_task = evaluate_model(model=train_task.outputs['model'],
test=split_task.outputs['test'])
with dsl.If(eval_task.outputs['new_score'] > eval_task.outputs['current_score'],
name='beats-current'):
deploy_model(model=train_task.outputs['model'])
with dsl.Else():
notify(message='Candidate did not beat the current model')go deeper
Recall that branching in a pipeline uses with dsl.If(...) around the tasks to gate, and that dsl.Condition is the deprecated older spelling.
Explain why a Python if cannot work at compile time, what operands a dsl.If condition accepts, and how Elif, Else and OneOf fit together.
Build the gate so it fails safe: scores returned as parameters, the deploy only inside dsl.If, an Else that reports, and an ExitHandler for failures; catch the silent always-deploy bug in review.
Separate concerns: the promotion rule and rollback policy are owned outside the pipeline SDK, while the pipeline only evaluates that rule reproducibly and records why each week deployed or not.
## Why a Python if cannot gate a step A `@dsl.pipeline` function in Kubeflow Pipelines (KFP) v2 runs once, at compile time, to build a graph. At that moment `eval_task.outputs['new_score']` is not a number. It is a **pipeline channel**, a reference to a value that will exist only after the evaluate step has run on the cluster. Comparing two channels with `>` does not produce `True` or `False`. It produces a condition object that describes the comparison. That is why a plain Python `if` goes wrong: ```python if eval_task.outputs['new_score'] > eval_task.outputs['current_score']: deploy_model(model=train_task.outputs['model']) # always compiled in ``` The condition object is truthy, so Python takes the branch while the graph is being built, and `deploy_model` lands in the graph with no condition at all. Every weekly run then deploys, whatever the scores. Nothing errors, which is what makes the bug dangerous. ## dsl.If, dsl.Elif and dsl.Else The fix is to hand the comparison to a control-flow group that the backend evaluates at run time: - `with dsl.If(condition, name=...)`: the tasks inside run only when the condition holds. - `with dsl.Elif(...)` and `with dsl.Else()`: further branches, which must come **immediately** after the `dsl.If` group. ## What a condition may compare - At least one operand must be a **task output or a pipeline parameter**. `dsl.If` with a constant boolean raises `ValueError` ("Got constant boolean ... as a condition"). - The operands are **parameter** values: `==`, `!=`, `<`, `<=`, `>`, `>=` between parameter channels or between a channel and a constant. - An artifact cannot be compared. If evaluation writes a `Metrics` artifact, have the same component also return the score as a `float`, or a `bool` such as `promote`, so that the gate has a parameter to test. ## From dsl.Condition to dsl.If KFP v1 pipelines, and early v2 ones, used `with dsl.Condition(...)`. In SDK 2.17 `dsl.Condition` still exists, as a subclass of `dsl.If` that is functionally identical. On entry it emits a `DeprecationWarning`: "dsl.Condition is deprecated. Please use dsl.If instead." Migrating means renaming it, and then using `dsl.Elif` or `dsl.Else` where the old code had a second condition with the comparison negated. Compare this with `ContainerOp`, which was removed in v2 outright. `dsl.Condition` is deprecated, not gone, so old code keeps compiling but warns. ## Related control flow | Construct | Purpose | |---|---| | `dsl.OneOf(t1.output, t2.output, t3.output)` | one output taken from whichever branch of an If/Elif/Else group ran; the group must include `dsl.Else` and the outputs must share a type | | `dsl.ParallelFor(items=..., parallelism=...)` with `dsl.Collected(...)` | fan out over a static list, then gather the outputs as a list | | `dsl.ExitHandler(exit_task=...)` | run a task after the tasks in its scope finish, even if one fails | | `.ignore_upstream_failure()` | let one task run even when the tasks it depends on failed | ## Making the gate easy to audit A gate that decides silently is hard to trust, so make each week's decision visible: - Log both scores in a `Metrics` artifact, `metrics.log_metric('new_score', ...)`, so that the run page shows the numbers next to the decision. - Name the branch, as in `dsl.If(..., name='beats-current')`, so that the run graph shows which group ran and which was skipped. - Make the `Else` branch do something observable, such as a notify step, rather than leaving it empty, so that "did not deploy" is a recorded outcome and not just an absence. ## Putting the gate together 1. `train_classifier` writes `Output[Model]`. 2. `evaluate_model` scores the candidate and the current production model on the same held-out split. It returns `new_score` and `current_score` as a `NamedTuple` of floats, and can also log a `Metrics` artifact for the UI. 3. `with dsl.If(eval_task.outputs['new_score'] > eval_task.outputs['current_score'], name='beats-current'):` wraps `deploy_model(model=train_task.outputs['model'])`. 4. `with dsl.Else():` wraps a `notify` step that reports the skipped deployment. 5. Optionally, wrap the whole graph in `dsl.ExitHandler` so that a status report goes out even when training fails. What counts as "beats", whether by a margin, on which slices and with what rollback, is a promotion-policy decision taken outside the pipeline SDK. The pipeline's job is to evaluate that rule at run time instead of at compile time.
- How do you send a status report whether the run deploys, skips or fails?Wrap the graph in `with dsl.ExitHandler(exit_task=report_task):`. The exit task runs after the tasks in scope finish, even if one fails, much like a `finally:` block. It may not depend on other tasks' outputs, but it can declare a `dsl.PipelineTaskFinalStatus` input, which the backend fills with the run's state and any error code and message.
- How does a later step consume an output from whichever branch ran?Use `dsl.OneOf(t1.output, t2.output, t3.output)` over tasks inside one If/Elif/Else group. The compiler requires the group to include a `dsl.Else`, so that exactly one branch runs, and the outputs passed to `dsl.OneOf` must share a type. They cannot themselves be `dsl.OneOf` or `dsl.Collected`.
- A Python if on a task output is evaluated when the run executes
- dsl.Condition was removed in KFP v2 along with ContainerOp
- A Metrics artifact can be compared directly inside dsl.If
- dsl.Else can sit anywhere later in the pipeline body
Assumes KFP SDK 2.17.0, where dsl.Condition remains as a deprecated alias of dsl.If and emits a DeprecationWarning.