skip to content

Kubeflow

Kubeflow runs ML on Kubernetes: Pipelines compiles Python steps into containers, Trainer launches distributed TrainJobs and Katib tunes them. Interviewers ask when that platform pays off.

on this pageshow

explore

questions

12

In the KFP v2 SDK, how do @dsl.component, @dsl.pipeline and the compiler turn Python functions into a run on a Kubeflow Pipelines backend?

level: juniorimportance: must knowfreq 22%

answer

  1. a graph, not a function call
  2. task outputs become the edges
  3. Compiler().compile writes IR YAML
  4. kfp.Client files the run in an experiment
  5. backend turns IR into an Argo Workflow

basics

~20 s

@dsl.component makes a typed function a container step, @dsl.pipeline wires steps into a graph through their outputs, Compiler().compile writes that graph as IR YAML, and kfp.Client submits it as a run in which each step gets its own container.

solid answer

~40 s

`@dsl.component` turns a type-annotated Python function into a component: an image, a command and a declared set of inputs and outputs. Inside a `@dsl.pipeline` function, calling a component does not run it; it returns a `PipelineTask`, and passing `ingest_task.outputs['articles']` into the next call adds an edge to the graph. `kfp.compiler.Compiler().compile(pipeline_func=..., package_path='article_topics.yaml')`, or `kfp dsl compile`, writes that graph as IR YAML, a platform-neutral `PipelineSpec`. `kfp.Client(host=...)` then submits it with `create_run_from_pipeline_package`, or with `create_run_from_pipeline_func`, which compiles for you, and files the run under an experiment. The backend translates the IR into an Argo Workflow and runs each task in its own container, writing artifacts under the pipeline root in the object store.

code

python · 24 lines
python
import kfp
from kfp import compiler, dsl
from kfp.dsl import Dataset, Input, Model, Output

@dsl.component(base_image='python:3.11')
def ingest_articles(week: str, articles: Output[Dataset]):
    with open(articles.path, 'w') as f:
        f.write(f'articles for {week}\n')  # stand-in for the newsroom export

@dsl.component(base_image='python:3.11')
def train_classifier(articles: Input[Dataset], model: Output[Model]):
    with open(articles.path) as src, open(model.path, 'w') as dst:
        dst.write(src.read())  # stand-in for fitting the topic classifier

@dsl.pipeline(name='article-topics-weekly')
def article_topics(week: str):
    ingest_task = ingest_articles(week=week)
    train_classifier(articles=ingest_task.outputs['articles'])

compiler.Compiler().compile(article_topics, package_path='article_topics.yaml')
client = kfp.Client(host='http://localhost:3000')
client.create_run_from_pipeline_package(
    'article_topics.yaml', arguments={'week': '2026-W41'},
    experiment_name='article-topics')

go deeper

for a junior

Recall the chain: decorate functions as components, wire them in a pipeline through task outputs, compile to IR YAML, submit with kfp.Client into an experiment.

for a middle

Explain that the pipeline body builds a graph of PipelineTask placeholders, what IR YAML contains, and the difference between submitting a package and submitting a function.

for a senior

Show you know where things run: each task in its own container via an Argo Workflow, data through JSON parameters or object-store artifacts, and local runners for fast iteration before a cluster run.

for a principal

Weigh what the compiled, platform-neutral spec buys a team: reviewable pipeline versions, one artifact store with lineage, and runs anyone can repeat from a pipeline version rather than a notebook.

## Three objects: component, pipeline, compiled spec Kubeflow Pipelines (KFP) v2 splits a pipeline's life into three phases: you author it in Python, compile it to a portable spec, and run it on a cluster. This answer assumes KFP SDK 2.17.0 against a 2.17 backend. Three terms carry the whole flow: - **Component**: a function decorated with `@dsl.component` (or `@dsl.container_component`). Its type annotations declare its inputs and outputs, and the SDK turns it into a container definition: an `image`, a `command` and `args`. Every component executes as a container, Python ones included. - **Pipeline**: a function decorated with `@dsl.pipeline`. Its body calls components and wires the outputs of some into the inputs of others. - **IR YAML**: the compiled form, an instance of the `PipelineSpec` protocol-buffer message. The compiler writes it and the backend reads it. ## What the pipeline function really does A `@dsl.pipeline` body looks like ordinary Python, but it only describes topology. Calling `ingest_articles(week=week)` inside it fetches no articles. The call returns a **`PipelineTask`**, a placeholder for an execution that will happen later on the cluster. Passing `ingest_task.outputs['articles']` (or `.output` for a single unnamed output) into `train_classifier(...)` records a data dependency, and that dependency becomes an edge in the graph. When two steps must run in order without exchanging data, `.after(other_task)` adds the edge explicitly. Task-level settings hang off the same handle: `set_caching_options`, `set_retry`, `set_cpu_limit`, `set_memory_limit`, `set_accelerator_type`, `set_env_variable`. They are recorded in the spec. None of them is applied in your Python process. ## Compiling to IR YAML `kfp.compiler.Compiler().compile(pipeline_func=article_topics, package_path='article_topics.yaml')` type-checks the wiring (`type_check=True` by default) and writes the IR YAML. The CLI equivalent is `kfp dsl compile --py pipeline.py --output article_topics.yaml`. Unlike the KFP v1 component YAML, the file is not meant to be written by hand, but you can still read it: | Section | What it holds | |---|---| | `components` | each component's interface: inputs, outputs, executor reference | | `deployment_spec` | the executors: image, command and args per component | | `root` | the outermost graph, the one that runs when you submit | | `pipeline_info` | the pipeline name, used to group runs and artifacts | | `sdk_version`, `schema_version` | which SDK and which schema produced the file | | `default_pipeline_root` | where artifacts go unless something overrides it | Because the spec is platform-neutral, any conformant backend can accept the same file. ## Submitting a run with kfp.Client A **run** is one execution of a pipeline, and an **experiment** is a named group of runs. With a client connected to the backend, `kfp.Client(host=...)`: 1. `create_run_from_pipeline_package('article_topics.yaml', arguments={...}, experiment_name='article-topics')` submits a file you already compiled. 2. `create_run_from_pipeline_func(article_topics, arguments={...})` compiles to a temporary IR YAML file and submits it in a single call. 3. If you give no experiment, the client looks for an experiment called `Default` (or the name in the `KF_PIPELINES_DEFAULT_EXPERIMENT_NAME` environment variable) and creates it if it is missing. 4. `upload_pipeline` and `upload_pipeline_version` store the spec on the backend as a named, versioned pipeline. The UI and recurring runs can then start runs from it without your local file. Each submission returns a `RunPipelineResult` with a `run_id`, and its `wait_for_run_completion()` blocks until the run ends. ## What the backend does with it The open-source backend compiles the `PipelineSpec` once more, into an Argo Workflow, which is the definition it actually executes. Each task therefore runs as its own container on the cluster, and data never moves through shared memory: - **parameters** (strings, numbers, booleans, lists, dicts) travel as small JSON values; - **artifacts** (datasets, models, metrics) are files the backend copies to the **pipeline root**, a path in an object store; - the UI shows the run graph, each step's logs and its output artifacts. Since KFP 2.15 the default deployment ships SeaweedFS as that object store in place of MinIO. MinIO and any other S3-compatible store remain supported. ## Trying it on a laptop first You can exercise the same code before a cluster is involved. Run `from kfp import local`, then `local.init(runner=local.DockerRunner())`, and call a component or the pipeline like a function. The outputs come back as real values, and artifact files land under a local pipeline root. The docs recommend `DockerRunner`, which gives every task its own container; `SubprocessRunner` is for environments without Docker. Resource settings such as `set_memory_limit` have no effect locally, and nothing is sent to the backend.

  • How do you try the same pipeline on a laptop before submitting it?
    Call `local.init(runner=local.DockerRunner())` from `kfp.local`, then call the component or pipeline like a function. Outputs come back as real values and artifact files under a local pipeline root (`./local_outputs` by default). `DockerRunner` runs each task in its own container and is closest to the cluster; `SubprocessRunner` is for machines without Docker. Resource settings like `set_memory_limit` are ignored locally.
  • When would you upload the pipeline rather than submit the compiled file each time?
    When others, the UI or a schedule need to start runs from it. `upload_pipeline` and `upload_pipeline_version` store the IR YAML on the backend as a named pipeline with versions, so a run or a recurring run can reference a pipeline ID and version ID. `create_run_from_pipeline_package` just sends the spec with one run and leaves nothing reusable behind.

  • Calling a component inside the pipeline function runs its code right there
  • The compiler's output is an Argo Workflow you apply with kubectl
  • All steps of a run share one Python process and pass objects in memory
  • IR YAML is meant to be hand-written like a v1 component YAML
  • A run cannot be submitted until an experiment has been created by hand

Assumes KFP SDK 2.17.0 with a 2.17 backend; SeaweedFS replaced MinIO as the default object store deployment in KFP 2.15.

open as a page

In Kubeflow Trainer v2, what does a TrainJob's runtimeRef point at, and which part of the training job does that runtime own?

level: juniorimportance: must knowfreq 14%

basics

~20 s

A TrainJob's runtimeRef names a ClusterTrainingRuntime (the default kind) or a same-namespace TrainingRuntime. That platform-owned runtime holds the JobSet template and mlPolicy; the TrainJob adds only run values such as image, command, node count and resources.

open as a page

In a KFP v2 pipeline, when should a step pass data as an Input[Dataset] or Output[Model] artifact instead of a str parameter, and why?

level: middleimportance: must knowfreq 16%

basics

~20 s

Parameters (str, int, float, bool, list, dict) are small JSON values passed with a task's inputs. Datasets, models and metrics belong in artifacts: written to .path, then copied by the backend to object storage with lineage.

open as a page

In a KFP v2 pipeline, how do you run the deploy step only when the new model beats the current one, and what replaced dsl.Condition?

level: middleimportance: should knowfreq 8%

basics

~20 s

Have the evaluate step return both scores as parameter outputs and call deploy inside a with dsl.If(...) block that compares them; the backend evaluates the condition at run time. dsl.If, with dsl.Elif and dsl.Else, replaced the deprecated dsl.Condition.

open as a page

Why does a KFP v2 lightweight @dsl.component that relies on a module-level import and a same-file helper fail on the cluster with NameError?

level: middleimportance: should knowfreq 10%

basics

~20 s

A lightweight component ships only its own function source, run in base_image after KFP's own imports, so module-level imports and helpers never arrive. Import inside the body, list packages in packages_to_install, pass helpers through additional_funcs, or build a containerized component.

open as a page

How do Katib's Experiment, Suggestion and Trial objects split the tuning of a Kubeflow TrainJob, and how many trials run at once by default?

level: middleimportance: should knowfreq 8%

basics

~20 s

The Experiment declares objective, search space, algorithm, limits and a trialTemplate; its Suggestion obtains parameter assignments from the algorithm; each assignment becomes a Trial that renders and runs one TrainJob and records its metric. parallelTrialCount defaults to 3.

open as a page

A Kubeflow Trainer TrainJob on the torch-distributed runtime sets trainer.numNodes: 4 — how does that become pods, and how does each torchrun learn its place?

level: middleimportance: should knowfreq 9%

basics

~10 s

trainer.numNodes overrides the runtime's mlPolicy.numNodes, so the JobSet's node job runs four pods. The Torch policy launches training through torchrun and injects PET_NNODES, PET_NPROC_PER_NODE, PET_NODE_RANK, PET_MASTER_ADDR and PET_MASTER_PORT into each node container.

open as a page

In KFP v2, why does a weekly run's ingest step show as cached and reuse last week's articles, and which settings stop it at task, run or compile level?

level: seniorimportance: should knowfreq 9%

basics

~20 s

KFP compiles every task with caching on, and an ingest step whose inputs never change matches last week's execution, so its old output is reused. Disable caching on that task with set_caching_options, or override it per run or at compile time.

open as a page

When migrating a Kubeflow Training Operator v1 PyTorchJob manifest to a Kubeflow Trainer v2.3.0 TrainJob, what changes, and where do its per-pod settings go?

level: seniorimportance: should knowfreq 6%

basics

~20 s

The kubeflow.org/v1 PyTorchJob with Master and Worker pod templates becomes a trainer.kubeflow.org/v1alpha1 TrainJob that names a runtime, with Master plus Worker replicas folded into trainer.numNodes. Pod-level settings move into the runtime's template or a runtimePatches entry.

open as a page

After a new version of a KFP pipeline is uploaded, why does an existing recurring run keep executing the old pipeline, and how do you move its weekly schedule over?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

A KFP recurring run keeps its own copy of the pipeline spec, or a pinned pipeline version, with fixed parameters. Uploading a new version changes neither, so create a recurring run on the new version and disable the old one.

open as a page

Every Trial in a Katib Experiment tuning a Kubeflow TrainJob ends MetricsUnavailable although each TrainJob reaches Complete — what do you check, and in what order?

level: seniorimportance: nice to knowfreq 4%

basics

~10 s

Trace the default StdOut collector: the namespace needs the label katib.kubeflow.org/metrics-collector-injection: enabled, the sidecar wraps only the first node pod's primary container, and that process must print objectiveMetricName=value lines the default metricsFormat regex parses.

open as a page

With the Kubeflow SDK 0.5, what does OptimizerClient().optimize() create on the cluster, and can you kubectl apply an OptimizationJob instead?

level: seniorimportance: nice to knowfreq 3%

basics

~20 s

optimize() creates a Katib kubeflow.org/v1beta1 Experiment whose trial template is a Kubeflow Trainer TrainJob, and returns its name. OptimizationJob is the SDK's name for that object and a design proposal, KEP-3562, in Trainer v2.3.0; no such kind is installed.

open as a page