In the KFP v2 SDK, how do @dsl.component, @dsl.pipeline and the compiler turn Python functions into a run on a Kubeflow Pipelines backend?
answer
- a graph, not a function call
- task outputs become the edges
- Compiler().compile writes IR YAML
- kfp.Client files the run in an experiment
- backend turns IR into an Argo Workflow
basics
~20 s@dsl.component makes a typed function a container step, @dsl.pipeline wires steps into a graph through their outputs, Compiler().compile writes that graph as IR YAML, and kfp.Client submits it as a run in which each step gets its own container.
solid answer
~40 s`@dsl.component` turns a type-annotated Python function into a component: an image, a command and a declared set of inputs and outputs. Inside a `@dsl.pipeline` function, calling a component does not run it; it returns a `PipelineTask`, and passing `ingest_task.outputs['articles']` into the next call adds an edge to the graph. `kfp.compiler.Compiler().compile(pipeline_func=..., package_path='article_topics.yaml')`, or `kfp dsl compile`, writes that graph as IR YAML, a platform-neutral `PipelineSpec`. `kfp.Client(host=...)` then submits it with `create_run_from_pipeline_package`, or with `create_run_from_pipeline_func`, which compiles for you, and files the run under an experiment. The backend translates the IR into an Argo Workflow and runs each task in its own container, writing artifacts under the pipeline root in the object store.
code
python · 24 linesimport kfp
from kfp import compiler, dsl
from kfp.dsl import Dataset, Input, Model, Output
@dsl.component(base_image='python:3.11')
def ingest_articles(week: str, articles: Output[Dataset]):
with open(articles.path, 'w') as f:
f.write(f'articles for {week}\n') # stand-in for the newsroom export
@dsl.component(base_image='python:3.11')
def train_classifier(articles: Input[Dataset], model: Output[Model]):
with open(articles.path) as src, open(model.path, 'w') as dst:
dst.write(src.read()) # stand-in for fitting the topic classifier
@dsl.pipeline(name='article-topics-weekly')
def article_topics(week: str):
ingest_task = ingest_articles(week=week)
train_classifier(articles=ingest_task.outputs['articles'])
compiler.Compiler().compile(article_topics, package_path='article_topics.yaml')
client = kfp.Client(host='http://localhost:3000')
client.create_run_from_pipeline_package(
'article_topics.yaml', arguments={'week': '2026-W41'},
experiment_name='article-topics')go deeper
Recall the chain: decorate functions as components, wire them in a pipeline through task outputs, compile to IR YAML, submit with kfp.Client into an experiment.
Explain that the pipeline body builds a graph of PipelineTask placeholders, what IR YAML contains, and the difference between submitting a package and submitting a function.
Show you know where things run: each task in its own container via an Argo Workflow, data through JSON parameters or object-store artifacts, and local runners for fast iteration before a cluster run.
Weigh what the compiled, platform-neutral spec buys a team: reviewable pipeline versions, one artifact store with lineage, and runs anyone can repeat from a pipeline version rather than a notebook.
## Three objects: component, pipeline, compiled spec Kubeflow Pipelines (KFP) v2 splits a pipeline's life into three phases: you author it in Python, compile it to a portable spec, and run it on a cluster. This answer assumes KFP SDK 2.17.0 against a 2.17 backend. Three terms carry the whole flow: - **Component**: a function decorated with `@dsl.component` (or `@dsl.container_component`). Its type annotations declare its inputs and outputs, and the SDK turns it into a container definition: an `image`, a `command` and `args`. Every component executes as a container, Python ones included. - **Pipeline**: a function decorated with `@dsl.pipeline`. Its body calls components and wires the outputs of some into the inputs of others. - **IR YAML**: the compiled form, an instance of the `PipelineSpec` protocol-buffer message. The compiler writes it and the backend reads it. ## What the pipeline function really does A `@dsl.pipeline` body looks like ordinary Python, but it only describes topology. Calling `ingest_articles(week=week)` inside it fetches no articles. The call returns a **`PipelineTask`**, a placeholder for an execution that will happen later on the cluster. Passing `ingest_task.outputs['articles']` (or `.output` for a single unnamed output) into `train_classifier(...)` records a data dependency, and that dependency becomes an edge in the graph. When two steps must run in order without exchanging data, `.after(other_task)` adds the edge explicitly. Task-level settings hang off the same handle: `set_caching_options`, `set_retry`, `set_cpu_limit`, `set_memory_limit`, `set_accelerator_type`, `set_env_variable`. They are recorded in the spec. None of them is applied in your Python process. ## Compiling to IR YAML `kfp.compiler.Compiler().compile(pipeline_func=article_topics, package_path='article_topics.yaml')` type-checks the wiring (`type_check=True` by default) and writes the IR YAML. The CLI equivalent is `kfp dsl compile --py pipeline.py --output article_topics.yaml`. Unlike the KFP v1 component YAML, the file is not meant to be written by hand, but you can still read it: | Section | What it holds | |---|---| | `components` | each component's interface: inputs, outputs, executor reference | | `deployment_spec` | the executors: image, command and args per component | | `root` | the outermost graph, the one that runs when you submit | | `pipeline_info` | the pipeline name, used to group runs and artifacts | | `sdk_version`, `schema_version` | which SDK and which schema produced the file | | `default_pipeline_root` | where artifacts go unless something overrides it | Because the spec is platform-neutral, any conformant backend can accept the same file. ## Submitting a run with kfp.Client A **run** is one execution of a pipeline, and an **experiment** is a named group of runs. With a client connected to the backend, `kfp.Client(host=...)`: 1. `create_run_from_pipeline_package('article_topics.yaml', arguments={...}, experiment_name='article-topics')` submits a file you already compiled. 2. `create_run_from_pipeline_func(article_topics, arguments={...})` compiles to a temporary IR YAML file and submits it in a single call. 3. If you give no experiment, the client looks for an experiment called `Default` (or the name in the `KF_PIPELINES_DEFAULT_EXPERIMENT_NAME` environment variable) and creates it if it is missing. 4. `upload_pipeline` and `upload_pipeline_version` store the spec on the backend as a named, versioned pipeline. The UI and recurring runs can then start runs from it without your local file. Each submission returns a `RunPipelineResult` with a `run_id`, and its `wait_for_run_completion()` blocks until the run ends. ## What the backend does with it The open-source backend compiles the `PipelineSpec` once more, into an Argo Workflow, which is the definition it actually executes. Each task therefore runs as its own container on the cluster, and data never moves through shared memory: - **parameters** (strings, numbers, booleans, lists, dicts) travel as small JSON values; - **artifacts** (datasets, models, metrics) are files the backend copies to the **pipeline root**, a path in an object store; - the UI shows the run graph, each step's logs and its output artifacts. Since KFP 2.15 the default deployment ships SeaweedFS as that object store in place of MinIO. MinIO and any other S3-compatible store remain supported. ## Trying it on a laptop first You can exercise the same code before a cluster is involved. Run `from kfp import local`, then `local.init(runner=local.DockerRunner())`, and call a component or the pipeline like a function. The outputs come back as real values, and artifact files land under a local pipeline root. The docs recommend `DockerRunner`, which gives every task its own container; `SubprocessRunner` is for environments without Docker. Resource settings such as `set_memory_limit` have no effect locally, and nothing is sent to the backend.
- How do you try the same pipeline on a laptop before submitting it?Call `local.init(runner=local.DockerRunner())` from `kfp.local`, then call the component or pipeline like a function. Outputs come back as real values and artifact files under a local pipeline root (`./local_outputs` by default). `DockerRunner` runs each task in its own container and is closest to the cluster; `SubprocessRunner` is for machines without Docker. Resource settings like `set_memory_limit` are ignored locally.
- When would you upload the pipeline rather than submit the compiled file each time?When others, the UI or a schedule need to start runs from it. `upload_pipeline` and `upload_pipeline_version` store the IR YAML on the backend as a named pipeline with versions, so a run or a recurring run can reference a pipeline ID and version ID. `create_run_from_pipeline_package` just sends the spec with one run and leaves nothing reusable behind.
- Calling a component inside the pipeline function runs its code right there
- The compiler's output is an Argo Workflow you apply with kubectl
- All steps of a run share one Python process and pass objects in memory
- IR YAML is meant to be hand-written like a v1 component YAML
- A run cannot be submitted until an experiment has been created by hand
Assumes KFP SDK 2.17.0 with a 2.17 backend; SeaweedFS replaced MinIO as the default object store deployment in KFP 2.15.