In Prefect, why can a persisted task result be unreadable to a worker on another machine?

## States hold references, not data Prefect's orchestration layer is metadata: run records, state transitions, logs, artifacts. When a task returns a value, the value itself is not what the API stores. If the result is persisted, Prefect writes it to **result storage** and records a *reference* to it on the state. Everything that later "reuses" a task's output — a cache hit, a downstream task resolving a future in another process, an inspection call on a state — is resolving that reference. That design is why Prefect can be a hybrid system: your data stays in storage you control, while the API sees only where it is. ## The three knobs - `persist_result` — whether the return value is written to storage at all. Off, the value lives only in memory for the duration of the run; nothing outside that process can retrieve it later. - `result_storage` — *where* it is written: a filesystem block such as `LocalFileSystem`, or a remote block such as `S3Bucket`, GCS or Azure storage. - `result_serializer` — *how* it is encoded: pickle by default, or JSON, or a custom serializer. All three can be set per task, per flow (children inherit), or as an installation default. ```python from prefect import flow, task from prefect_aws import S3Bucket store = S3Bucket.load("prefect-results") @task(persist_result=True, result_storage=store, result_serializer="json") def extract(day: str) -> dict: ... @flow(result_storage=store) def pipeline(day: str): extract(day) ``` ## Why local storage bites in production The default store is a directory under the Prefect home on the executing machine. On a laptop, every run shares that directory, so caching and result loading work and everything feels fine. Then the same code deploys to a work pool where each flow run gets its own container or Kubernetes pod, and: - A cache hit computed by yesterday's pod points at a path in a filesystem that no longer exists. - A run retried on a different node cannot read what the first node wrote. - Two concurrent workers each build their own private, non-shared cache, so the hit rate collapses without any error message. The symptom is confusing precisely because the *state* is intact — the API cheerfully reports Completed and Cached — while the payload is gone. The failure surfaces as an error loading the result, or as work being silently redone. ## Fixing it Point `result_storage` at a store every executor can reach. A bucket block is the usual answer; a shared network filesystem works if you already run one. Set it once at the flow level (or as the default for the deployment) rather than sprinkling it across tasks, so a new task does not silently opt back into local storage. `result_storage_key` lets you template the object name — including run or parameter values — when you want predictable, greppable paths rather than opaque UUIDs. ## Serializers are the second trap Pickle is the default because it round-trips arbitrary Python objects, and that is exactly its weakness for stored results: unpickling needs a compatible interpreter and compatible versions of every library involved in the object graph. A result written by a container running one pandas version and read by a container running another can fail to load, and a pickle read from storage is code execution — a real consideration if the store is shared across teams. JSON is portable, inspectable and safe, at the price of only handling serializable shapes. The pragmatic rule: return small, plain data structures from tasks and serialize them as JSON; keep big frames in the warehouse or object store and return a pointer. ## What this means for caching and retries Caching depends entirely on this machinery — a cache key resolves to a stored result, so a cache without reachable storage is just a way to skip work and then fail. In Prefect 3, declaring a cache policy switches persistence on for that reason. Retries within a run do not need persistence (the process still holds memory), but an externally re-run flow benefits enormously from it: previously completed tasks can be served from cache rather than recomputed, which is what makes a long pipeline resumable rather than all-or-nothing. ## Security and the hybrid model Because only references and metadata reach the API, persisted result data never leaves your infrastructure for Prefect Cloud. Do not undo that by hand: parameters, log lines and artifacts *are* metadata and do travel, so a task that logs a full row or accepts a secret as a plain parameter leaks what the result design was protecting. Keep secrets in blocks, keep payloads in result storage, and keep the API surface to counts, keys and pointers. ## Checklist for a review Is persistence on where caching or resumability is expected? Is `result_storage` a shared block rather than the local default? Is the serializer appropriate for how long the result must live and who reads it? Are results small enough that storing them is cheaper than recomputing? Is there a retention story, or will the bucket grow forever?