You add a replication rule to an Amazon S3 bucket that already holds millions of objects. New uploads appear in the destination bucket within minutes, but none of the pre-existing objects ever show up. Why, and how do you copy the backlog?
answer
- rules are forward-looking, not retroactive
- new objects fine, history untouched
- a batch job replays the same rule
- generated or supplied manifest, plus completion report
- priced per object — size it first
basics
~20 sAn S3 replication rule only applies to objects written after the rule is saved; it never walks the existing contents of the bucket. To copy the backlog you run S3 Batch Replication, an S3 Batch Operations job that replicates existing object versions using the same rule.
solid answer
~50 sThis is expected behaviour, not a misconfiguration — the fact that new uploads replicate proves the rule, the versioning and the IAM role are all correct. S3 replication is event-driven: it reacts to writes that happen after the configuration is in place, and it does not scan the bucket for what was already there. For the backlog you use **S3 Batch Replication**, an S3 Batch Operations job that replicates existing objects according to the same replication rules. You can let S3 generate the manifest for you or supply your own manifest of specific versions, and the job produces a completion report so you can see what succeeded and what failed. The same mechanism also re-drives objects whose replication previously failed, and replicates objects that themselves arrived by replication from a third bucket. Budget for it: it is billed per object processed plus the usual replication request, transfer and storage charges.
code
json · 16 lines{
"Role": "arn:aws:iam::111122223333:role/s3-replication-role",
"Rules": [
{
"ID": "replicate-reports-to-dr",
"Priority": 1,
"Status": "Enabled",
"Filter": { "Prefix": "reports/" },
"DeleteMarkerReplication": { "Status": "Disabled" },
"Destination": {
"Bucket": "arn:aws:s3:::example-dr-bucket",
"StorageClass": "STANDARD_IA"
}
}
]
}go deeper
Remember the headline rule: an S3 replication rule copies objects written after you save it, never the ones already in the bucket. Name S3 Batch Replication as the tool for the backlog.
Explain that Batch Replication is an S3 Batch Operations job replaying the same replication configuration, that manifests can be generated or supplied, and that completion reports tell you what failed.
Diagnose before configuring: use the object's replication status to prove nothing was ever attempted, then plan the backfill in slices so it does not starve live replication, and state the per-object and storage costs up front.
Frame it as a data-migration exercise with a budget and a schedule: which datasets deserve a backfill at all, what the doubled storage costs over the retention period, and whether a filtered rule targeting only current data meets the requirement more cheaply.
## Why nothing happened S3 replication is triggered by writes. When a replication configuration is saved on a bucket, S3 begins evaluating **subsequent** object-version creations against the rules; it does not enumerate the objects that were already sitting in the bucket. So a rule added to a bucket with ten years of history replicates only what lands from that moment on. The symptom in the question is actually a healthy signal. If new uploads are replicating, then all three prerequisites are already satisfied: versioning is on at both ends, the IAM role exists and is trusted by S3, and the rule's filter matches. The only thing missing is a mechanism to walk history. ## The fix: S3 Batch Replication **S3 Batch Replication** is an S3 Batch Operations job whose operation is object replication. It does not define its own copy semantics — it replays existing object versions through the bucket's existing replication configuration, so filters, destination, storage-class override and ownership override all behave exactly as they do for live traffic. It handles three populations: - objects that existed **before** the replication rule was created; - objects whose replication previously **failed** (for example while the destination bucket policy was wrong); - objects that were themselves **replicas** created by replication from another bucket, which live replication does not chain onward by default. You can let S3 **generate the manifest** for you — it produces the list of eligible objects from the bucket and shows an estimate before you confirm — or you can supply your own manifest listing specific keys and version IDs, which is what you want when only a known slice needs backfilling. The job runs with an IAM role, reports progress, and writes an optional **completion report** to a bucket you choose so you can audit successes and failures per object. ## Costs and blast radius Backfilling millions of objects is not free, and an interviewer will expect you to say so unprompted: - a per-object charge for objects processed by the Batch Operations job, plus a job charge; - the ordinary replication **request** charges for each replicated object; - **data-transfer** charges if the destination is in another Region; - and the destination **storage** cost, which for a full backfill roughly doubles what you are paying for that dataset. There is also a throughput consideration: a large backfill competes with live replication traffic. If the backlog is enormous and the live stream is latency-sensitive, it is reasonable to backfill in slices — by prefix, or by manifest — rather than in one job. ## Related sharp edges in the same family While you are looking at what did not replicate, a few adjacent rules explain most of the remaining surprises: - Objects already stored in the **S3 Glacier Flexible Retrieval** or **S3 Glacier Deep Archive** storage class in the source are not replicated. - Objects created in the source **by replication from another bucket** are not replicated onward by a live rule; Batch Replication is the way to fan them out. - Actions performed by **S3 Lifecycle** on the source are not mirrored to the destination — the destination has its own lifecycle configuration, or none. ## Checking a specific object Every object version carries a replication status, exposed as the `x-amz-replication-status` header and as `ReplicationStatus` on a `HeadObject` response. On the source it reads `PENDING`, `COMPLETED` or `FAILED`; on the destination copy it reads `REPLICA`. A pre-existing object that was never eligible for replication simply has **no** replication status at all — which is itself the diagnostic that distinguishes "never attempted" from "attempted and failed". ```bash aws s3api head-object --bucket example-source --key legacy/report.parquet \ --query ReplicationStatus ``` An empty result for old objects and `COMPLETED` for new ones is exactly the fingerprint of this scenario, and it is a much faster answer than reading the rule JSON again. ## How to say it in an interview "Replication is forward-looking by design. New objects replicate, history does not. I would first confirm with `HeadObject` that old objects carry no replication status at all, which rules out a permissions problem, then run an S3 Batch Replication job — generated manifest if I want everything, my own manifest if I want a slice — and size the cost before pressing go, because it is priced per object and it doubles the storage for that dataset."
- How would you confirm the backlog was never attempted, rather than attempted and denied?Call `HeadObject` on an old object and look at `ReplicationStatus`. Objects that were never eligible carry no replication status at all, while objects the role could not read or write show `FAILED`. That single check separates "the rule came later" from "the role or destination policy is wrong" before you touch any configuration.
- Besides pre-existing objects, what else can an S3 Batch Replication job replicate?It also re-drives objects whose replication previously failed — useful after fixing a destination bucket policy or KMS key policy — and it replicates objects that arrived in the source by replication from a third bucket, which live rules do not chain onward. In all cases it replays the bucket's existing replication configuration rather than defining new behaviour.
- Are there objects a backfill still will not move?Yes. Objects already in the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage class in the source are not replicated, so an archive-heavy bucket needs a restore-and-copy strategy instead. Anything excluded by the rule's own prefix or tag filter is also skipped, since the batch job honours the same filters as live replication.
saying these in an interview costs you the question
- Assuming a new rule backfills the whole bucket automatically
- Blaming the IAM role when new objects replicate fine
- Suggesting a manual aws s3 sync as the only option
- Not knowing S3 Batch Replication exists
- Ignoring that a backfill doubles storage cost for the dataset