skip to content

A nightly job that has always read files with a plain S3 GetObject call starts failing with an InvalidObjectState error, shortly after a lifecycle rule was added to the bucket. What happened, and what are your options for fixing it?

level: seniorimportance: should knowfreq 56%

answer

  1. the object exists but is not online
  2. a class, not a permission, is refusing you
  3. fetching it is a request, then a wait
  4. the archived original never moves
  5. there is a millisecond archive class

basics

~20 s

The lifecycle rule transitioned those objects into S3 Glacier Flexible Retrieval or Deep Archive, which cannot be read by GetObject at all. You must first issue a RestoreObject request and wait, or move the data to S3 Glacier Instant Retrieval, which serves GETs directly.

solid answer

~50 s

`InvalidObjectState` is S3 telling you the object is in an archived class — `GLACIER` (Glacier Flexible Retrieval) or `DEEP_ARCHIVE` — where the bytes are not directly readable. The fix depends on the access requirement. If the job genuinely needs the data on demand, the archive class was the wrong choice: `GLACIER_IR` gives millisecond GETs at a lower storage rate than Standard-IA, with no restore step. If archival really is right and this read is rare, the job must call `RestoreObject` first, choose a retrieval tier — Expedited, Standard or Bulk, trading minutes against hours against price — poll until the restore completes, then GET the temporary copy. For a whole prefix you drive the restores with S3 Batch Operations rather than a loop. Either way the code has to stop assuming a GET always works.

code

bash · 11 lines
bash
# Ask S3 to make an archived object readable for 3 days
aws s3api restore-object \
  --bucket archive-bucket \
  --key reports/2025-01-01.parquet \
  --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}'

# Poll until ongoing-request flips to false
aws s3api head-object \
  --bucket archive-bucket \
  --key reports/2025-01-01.parquet \
  --query 'Restore'

go deeper

for a junior

Know that objects in the Glacier Flexible Retrieval and Deep Archive classes cannot be downloaded directly — you first request a restore and wait, and that is what an InvalidObjectState error is telling you.

for a middle

Explain the restore mechanics: RestoreObject with a Days window and a tier, an asynchronous wait you poll or subscribe to, a temporary copy while the object keeps its archive class, and the s3:RestoreObject permission it needs.

for a senior

Show the judgment that prevents this: enumerate the readers of a prefix and their latency tolerance before archiving it, pick Glacier Instant Retrieval when anyone may need on-demand access, and rehydrate at scale with S3 Batch Operations rather than a loop.

for a principal

Own the retention contract itself — which datasets may go to a class that cannot be read on demand, what recovery time that commits the organisation to during an audit or incident, and who signs off when a cost rule changes an access guarantee.

## What the error means `InvalidObjectState` from `GetObject` means the object exists, you are allowed to read it, and S3 still will not hand it over — because its storage class is one where the data is not kept online. That is S3 Glacier Flexible Retrieval (`GLACIER`) and S3 Glacier Deep Archive (`DEEP_ARCHIVE`). Objects in those classes are archived: you can list them, read their metadata, delete them, but a plain GET fails until a restore has been performed. This is the classic aftermath of a well-intentioned cost rule. Someone wrote a lifecycle staircase ending in Glacier at 90 days, nobody checked which readers touch data older than 90 days, and the failure arrives a quarter later with no code change to blame. ## The restore workflow Restoring is asynchronous and explicit: 1. Call `RestoreObject` (`aws s3api restore-object`) with a `Days` value — how long the retrieved copy stays available — and a retrieval tier. 2. Poll. `HeadObject` reports an `x-amz-restore` header that reads `ongoing-request="true"` while the restore is running and switches to `ongoing-request="false", expiry-date="..."` when the copy is ready. S3 can also fire an event notification on restore completion so you do not have to poll at all. 3. GET normally, for as many days as you requested. Crucially the object does **not** change storage class. What you get is a temporary readable copy alongside the archived original; when the requested days elapse the copy disappears and you are back to needing a restore. You pay for both the archived object and, for the restore window, the retrieved copy. ## Retrieval tiers For Glacier Flexible Retrieval you choose among **Expedited** (minutes, most expensive, and optionally backed by provisioned capacity if you need it to be guaranteed), **Standard** (a few hours, the default), and **Bulk** (slower, cheapest — free for this class). Deep Archive offers Standard (around half a day) and Bulk (up to two days), and no Expedited tier at all. Getting the tier wrong is how a "quick check" turns into a two-day wait during an incident. ## The three real fixes **Change the class.** If a job needs this data on a schedule, it is not archival data. `GLACIER_IR` — Glacier Instant Retrieval — stores at an archive-ish rate with millisecond first-byte latency and no restore step, in exchange for a 90-day minimum storage duration, a 128 KB minimum billable size, and a per-GB retrieval charge. It exists precisely for "archived, but occasionally read right now" data. Amending the lifecycle rule to target `GLACIER_IR` instead of `GLACIER` makes the job work again, and you copy the already-archived objects back to the new class after restoring them once. **Make the reader archive-aware.** If the read really is exceptional — a legal request, a once-a-year reprocessing — then teach the job to restore and wait. Handle `InvalidObjectState` explicitly, issue the restore, and resume from a completion event rather than blocking a worker for four hours. **Batch it.** Restoring a million objects with a loop of API calls is slow and fragile. S3 Batch Operations takes an S3 Inventory manifest and runs `RestoreObject` across every object in it, with retries, a completion report and a job-level IAM role — the supported way to rehydrate a prefix. ```bash aws s3api restore-object \ --bucket archive-bucket \ --key reports/2025-01-01.parquet \ --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}' ``` ## Permissions and the near-miss diagnosis The caller needs `s3:RestoreObject` in addition to `s3:GetObject`; a role scoped only to reads will fail the restore with an authorization error, which is a different failure from `InvalidObjectState` and worth distinguishing out loud. Also worth distinguishing: a KMS key problem returns an access or key-state error, not `InvalidObjectState`, so the error name itself is the diagnosis here. ## The lesson to state Archive classes are not "cheap S3" — they are a different access contract. Before a lifecycle rule sends data to `GLACIER` or `DEEP_ARCHIVE`, you have to know every reader of that prefix and the worst-case latency each can tolerate. When the answer is "someone might need it quickly", the correct class is Glacier Instant Retrieval, not a faster restore tier.

  • After a successful restore, what storage class does the object report?
    Still the archive class it was in. RestoreObject creates a temporary readable copy for the number of days you requested; the archived original is untouched and the class is unchanged. When the window expires the copy vanishes and a GET fails again, so a permanent fix means copying the object into a directly-readable class.
  • How would you rehydrate an entire prefix of several million archived objects?
    With S3 Batch Operations driven by an S3 Inventory manifest, running RestoreObject across the manifest under a job IAM role. It handles retries, throttling and a completion report. Hand-rolling a loop of API calls is slow, easy to abandon halfway, and gives you no record of which objects succeeded.
  • When would you pick Glacier Instant Retrieval over Glacier Flexible Retrieval?
    When the data is cold but occasionally needed immediately — compliance archives someone may query, older media served on demand. Instant Retrieval costs more per GB than Flexible Retrieval and carries a 90-day minimum duration and a retrieval fee, but GETs work directly with millisecond latency, so no reader needs restore logic.

saying these in an interview costs you the question

  • Thinks InvalidObjectState is an IAM permissions problem
  • Believes a restore permanently moves the object to S3 Standard
  • Assumes GET on a Glacier object just returns slowly instead of failing
  • Treats Deep Archive as offering an Expedited retrieval tier
  • Plans to restore millions of objects with a loop of single API calls

context

resolving links…