Rotating which key leaves stored ciphertext untouched, and rotating which one forces you to rewrite every row?
answer
- two keys, two very different bills
- count objects or count bytes
- the payload is never opened
- unwrap and wrap the small key
- the data key itself is unchanged
basics
~20 sRotating the key that wraps each object's data key is a rewrap: only the small wrapped keys change and the ciphertext is never read. Rotating the key the rows are encrypted under means decrypting and re-encrypting every row.
solid answer
~50 sWhere each object is encrypted under its own data key and that data key is stored wrapped by a managed key, there are two rotatable keys with very different bills. Rotating the **wrapping** key is a *rewrap*: unwrap each stored data key under the version that wrapped it, wrap it again under the new version, write back a few hundred bytes. The payload is never read, so the work is proportional to the number of objects. Rotating the **data** key means the rows themselves must be decrypted and re-encrypted - work proportional to total bytes, plus the read amplification and the rewrite. Where rows are encrypted directly under managed key versions there is no rewrap available at all: moving a row to a new version is a full backfill, and that backfill is the job people discover they have signed up for.
code
pseudocode · 13 lines# rotating the key that WRAPS per-object data keys
newWrapVersion = manager.rotate(wrappingKeyName)
for each object in store:
dataKey = manager.unwrap(object.wrappedDataKey) # routed by wrappedDataKey.keyVersion
object.wrappedDataKey = manager.wrap(dataKey, newWrapVersion)
# object.ciphertext is never read and never written
# rotating the key the ROWS are encrypted under
newDataVersion = manager.rotate(dataKeyName)
for each row in store:
plaintext = manager.decrypt(row.ciphertext) # routed by row.keyVersion
row.ciphertext = manager.encrypt(plaintext, newDataVersion)
row.keyVersion = newDataVersiongo deeper
Recall that there can be two keys in play: a small key that encrypted the object, and a managed key that protects that small key. Rotating the managed one rewrites something tiny; rotating the one over the bytes rewrites everything.
Explain the cost model: a rewrap scales with the number of objects and never opens the payload, while re-encryption scales with total bytes read and written. Be able to say which rotation a team actually performed when it finished suspiciously fast.
Demonstrate you have run the expensive one: an idempotent, resumable, rate-limited backfill driven by the rows' own version markers, with a plan for rows written while it runs and an honest statement about backups and archives it does not reach.
Own the structural call: whether objects get their own data keys at all decides whether every future rotation is an afternoon or a funded campaign. Decide it before the corpus exists, because retrofitting it is itself the expensive backfill.
## Two rotations wearing the same word Rotation is one word covering two operations whose costs differ by orders of magnitude. The distinction only exists where key material is layered: an object is encrypted under its own **data key**, and that data key is not stored in the clear - it is stored *wrapped* by a managed key that the manager holds. Two keys, two rotations. - **Rotating the wrapping key** produces a new version of the key that protects the stored data keys. Each object's wrapped data key is unwrapped under the version that produced it and wrapped again under the new one. The encrypted payload is never opened, never read and never rewritten. - **Rotating the data key** changes the key the bytes themselves were encrypted under. There is no shortcut: every row must be decrypted under its old version and re-encrypted under the new one. ## Why one is cheap A wrapped data key is small and fixed-size. A rewrap therefore costs *number of objects x a small constant*, and it is the same constant whether an object holds a kilobyte or a terabyte. That is the property people mean when they say a rotation over a huge corpus finished in minutes: what finished was a rewrap of the wrapped keys, not a re-protection of the data. Re-encryption costs *total bytes*, twice: once read, once written. On a warehouse holding seven years of columns that is the largest batch job the team will run that year, and it competes with the warehouse's own load. | | Rewrap the wrapping key | Re-encrypt under a new data key | |---|---|---| | work scales with | number of objects | total stored bytes | | the payload is | never read | decrypted and rewritten | | what moves version | the wrapped data key | the ciphertext itself | | what is unchanged | the data key, and the bytes | nothing about the row | | typical duration | minutes to hours | a campaign measured in nights | ## When there is nothing to rewrap The cheap path is not always available. Where rows were encrypted directly under managed key versions - no per-object data key in between - the only way to move a row onto a newer version is to rewrite the row. That single structural choice, usually made years earlier and often by whoever set up the table, decides whether a future rotation is an afternoon or a quarter. ## The backfill you take on otherwise If you do choose to move the corpus, treat it as a real campaign rather than a script: 1. **Make it idempotent from the data.** A row already carrying the target version is skipped by its own marker; nothing external tracks progress. 2. **Make it resumable.** It will be stopped, and restarting from the beginning is not an option on seven years of partitions. 3. **Rate-limit it.** It is competing with production reads and writes on the same storage. 4. **Decide what happens to rows written while it runs.** Partitions closed before the job started are final; anything still being written has to be swept again at the end. 5. **Prove completion from the markers,** not from the job's exit code - a census of version markers is the only evidence that no row is left behind. And be honest about scope: backups, read replicas, exports and archives hold ciphertext at whatever version was current when they were taken. A backfill of live storage does not move them, which matters the moment anyone wants to retire the old version. ## What a rewrap does not buy This is where the direction gets stated backwards most often. A rewrap changes **which key version protects the data key**. It does not change the data key, and it does not change a single byte of the ciphertext. So: - anyone who ever obtained that object's data key can still decrypt that object, rewrap or no rewrap; - the algorithm the payload was encrypted with is unchanged; - the object's ciphertext is byte-identical before and after. What the rewrap does buy is real but narrower: the stored data keys now depend on a fresh version of the wrapping key, so the older wrapping version stops being something the estate depends on and becomes retirable. That is a custody improvement one level up - not a re-protection of the data. The sentence to keep straight: **rotating the outer key is a rewrap that touches no data; rotating the inner key is a re-encryption that touches all of it.** Stated the other way around it is still grammatical, still confident, and wrong.
- A rotation over a very large corpus finished in minutes. What can you conclude about it?That it was a rewrap of wrapped data keys, or that it moved a pointer and rewrote nothing at all. What it cannot have been is a re-encryption of the corpus, which is bounded below by the time to read and write every byte. The follow-up question worth asking is which version the stored rows now name - often, unchanged.
- Does a rewrap make the previous wrapping key version retirable?Once every wrapped data key the estate can still be asked to unwrap has been rewrapped, yes - that is the point of doing it. The catch is the word every: backups, replicas and archives carry wrapped keys at the version current when they were taken, so the old wrapping version stays needed until those copies are past the horizon where anyone would restore them.
- Why does a backfill need to be idempotent from the data rather than from a progress table?Because the version marker on each row already says whether that row is done, and it survives the job crashing, being restarted, or being run twice in parallel. A separate progress table is a second source of truth that can disagree with the data, and when it does, you cannot tell which rows were actually moved.
The contents of a safe are locked with a small key, and that small key is kept in a second, outer safe. Changing the outer safe's lock does not mean re-packing the contents - but it also does not change the small key, so anyone who copied it can still open the contents.
saying these in an interview costs you the question
- Says rotating the wrapping key re-encrypts the stored data
- Says rotating the data key is cheap because ciphertext is untouched
- Thinks a rewrap changes the bytes of the stored object
- Believes a rewrap stops a holder of the data key from decrypting
- Assumes every design offers a rewrap path
- Forgets that backups and archives hold old versions after a backfill