skip to content

A team is about to hand-build and maintain three separate index tables over a dataset stored in DynamoDB, to support queries by three different non-key attributes. What native DynamoDB feature should they evaluate first before committing to hand-rolled index tables, and under what circumstances would hand-rolled index tables still be the right call despite that feature existing?

level: principalimportance: nice to knowfreq 35%

answer

  1. GSI = DynamoDB-managed secondary index, auto-propagated
  2. up to 20 GSIs per table
  3. GSIs are eventually consistent only, no strongly-consistent GSI reads
  4. hand-roll only for: no native index, need strong consistency, computed/derived keys, cross-table/region isolation, GSI limit exceeded
  5. DynamoDB Streams can feed a hand-rolled index as a hybrid

basics

~20 s

DynamoDB has a built-in feature called a Global Secondary Index that does most of this automatically — the database keeps it in sync for you. You'd only build your own index table by hand if you need something the built-in version can't do, like guaranteed instant consistency or very complex derived keys.

solid answer

~50 s

They should evaluate DynamoDB's Global Secondary Indexes (GSIs) first — up to 20 per table, each with its own partition/sort key, automatically and asynchronously kept in sync by DynamoDB itself with no application code required to maintain them. For three straightforward attribute lookups this is almost always the right default: less code, no hand-rolled consistency mechanism, no reconciliation jobs. Hand-rolled index tables remain justified when: the store doesn't support native secondary indexes at all (e.g., Azure Table Storage); you need strongly/read-your-writes consistent lookups that a GSI's inherent eventual consistency can't provide; the index key needs to be a computed/derived value beyond what a GSI's simple projection supports; you need the index data to live in a different table/account/region with independent scaling or access control; or you've already exceeded the per-table GSI limit and need additional index dimensions.

go deeper

for a junior

Not expected to know GSIs by name; credit for recognizing that 'maybe the database already has a built-in way to do this' is worth checking before building something custom.

for a middle

Should know that DynamoDB offers GSIs as a built-in alternative and that it removes the need for application-level dual writes.

for a senior

Should be able to name at least the eventual-consistency limitation of GSIs as a reason to still hand-roll in specific cases.

for a principal

Should give a structured decision framework — evaluate the native feature first, name multiple concrete disqualifying circumstances (consistency, computed keys, GSI limits, isolation needs), and recognize hybrid approaches like Streams-fed hand-rolled tables.

## What a GSI already does for you The first move here is recognizing that DynamoDB — unlike the classic Azure Table Storage case the index table pattern was originally documented against — actually does offer a native answer to 'I need to query by an attribute that isn't my primary key': the Global Secondary Index (GSI). A GSI is a second, DynamoDB-managed index on the same table with its own partition key and optional sort key, drawn from attributes already present on the base items, plus an optional projection of which attributes get copied into it. Once defined, DynamoDB itself is responsible for propagating every base-table write into the GSI — the application writes to the base table exactly once, using ordinary `PutItem/UpdateItem/DeleteItem` calls, and DynamoDB asynchronously derives and maintains the GSI's contents behind the scenes. A table can have up to 20 GSIs (a service-level default that can, in some accounts, be raised), each independently capacity-provisioned (or covered by on-demand billing). ## Why it fits this scenario For the scenario described — three non-key attributes needing lookup support — this is close to a textbook case for GSIs over hand-rolled index tables, and the reasoning is almost entirely about what you get to delete from your own responsibility list: - No application code writes to the index. - There's no dual-write logic, no transactional-scope puzzle, no outbox pattern, no reconciliation job. - None of the failure modes discussed earlier around orphaned or dangling index entries caused by a crash between two independent writes — DynamoDB's internal replication to the GSI is atomic with respect to the base write from the application's point of view (the base write either succeeds, in which case propagation to the GSI is DynamoDB's guaranteed job, or it doesn't happen at all). The team gets three query paths essentially for the cost of three GSI definitions and their provisioned/on-demand capacity, not three hand-built subsystems. ## When hand-rolling still wins — consistency That said, GSIs are not a strict superset of what a hand-rolled index table can do, and there are specific, real circumstances where the hand-rolled version is still the better engineering call. The most commonly cited one is consistency: DynamoDB GSIs are inherently eventually consistent — there is no way to request a strongly consistent read against a GSI, unlike the base table or a Local Secondary Index, which do support strongly consistent reads. If a query path needs guaranteed read-your-own-writes behavior immediately after a write (a security-sensitive uniqueness check on signup, for instance, where you cannot tolerate a race where a duplicate email briefly evades detection), a GSI's propagation lag — typically sub-second but not bounded or eliminable — may be unacceptable, and a hand-rolled index table maintained via a transactional write into the same partition as the base item can provide the stronger guarantee a GSI structurally cannot. ## When hand-rolling still wins — key complexity Second is key complexity: a GSI's key must be drawn from actual item attributes (with limited support for sparse indexing based on attribute presence), whereas a hand-rolled index table's key can be an arbitrary computed value — a normalized/lowercased email, a composite bucket derived from multiple fields with custom business logic, a value that depends on external state — anything the application can compute at write time. If the index key isn't a simple projection of existing attributes, a GSI can't express it and a hand-rolled table (or a hand-rolled attribute the application computes and stores on the item specifically so a GSI can key off it) becomes necessary. ## When hand-rolling still wins — scale and isolation Third is scale and isolation limits: - hitting the per-table GSI cap; - needing the index data physically isolated in a different table, account, or region for independent scaling, security boundary, or cost-allocation reasons; - or needing the index consumed by an entirely separate service with its own access pattern that shouldn't share the base table's throughput budget. All push toward a separately maintained index table, potentially fed by DynamoDB Streams rather than application dual-writes, which is itself a hybrid: DynamoDB-native change capture doing the propagation work, but landing in a hand-rolled destination table. ## The principle to carry The general principle a principal-level engineer should carry into this decision is: prefer the native secondary-index mechanism whenever the store offers one and its consistency/key/scale model fits the requirement, and reserve the hand-rolled index table pattern — with its associated write-amplification, consistency-strategy, and failure-mode costs covered elsewhere — for the store that has no native equivalent (Azure Table Storage, many key-value caches) or for the specific requirement a native index can't satisfy. Building three hand-rolled index tables on a store that already offers GSIs, without first ruling out GSIs on their merits, is a design smell worth pushing back on in review.

  • Why can't a GSI provide a strongly consistent read the way the base table can?
    Because DynamoDB propagates writes from the base table to a GSI asynchronously, behind the scenes, rather than as part of the same synchronous write path — that propagation delay is exactly what makes strongly consistent reads structurally impossible against a GSI, whereas the base table and Local Secondary Indexes are updated synchronously with the write and so can support it.
  • If a team needs an index keyed by a lowercased, trimmed version of an email address, can a GSI do that directly?
    Not directly — a GSI's key must be an actual attribute already on the item, not a computed transformation of one. The workaround is for the application to compute and store the normalized value as its own attribute on the base item at write time, and then key the GSI off that stored attribute, which still avoids hand-rolling a separate index table.
  • What's a reasonable hybrid approach if a team has exceeded DynamoDB's GSI limit but still needs more index dimensions?
    Keep using GSIs for the dimensions they already cover, and for the additional ones, consume DynamoDB Streams to propagate base-table changes into a separately maintained index table — this still avoids application code doing dual writes and the associated crash-consistency risk, while working around the per-table GSI cap.

It's like a library that just added a computerized catalog terminal that automatically updates itself whenever a book is shelved, versus keeping your own hand-written card index. If the terminal covers what you need, maintaining your own cards by hand is pure extra work — you'd only keep doing it for something the terminal genuinely can't do, like an instantly-guaranteed-accurate index for a security check at the front desk.

saying these in an interview costs you the question

  • Doesn't know DynamoDB has a native secondary-index feature at all and jumps straight to hand-rolling
  • Claims GSIs provide the same consistency guarantees as the base table
  • Recommends hand-rolled index tables as a default 'best practice' regardless of the store's native capabilities
  • Can't name a single concrete circumstance where hand-rolling is still justified despite GSIs existing
  • Confuses a Global Secondary Index with a Local Secondary Index or doesn't know the difference matters (LSI must be created at table creation time and shares partition key; GSI is more flexible)

context