An AWS Lambda handler calls Secrets Manager on every invocation to fetch a database password. What problems does that create, and how would you fix it without hard-coding the secret?

## Why the naive version is a problem Fetching inside the handler means one extra network call, plus TLS and signing work, on the critical path of every single request. At low traffic it is a barely noticeable tens of milliseconds. It stops being harmless as volume rises, in three separate ways. **Latency.** The added round trip is pure overhead on a request that the caller is waiting on. For a function fronting an API, it is a fixed tax on every p50. **Cost.** Secrets Manager bills per API call as well as per stored secret. A function doing a million invocations a day is doing a million credential lookups a day for a value that changed last quarter. **Throttling — the one that actually pages you.** `GetSecretValue` is subject to an account-wide request rate quota. Concurrency multiplies the call rate directly: a thousand concurrent executions each fetching on entry produce a burst of a thousand near-simultaneous calls. When you cross the quota, Secrets Manager returns `ThrottlingException`, the handler cannot obtain a password, and the failure surfaces as "the database is down" even though the database never noticed. The tell is that the failure rate rises with traffic and disappears when traffic falls — a self-inflicted, load-correlated outage. ## The fix: fetch once per execution environment A Lambda execution environment serves many invocations over its lifetime. Code at module scope — outside the handler function — runs once, when the environment initialises, and anything it puts in a module-level variable survives into every subsequent invocation that environment handles. That is the natural place for a credential fetch. ```python import os, time, json, boto3 _sm = boto3.client("secretsmanager") _cache = {} _TTL_SECONDS = 300 def _get_secret(secret_id): entry = _cache.get(secret_id) now = time.monotonic() if entry and now - entry[1] < _TTL_SECONDS: return entry[0] value = json.loads(_sm.get_secret_value(SecretId=secret_id)["SecretString"]) _cache[secret_id] = (value, now) return value def handler(event, context): creds = _get_secret(os.environ["DB_SECRET_ID"]) ... ``` The call rate collapses from once per invocation to roughly once per environment per TTL, which is orders of magnitude smaller and, crucially, no longer proportional to traffic. ## Why the cache needs a TTL The obvious version — fetch once, keep forever — trades one problem for another. Warm environments can survive for a long time, so a function that cached at start-up may still be holding a password that was rotated an hour ago. Every invocation on that environment then fails to authenticate while a freshly created environment works fine, producing intermittent, environment-dependent failures that are miserable to reproduce. Two defences, ideally both: - **A TTL** measured in minutes. It bounds staleness to something you can reason about and still removes essentially all of the call volume. - **Invalidate on failure.** Catch the database's authentication error, clear the cache entry, refetch once, and retry. This handles rotation the moment it bites rather than waiting out the TTL. Guard it so a genuinely wrong credential cannot become a retry loop. If the secret is rotated on a schedule you control, you can also stage the change: Secrets Manager's rotation keeps the previous version usable for a window, so both old and new credentials authenticate while caches turn over. ## The managed alternative AWS publishes the **Parameters and Secrets Lambda Extension** as a layer. It runs alongside your function and exposes a small local HTTP cache, so your code makes a localhost call instead of an AWS API call: ```bash curl -s -H "X-Aws-Parameters-Secrets-Token: ${AWS_SESSION_TOKEN}" \ "http://localhost:2773/secretsmanager/get?secretId=prod/db/password" ``` It listens on port 2773 by default (`PARAMETERS_SECRETS_EXTENSION_HTTP_PORT`), and TTLs are configured with `SECRETS_MANAGER_TTL` and `SSM_PARAMETER_STORE_TTL`. The token header is required — the extension refuses unauthenticated local requests, which stops other code in the environment from trivially reading cached secrets. The trade is a layer and a little memory in exchange for not maintaining cache code in every runtime you use. Powertools for AWS Lambda offers an equivalent in-process utility for teams that prefer a library. ## Permissions, which people forget Whichever route you take, the execution role needs `secretsmanager:GetSecretValue` on the specific secret ARN — and `kms:Decrypt` on the key if the secret is encrypted with a customer managed key. The same applies to Parameter Store: `ssm:GetParameter` or `ssm:GetParameters`, plus `kms:Decrypt` for a `SecureString`. A missing `kms:Decrypt` is the classic confusing failure here, because the Secrets Manager permission looks correct and the error still says access denied. Also remember that if the function is attached to a VPC, reaching either service needs an egress path or a VPC endpoint. ## What a strong answer sounds like Name all three costs, with throttling as the one that turns into an incident; explain module scope versus handler scope as the mechanism; insist on a TTL and give the rotation reason for it; mention the extension as the managed option; and close on the two IAM actions plus the network path.