A stored procedure that creates a temporary table runs thousands of times a minute, and the database starts showing contention and bloat unrelated to the business tables. What is happening, and how would you fix it?
answer
- cost is the DDL, not the rows
- catalog/allocation contention, not business tables
- new object per call = recompile per call
- create once per session + DELETE ROWS / truncate
- temp space is shared: one job hurts everyone
basics
~20 sEach creation writes system-catalog rows and allocates storage, so at high rates the catalog itself becomes the hot spot: latch or lock contention on metadata, catalog bloat, and constant plan recompilation. Fixes: create the table once per session and reuse it, drop the staging step entirely, or move the intermediate inline.
solid answer
~60 sThe cost is not in the rows — it is in the **object**. Every `CREATE TEMP TABLE` inserts rows into system catalogs and allocates pages; the matching drop deletes them. At thousands of calls a minute this produces: - **Metadata contention**: sessions serialize on catalog pages or allocation structures — the classic `tempdb` metadata contention in SQL Server, catalog bloat plus autovacuum pressure on `pg_class`/`pg_attribute` in PostgreSQL. - **Recompilation storms**: statements referencing a temp table are bound to that session's object, so a new object per call means re-planning per call, burning CPU. - **Shared temp space pressure**: every session's sorts and hash spills share that space, so one runaway job degrades unrelated queries. Fixes, in order: eliminate the staging step (inline the intermediate, or use a set-based statement); create the table **once per session** and use `ON COMMIT DELETE ROWS` or truncate per call; use engine features that reduce metadata cost (Oracle global temporary tables, SQL Server's memory-optimized tempdb metadata); and cap concurrency so temp space cannot be exhausted.
go deeper
Recognise that creating a table is a heavier operation than inserting rows, and that doing it constantly has a cost.
Name catalog/metadata work as the mechanism and propose creating the table once per session with per-call clearing instead of per-call creation.
Diagnose from metadata waits, compilation rate, and temp-space growth, then apply a fix ladder starting with eliminating the staging step and batching calls.
Treat temporary space as a shared, capacity-planned resource with concurrency caps and alerting, and set architectural guidance on when server-side staging belongs in a hot path at all.
## Why object creation is the expensive part A temporary table is cheap to *write into* — reduced logging, unshared buffers — but creating and dropping it is not cheap, because it is real DDL. The engine inserts rows describing the table, its columns, its indexes, and its statistics into system catalogs, allocates initial storage, and reverses all of that at drop time. That work touches structures shared by every session in the instance. At ten creations a minute nobody notices. At ten thousand, the catalog is the hottest object in the database and the profile looks bizarre: heavy contention with no business table involved. ## The three symptoms **Metadata contention.** Concurrent sessions creating temp objects collide on the same allocation and catalog pages. SQL Server's version — allocation-page (PFS/GAM/SGAM) and system-table contention in `tempdb` — was so common it drove multiple product changes: multiple tempdb data files, then memory-optimized tempdb metadata in SQL Server 2019. In PostgreSQL the same workload shows as rapid growth and bloat of `pg_class`, `pg_attribute`, and `pg_type`, with autovacuum unable to keep up and catalog lookups slowing for everyone. **Recompilation.** A cached plan referencing a temp table is tied to that session's specific object with its own statistics. Create a fresh object each call and the plan cannot be reused, so every call pays optimization cost. In a routine with several statements over the temp table, compilation can exceed execution. **Shared space exhaustion.** Temporary space is a global resource that also backs sorts, hash joins, and spills for every session. One job staging a huge intermediate can push unrelated queries into failure or force them to spill differently. Because the failure surfaces in innocent queries, diagnosis is often slow. ## A fourth, sneakier problem: pooled sessions With connection pooling, session-scoped temp tables persist across logical requests on the same physical connection. Two failure modes: stale rows read by an unrelated request, and unbounded accumulation of temp objects on long-lived connections that never disconnect, so the automatic cleanup never fires. ## The fixes, in the order to try them **1. Do you need the temp table at all?** In OLTP paths the intermediate is often small enough to be a derived table, a values list, or an array parameter. Removing the staging step removes every problem above. This is the fix that most often applies and is most often skipped. **2. Create once, reuse per call.** Keep the definition for the session and clear it per invocation — `ON COMMIT DELETE ROWS`, or an explicit truncate/delete at the start of the routine, with creation guarded so it happens only once. Catalog cost drops from per-call to per-session, and plans referencing the stable object become reusable. Oracle's global temporary tables take this further: the definition is permanent and created at deploy time, so no session ever pays DDL cost, while contents remain per-session. **3. Use engine-specific relief.** Memory-optimized tempdb metadata and multiple tempdb files in SQL Server; keeping temp objects small enough to stay in memory in MySQL; on PostgreSQL, aggressive autovacuum settings for the catalogs plus attention to the fact that a long-lived session's temp objects hold catalog entries. **4. Bound the blast radius.** Cap concurrency for the jobs that stage large intermediates, monitor temporary-space usage with alerting well below the limit, and set per-session limits where the engine supports them so one query cannot consume all of it. **5. Batch instead of repeating.** If a routine is called ten thousand times because the application loops row by row, the real fix is set-based: one call staging ten thousand rows beats ten thousand calls staging one each — which also collapses ten thousand round trips into one. ## Diagnosing it Look for wait or lock events on system catalog objects and allocation structures rather than user tables; a high rate of compilations relative to executions; growth in catalog table sizes not explained by schema changes; and temporary-space usage that tracks call rate rather than data volume. The tell that separates this from ordinary load is that the contention sits on metadata while business tables look idle. ## What a strong answer includes Name the mechanism (DDL against shared catalogs, not row writes), the three consequences (metadata contention, recompilation, shared space), and a fix ladder that starts with eliminating the staging step rather than tuning around it. Mentioning that the same procedure is harmless at low call rates shows you understand it is a rate problem, not a design taboo.
- How would you distinguish this from ordinary load on the business tables?The waits and contention sit on system catalog and allocation structures rather than on user tables, and the affected objects are the same regardless of which business tables the procedure touches. Compilation counts scale with call rate rather than with data volume, and temporary-space usage also tracks call rate. If business tables look idle while metadata is hot, object creation is the cause.
- Why does creating a fresh temporary table per call cause plan recompilation?A cached plan for a statement over a temporary table is bound to that session's specific object, including its identity and its statistics, so a newly created object cannot reuse the previous plan. Every call therefore pays parse and optimization cost, which in a short routine can exceed the execution cost. Creating the table once per session and clearing rows instead keeps the object stable enough for plans to be reused.
- When is a temporary table inside a frequently called procedure still the right design?When the intermediate is genuinely large or reused several times within the call, so that the statistics and indexability it provides prevent a much worse plan, and when the call rate is low enough that catalog cost is negligible. It is a rate-dependent judgment, not a prohibition. If the call rate is high, the same benefit should be obtained by batching many logical units into one call rather than by creating an object per unit.
Renting a locker for every parcel: carrying the parcel is trivial, but the queue at the rental desk is where everyone jams up.
saying these in an interview costs you the question
- Assuming temp tables are free because they are unlogged — the DDL is the cost.
- Blaming the business tables when the contention is on catalog metadata.
- Adding more temp tables or indexes on them to "speed it up" without addressing creation rate.
- Ignoring that temporary space is shared across all sessions.
- Concluding temp tables should never be used in procedures, rather than that creation rate is the problem.
- Leaving session-scoped temp objects to accumulate on long-lived pooled connections.