skip to content

Using dotnet-counters on a .NET 10 process at 90% CPU with climbing memory, which System.Runtime counters do you read first, and why?

level: middleimportance: should knowfreq 36%

answer

  1. attach by process id
  2. the default provider is System.Runtime
  3. managed heap versus working set
  4. allocations, collections and pause time
  5. queue length and exceptions

basics

~20 s

Run dotnet-counters monitor -p <pid> --showDeltas; it shows the System.Runtime meter by default. Compare CPU time with GC pause time and collections, the allocation total's growth, heap size against working set, the thread pool queue and exceptions.

solid answer

~40 s

Find the process with `dotnet-counters ps`, then run `dotnet-counters monitor -p <pid> --showDeltas`; with no `--counters` it monitors the `System.Runtime` provider, which on .NET 9 and later is a meter with `dotnet.*` instruments. For the CPU, compare `dotnet.process.cpu.time` with `dotnet.gc.pause.time` and the per-generation `dotnet.gc.collections`: a busy GC points at allocations, which `dotnet.gc.heap.total_allocated`'s delta measures. For the memory, compare `dotnet.gc.last_collection.heap.size` per generation with `dotnet.process.memory.working_set`: a gen 2 or LOH size that keeps rising across gen 2 collections suggests managed objects are being kept alive, while a flat heap under a growing working set points outside the managed heap. `dotnet.thread_pool.queue.length` and `dotnet.exceptions` catch backed-up work and exception storms. Counters say where to look; a trace or dump says why.

code

bash · 11 lines
bash
dotnet-counters ps

# live view; with no --counters it shows System.Runtime
dotnet-counters monitor -p 4127 --showDeltas

# only the counters for this incident
dotnet-counters monitor -p 4127 --showDeltas \
  --counters "System.Runtime[dotnet.process.cpu.time,dotnet.gc.pause.time,dotnet.gc.collections,dotnet.gc.heap.total_allocated,dotnet.gc.last_collection.heap.size,dotnet.process.memory.working_set,dotnet.thread_pool.queue.length,dotnet.exceptions]"

# ten minutes to a file, to compare with the next deploy
dotnet-counters collect -p 4127 --format csv -o pricing-before.csv --duration 00:00:10:00

go deeper

for a junior

Know that dotnet-counters ps finds the process and dotnet-counters monitor -p shows live runtime counters without a restart.

for a middle

Read the System.Runtime instruments: CPU time against GC pause time, the allocation total's delta, collections by generation, heap size against working set.

for a senior

Use counters to split an incident into GC-driven, allocation-driven, thread-pool-driven or native-memory problems, then pick the trace or dump that answers the next question.

for a principal

Make the same few counters the team's first look for every incident, and record them with dotnet-counters collect before and after risky deploys.

## What dotnet-counters is `dotnet-counters` is a diagnostics tool that reads **metrics** from a running .NET process without restarting it. It connects through the runtime's **diagnostic port** (a Unix domain socket on Linux and macOS, a named pipe on Windows) and asks the runtime to publish counters over **EventPipe**, the runtime's in-process event transport. It must run as the same user as the target process or as root, and on Linux and macOS both must share the same `TMPDIR`, because that is where the socket lives. Its main commands: - `dotnet-counters ps` lists .NET processes you can attach to. - `dotnet-counters monitor` shows live values in the console, refreshed every second by default (`--refresh-interval`). - `dotnet-counters collect` writes values to a file in `csv` (the default) or `json` with `--format`, for comparing before and after. With no `--counters`, both print `--counters is unspecified. Monitoring System.Runtime counters by default.` and show the `System.Runtime` provider. `--showDeltas` adds a column with the change since the previous refresh, which turns a running total into a rate you can read. ## Two generations of the same provider On **.NET 9 and later**, `System.Runtime` is a `System.Diagnostics.Metrics` meter whose instruments are named `dotnet.*`. A small app run on .NET 10.0.12 published these instruments, among others: `dotnet.process.cpu.time`, `dotnet.gc.collections`, `dotnet.gc.heap.total_allocated`, `dotnet.gc.last_collection.heap.size`, `dotnet.gc.pause.time`, `dotnet.process.memory.working_set`, `dotnet.thread_pool.queue.length`, `dotnet.exceptions`. On .NET 8 and earlier the meter does not exist and the tool falls back to the older **EventCounters** with names such as `GC Heap Size (MB)`, `Allocation Rate` and `% Time in GC since last GC`. Old screenshots and runbooks use the old names. ## Reading the CPU side 1. **`dotnet.process.cpu.time`**, split by `cpu.mode` into `user` and `system`, is CPU seconds since start; its delta per refresh is how busy the process is. 2. **`dotnet.gc.pause.time`** is total seconds paused in GC. If its delta is a large share of the CPU delta, the CPU is going into collections. 3. **`dotnet.gc.collections`**, broken down by `gc.heap.generation`, shows which generations are collected. Frequent gen 0 collections follow allocation; growing gen 2 counts are more expensive. 4. **`dotnet.gc.heap.total_allocated`** is approximate managed bytes allocated since start; its delta is the allocation rate. 5. **`dotnet.monitor.lock_contentions`** and **`dotnet.exceptions`** catch lock storms and code that throws in a loop. If none of these move much, the CPU is in your own code, and the next step is a CPU trace. ## Reading the memory side | Signal | Likely meaning | |---|---| | `dotnet.gc.last_collection.heap.size` for gen2 or loh keeps rising even after gen 2 collections | managed objects are being kept alive | | heap size flat, `dotnet.process.memory.working_set` growing | memory outside the managed heap: native allocations, stacks, mapped files | | `dotnet.gc.last_collection.heap.fragmentation.size` large | free space inside the heap the GC could not reuse | | `dotnet.gc.last_collection.memory.committed_size` far above heap size | the GC has committed more memory than the heap currently uses | The "last collection" instruments report the heap as observed during the latest garbage collection, so they track what survives collections, the number to trend when deciding whether something is leaking. ## The thread pool `dotnet.thread_pool.queue.length` counts queued work items and `dotnet.thread_pool.thread.count` the pool's threads. A queue that keeps growing while the thread count climbs slowly is the classic sign of blocked pool threads; diagnosing that belongs to the thread pool material, but the counter is where you first see it. ## Recording for comparison A live screen is for the incident; a file is for the review afterwards. - `dotnet-counters collect -p <pid> --format csv -o before.csv --duration 00:00:10:00` records ten minutes of the same counters; repeat after the fix under similar load and compare the deltas. - `--counters` accepts several providers, so your own meters can sit next to `System.Runtime` in one file. - When a process has both a meter and an EventSource with the same name, the tool prefers the meter; prefixing the provider with `EventCounters\` forces the older EventCounters view. ## What counters cannot tell you - They show **where** to look, never **which code**. A climbing gen 2 needs a heap snapshot; a busy CPU needs a trace. - Counters published as totals need `--showDeltas` or two `collect` samples to become rates. - `--counters` narrows the view, for example `System.Runtime[dotnet.gc.collections,dotnet.gc.heap.total_allocated]`, and also lists other providers such as your own meters.

  • Why does dotnet-counters show different counter names when you attach it to a .NET 8 process?
    The `System.Runtime` meter with `dotnet.*` instruments was added in .NET 9. On .NET 8 and earlier the tool falls back to the `System.Runtime` EventCounters, which use names such as `GC Heap Size (MB)` and `Allocation Rate (B / 1 sec)`. The data overlaps, but dashboards and runbooks have to match the runtime they watch.
  • How do you watch a process's counters from its very first second, before you can find its process id?
    Start it under the tool, `dotnet-counters monitor -- dotnet exec app.dll`, which launches the app and connects to it, or set `DOTNET_DiagnosticPorts` in the app's environment and run the tool with `--diagnostic-port` so the runtime connects to it at startup. The launch form redirects the app's console, so the diagnostic port form suits interactive apps.

  • GC heap size and working set always move together, so either one is enough.
  • dotnet-counters needs the app restarted with profiling enabled.
  • dotnet.gc.heap.total_allocated growing means the process is leaking.
  • A high CPU number on the counters screen already tells you which method is hot.
  • The counter names in an old runbook still apply on .NET 10.

Assumes a .NET 10 target; the System.Runtime meter and its dotnet.* instruments exist since .NET 9, and dotnet-counters falls back to the older EventCounters names on .NET 8 and earlier.