A .NET 10 service sits at 90% CPU in production; how do you capture and read a CPU profile with dotnet-trace, and what are its limits?
answer
- EventPipe through the diagnostic port
- the default profiles on collect
- sampled thread time, not on-CPU time
- nettrace converts to Speedscope
- a kernel-aware view on Linux
basics
~20 sRun dotnet-trace collect -p <pid> --duration for a short window; it enables the dotnet-common and dotnet-sampled-thread-time profiles over EventPipe. Open the nettrace in PerfView or Visual Studio on Windows, or convert it to Speedscope, and read the hottest stacks.
solid answer
~40 sConfirm the CPU with counters, then `dotnet-trace ps` and `dotnet-trace collect -p <pid> --duration 00:00:00:30` while the process is hot. With no `--profile` or `--providers`, `collect` enables `dotnet-common` (runtime events) and `dotnet-sampled-thread-time`, which samples managed thread stacks about 100 times a second over EventPipe, with no restart. The output is a `.nettrace` file: open it in PerfView or Visual Studio on Windows, convert it with `--format Speedscope` or `dotnet-trace convert`, or print `dotnet-trace report <file> topN`. Because thread-time sampling records threads whether they are running or waiting, focus on threads that are actually busy. Its limits are safe-point bias and managed-only stacks; on Linux with .NET 10, `dotnet-trace collect-linux` (a preview verb that needs root) adds kernel CPU sampling and native frames.
code
bash · 11 linesdotnet-trace ps
# 30 seconds of the default profiles, plus a Speedscope copy
dotnet-trace collect -p 4127 --duration 00:00:00:30 \
--format Speedscope -o pricing-cpu.nettrace
# text summary of where samples landed
dotnet-trace report pricing-cpu.nettrace topN -n 15
# GC-only view when counters show heavy GC pauses
dotnet-trace collect -p 4127 --profile gc-collect --duration 00:00:01:00go deeper
Know that dotnet-trace collect records a trace file from a running process, viewed in PerfView or Visual Studio on Windows or in Speedscope after conversion.
Explain EventPipe and the diagnostic port, the default profiles, and the difference between exclusive and inclusive time in a call tree.
Run it safely in production: a short window, the right profile, waiting threads filtered out, and a kernel-aware tool when native code or the kernel is the suspect.
Decide where diagnostics stay enabled, who may attach tools to production processes, and how containers expose the diagnostic port.
## How dotnet-trace reaches the process By default every .NET process opens a **diagnostic port**: a Unix domain socket named `dotnet-diagnostic-{pid}-{key}-socket` in the temp directory on Linux and macOS, or the named pipe `dotnet-diagnostic-{pid}` on Windows. `dotnet-trace` connects to it and starts an **EventPipe** session, the runtime's cross-platform event transport, naming the providers and keywords to enable. Events stream back into a `.nettrace` file. Nothing is restarted, which is why this works on a production process, subject to three conditions: - The tool runs as the same user as the process, or as root. - On Linux and macOS the tool and the app share `TMPDIR`; in containers that means running the tool in the same container or sharing the socket's directory. - The app has not set `DOTNET_EnableDiagnostics=0`, which disables the diagnostic ports and, with them, every dotnet-* tool. ## Capturing a CPU trace 1. Confirm the symptom with `dotnet-counters monitor`: CPU time is high and GC pause time is not the main cause. 2. Find the process with `dotnet-trace ps`. 3. Run `dotnet-trace collect -p <pid> --duration 00:00:00:30` while the CPU is high. Keep the window short: traces grow quickly and you only need enough samples. 4. Without `--profile`, `--providers` or `--clrevents`, the tool prints that it is defaulting to `dotnet-common` plus `dotnet-sampled-thread-time`. 5. Add `--format Speedscope` to get a Speedscope file next to the `.nettrace`, or convert later. ## Collect options worth knowing - `-o` names the output file; without it the tool writes `<appname>_<yyyyMMdd>_<HHmmss>.nettrace`. - `--providers` and `--clrevents` add specific event sources or runtime keywords on top of, or instead of, profiles. - `--stopping-event-provider-name`, `--stopping-event-event-name` and `--stopping-event-payload-filter` end the trace when a specific event arrives, which catches a short spike without a long recording. - `-- <command>` launches the app under the tool, for problems that happen at startup. ## Profiles | Profile | Verb | What it enables | |---|---|---| | `dotnet-common` | collect, collect-linux | runtime events at informational level (GC, JIT, loader and others) | | `dotnet-sampled-thread-time` | collect | managed thread stacks sampled at about 100 Hz | | `gc-verbose` | collect, collect-linux | GC collections plus sampled object allocations | | `gc-collect` | collect, collect-linux | GC collections only, at very low overhead | | `cpu-sampling` | collect-linux | kernel CPU sampling through perf_events | Older releases of the tool offered `cpu-sampling` on `collect`. It was removed from that verb because the name was misleading: it sampled every thread regardless of CPU use. Passing it to `collect` today fails with an error that the profile does not apply to `dotnet-trace collect`. `--providers` takes explicit `Provider:Keywords:Level` entries when a profile does not fit. ## Reading the trace - **PerfView** and **Visual Studio** open `.nettrace` directly on Windows; PerfView's call trees show inclusive and exclusive time per method. - **Speedscope** shows flame graphs in a browser. `dotnet-trace convert trace.nettrace --format Speedscope` keeps the original file, because conversion loses information. - **`dotnet-trace report trace.nettrace topN`** prints the methods on the stack longest, by exclusive time unless you add `--inclusive`. Read exclusive time to find the method doing the work and inclusive time to find the caller responsible for it. ## Limits - **Thread time, not CPU time.** `dotnet-sampled-thread-time` samples threads whether they are running or blocked, so a thread waiting on a lock or a socket still collects samples. Filter to threads that are actually busy before blaming a method. - **Safe-point bias.** Managed samples are taken where the runtime can stop a thread, so attribution is approximate. - **Managed frames only.** Native code, the kernel and the GC's own work are invisible in the stacks. - **Kernel-aware alternatives.** On Linux, .NET 10 adds `dotnet-trace collect-linux`, a preview verb that combines EventPipe with perf_events for managed and native stacks; it needs root, kernel 6.4 or later with user_events, and glibc 2.27 or later. On Windows, ETW-based tools such as PerfView give a kernel-aware view. - **Overhead.** Sampling and runtime events cost CPU in the target; keep sessions short on a struggling process. ## When the CPU is the GC If counters show large GC pause time, a CPU profile will be full of GC work you cannot see in managed stacks. Use the `gc-collect` or `gc-verbose` profile to see collections and allocation sources, and leave GC configuration to the garbage-collection material.
- How do you trace a .NET process from startup, before it has a process id you can attach to?Launch it under the tool, `dotnet-trace collect -- dotnet exec app.dll`, which starts the app with a diagnostic port the tool listens on. Alternatively set `DOTNET_DiagnosticPorts` in the app's environment and pass `--diagnostic-port` to the tool. The runtime then connects to the tool and, unless `nosuspend` is set, waits for it before running managed code.
- Why might a trace collected from a container's host time out?On Linux the diagnostic port is a socket in the target's temp directory, and the tool finds it by process id. From outside the container, the process id and temp directory differ, so the tool cannot find the socket. Run the tool inside the container, share the socket's directory with a sidecar, or configure `DOTNET_DiagnosticPorts` and `--diagnostic-port` explicitly.
- dotnet-trace needs the service restarted with a profiler attached.
- Every sampled stack in a thread-time trace was burning CPU.
- dotnet-trace collect --profile cpu-sampling is the standard way to profile CPU today.
- Converting the nettrace to Speedscope keeps every event, so the original can be deleted.
- A managed sampling trace shows native and kernel frames as well.
Assumes .NET 10 and a current dotnet-trace; collect's default profiles changed when the old cpu-sampling profile was removed, and collect-linux needs a .NET 10 runtime.