In an ASP.NET Core 10 checkout that calls seat-inventory and payment services, how do you add a custom span with ActivitySource, and why might it never be exported?
answer
- one long-lived source per component
- StartActivity returns Activity?
- AddSource must match the name
- using ends the span
- SetStatus and AddException on failure
basics
~20 sKeep one long-lived ActivitySource, wrap the step in using var activity = source.StartActivity("HoldSeat"), tag via activity?.SetTag, and list the name in AddSource. Spans go missing when the name is unsubscribed, the trace unsampled, or the activity never stopped.
solid answer
~40 sI keep one `ActivitySource` per component for the app's lifetime, usually `static readonly ActivitySource Source = new("Airline.Booking")`. Around the seat hold I write `using var activity = Source.StartActivity("HoldSeat");`, which becomes a child of `Activity.Current`, normally the request's activity, so it nests inside the checkout trace. `StartActivity` returns `Activity?`: it is `null` when no tracer provider listens to that source name or the sampler drops the trace, so every call is `activity?.SetTag(...)`. On failure I call `SetStatus(ActivityStatusCode.Error, ...)` and `AddException(ex)`. The span ends when the `using` disposes it, and only ended spans are exported. When a span never shows up, the usual causes are a missing or misspelt `AddSource("Airline.Booking")` in `WithTracing`, a parent that was not sampled, or an activity that was never stopped.
code
csharp · 24 linesusing System.Diagnostics;
public sealed class SeatHoldClient(HttpClient inventory)
{
public static readonly ActivitySource Source = new("Airline.Booking");
public async Task HoldAsync(string flight, string seat, CancellationToken ct)
{
using var activity = Source.StartActivity("HoldSeat"); // null if unsubscribed or unsampled
activity?.SetTag("airline.flight", flight);
activity?.SetTag("airline.seat", seat);
try
{
var response = await inventory.PostAsJsonAsync($"/holds/{flight}", new { seat }, ct);
response.EnsureSuccessStatusCode();
}
catch (Exception ex)
{
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
activity?.AddException(ex);
throw;
}
}
}go deeper
Remember the shape: a static ActivitySource, using var activity = Source.StartActivity(...), and AddSource with the same name in WithTracing.
Explain why StartActivity returns null, how Activity.Current makes the span a child of the request span, and why using is what ends and exports it.
Diagnose a missing span: name mismatch with AddSource, unsampled parent, never-stopped activity, startup timing or a filtered parent. Add failure status and exception events correctly.
Set conventions for source names, span names and tag keys across services so traces stay searchable, and decide which business steps deserve spans at all.
## The scenario The airline's **booking-api** handles `POST /checkout`. Inside that request it holds a seat in **seat-inventory** and charges the card in **payments**, both over `HttpClient`. The ASP.NET Core instrumentation already gives one server span per request in each service, and HttpClient instrumentation gives the client spans between them. What is missing is a span for the business step itself, for example "hold the seat", so that time spent in your own code, retries and failures are visible in the trace. ## Step 1: one long-lived `ActivitySource` An `ActivitySource` is the factory for spans and its **name** is how the SDK finds it. - Create it **once** per component and keep it for the app's lifetime: a `static readonly` field, or a singleton registered in DI. - Pick a stable, dotted name, for example `"Airline.Booking"`; the constructor also accepts a version: `new ActivitySource("Airline.Booking", "1.0.0")`. - Creating a new source per request works, but it wastes allocations and listener bookkeeping. ## Step 2: start the span with `using` `ActivitySource.StartActivity(string name = "", ActivityKind kind = ActivityKind.Internal)` returns **`Activity?`**: 1. If no `ActivityListener` subscribes to the source's name, it returns `null` immediately. This is what makes unused instrumentation cheap. 2. If a listener exists, the SDK's **sampler** decides. When it drops a span whose parent is local, such as the request span, `StartActivity` also returns `null`; a dropped root or remote-parent span comes back as a propagation-only activity whose `IsAllDataRequested` is `false`. 3. Otherwise you get a started `Activity` whose parent is **`Activity.Current`**, in a request the hosting layer's `Microsoft.AspNetCore.Hosting.HttpRequestIn` activity, and it becomes `Activity.Current` itself until it stops. Hence the two habits: always use the null-conditional operator (`activity?.SetTag(...)`), and always end the span with `using` (or `Dispose`/`Stop` in a `finally`). An activity that is never stopped is never handed to the processors, so it is never exported, and `Activity.Current` keeps pointing at it. ## Step 3: tags, status and errors | Call | Purpose | |---|---| | `activity?.SetTag("airline.flight", flight)` | adds or replaces an attribute | | `activity?.AddEvent(new ActivityEvent("seat-held"))` | a timestamped event inside the span | | `activity?.SetStatus(ActivityStatusCode.Error, ex.Message)` | marks the span as failed | | `activity?.AddException(ex)` | records the exception as a span event | | `activity?.IsAllDataRequested` | `false` when the activity only propagates context; skip expensive tags then | Keep personal data such as passenger names out of tags; traces are usually retained and searched widely. ## Step 4: subscribe the SDK to the source In `Program.cs`, the tracer provider must name the source: ```csharp builder.Services.AddOpenTelemetry() .WithTracing(t => t .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddSource("Airline.Booking")); ``` `AddSource` accepts exact names and wildcards such as `"Airline.*"`. `AddAspNetCoreInstrumentation()` does **not** cover custom sources. ## Choosing the `ActivityKind` `StartActivity` defaults to `ActivityKind.Internal`, which is right for an in-process step such as validating a fare or holding a seat in local state. The other kinds describe a boundary: - `Client` for an outgoing synchronous call your code makes without an instrumented library, for example a hand-rolled TCP protocol; - `Server` for an incoming request handled outside ASP.NET Core's hosting layer; - `Producer` and `Consumer` for messages put on or taken off a queue. Backends draw service maps from these kinds, so a wrong kind produces a misleading map: a `Client` span with no matching `Server` span on the other side looks like a call to an unknown dependency. When a library such as `HttpClient` already creates the boundary span, keep yours `Internal` and let it wrap the library's span. ## Why the span never appears: a checklist 1. **Name mismatch.** `AddSource("Airline.Bookings")` versus `new ActivitySource("Airline.Booking")`: `StartActivity` returns `null` every time and nothing fails. 2. **Not stopped.** An activity created without `using`, or stopped only on the success path, is never exported. 3. **Not sampled.** If the sampler drops the trace (for example because the caller marked it unsampled), `StartActivity` returns `null` for children too. 4. **Started before the SDK exists.** Code that runs before the host builds the tracer provider, such as early startup work, has no listener yet. 5. **Filtered or not exported.** A `Filter` on the ASP.NET Core instrumentation that rejects the request clears the request activity's recorded flag; with the SDK's default parent-based sampler, every child span in that request, yours included, is then dropped as well. And with no exporter, nothing leaves at all. ## Across the three services The custom span lives in one process. The trace continues into seat-inventory and payments because the outgoing `HttpClient` request carries the trace context in its headers and the receiving service's hosting layer continues it. The custom span therefore needs no cross-service code; it simply nests under the request span and above the `HttpClient` client span it wraps.
- Why does the code use activity?.SetTag instead of activity.SetTag?`StartActivity` returns `Activity?`. It returns `null` when no listener subscribes to the source name or the sampler drops the trace, which is the normal state in tests or with tracing off. Calling `SetTag` on that `null` would throw `NullReferenceException`, so every use is null-conditional.
- The HoldSeat span shows up but appears as its own trace instead of under the checkout request. What happened?It was started when `Activity.Current` was not the request activity, for example from work queued to a `BackgroundService` or started after the request ended. `StartActivity` parents to `Activity.Current`; with none it starts a new trace. Pass an explicit parent `ActivityContext` or start the span inside the request's flow.
- StartActivity always returns an Activity, so the null-conditional operator is unnecessary.
- AddAspNetCoreInstrumentation also exports spans from your own ActivitySource instances.
- A span is exported as soon as StartActivity returns, so disposing it is optional.
- A custom span must be given the request's parent explicitly to nest under it.
- Creating a new ActivitySource per request is required to keep spans isolated.