skip to content

In an ASP.NET Core 10 checkout that calls seat-inventory and payment services, how do you add a custom span with ActivitySource, and why might it never be exported?

level: seniorimportance: should knowfreq 40%

answer

  1. one long-lived source per component
  2. StartActivity returns Activity?
  3. AddSource must match the name
  4. using ends the span
  5. SetStatus and AddException on failure

basics

~20 s

Keep one long-lived ActivitySource, wrap the step in using var activity = source.StartActivity("HoldSeat"), tag via activity?.SetTag, and list the name in AddSource. Spans go missing when the name is unsubscribed, the trace unsampled, or the activity never stopped.

solid answer

~40 s

I keep one `ActivitySource` per component for the app's lifetime, usually `static readonly ActivitySource Source = new("Airline.Booking")`. Around the seat hold I write `using var activity = Source.StartActivity("HoldSeat");`, which becomes a child of `Activity.Current`, normally the request's activity, so it nests inside the checkout trace. `StartActivity` returns `Activity?`: it is `null` when no tracer provider listens to that source name or the sampler drops the trace, so every call is `activity?.SetTag(...)`. On failure I call `SetStatus(ActivityStatusCode.Error, ...)` and `AddException(ex)`. The span ends when the `using` disposes it, and only ended spans are exported. When a span never shows up, the usual causes are a missing or misspelt `AddSource("Airline.Booking")` in `WithTracing`, a parent that was not sampled, or an activity that was never stopped.

code

csharp · 24 lines
csharp
using System.Diagnostics;

public sealed class SeatHoldClient(HttpClient inventory)
{
    public static readonly ActivitySource Source = new("Airline.Booking");

    public async Task HoldAsync(string flight, string seat, CancellationToken ct)
    {
        using var activity = Source.StartActivity("HoldSeat"); // null if unsubscribed or unsampled
        activity?.SetTag("airline.flight", flight);
        activity?.SetTag("airline.seat", seat);
        try
        {
            var response = await inventory.PostAsJsonAsync($"/holds/{flight}", new { seat }, ct);
            response.EnsureSuccessStatusCode();
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.AddException(ex);
            throw;
        }
    }
}

go deeper

for a junior

Remember the shape: a static ActivitySource, using var activity = Source.StartActivity(...), and AddSource with the same name in WithTracing.

for a middle

Explain why StartActivity returns null, how Activity.Current makes the span a child of the request span, and why using is what ends and exports it.

for a senior

Diagnose a missing span: name mismatch with AddSource, unsampled parent, never-stopped activity, startup timing or a filtered parent. Add failure status and exception events correctly.

for a principal

Set conventions for source names, span names and tag keys across services so traces stay searchable, and decide which business steps deserve spans at all.

## The scenario The airline's **booking-api** handles `POST /checkout`. Inside that request it holds a seat in **seat-inventory** and charges the card in **payments**, both over `HttpClient`. The ASP.NET Core instrumentation already gives one server span per request in each service, and HttpClient instrumentation gives the client spans between them. What is missing is a span for the business step itself, for example "hold the seat", so that time spent in your own code, retries and failures are visible in the trace. ## Step 1: one long-lived `ActivitySource` An `ActivitySource` is the factory for spans and its **name** is how the SDK finds it. - Create it **once** per component and keep it for the app's lifetime: a `static readonly` field, or a singleton registered in DI. - Pick a stable, dotted name, for example `"Airline.Booking"`; the constructor also accepts a version: `new ActivitySource("Airline.Booking", "1.0.0")`. - Creating a new source per request works, but it wastes allocations and listener bookkeeping. ## Step 2: start the span with `using` `ActivitySource.StartActivity(string name = "", ActivityKind kind = ActivityKind.Internal)` returns **`Activity?`**: 1. If no `ActivityListener` subscribes to the source's name, it returns `null` immediately. This is what makes unused instrumentation cheap. 2. If a listener exists, the SDK's **sampler** decides. When it drops a span whose parent is local, such as the request span, `StartActivity` also returns `null`; a dropped root or remote-parent span comes back as a propagation-only activity whose `IsAllDataRequested` is `false`. 3. Otherwise you get a started `Activity` whose parent is **`Activity.Current`**, in a request the hosting layer's `Microsoft.AspNetCore.Hosting.HttpRequestIn` activity, and it becomes `Activity.Current` itself until it stops. Hence the two habits: always use the null-conditional operator (`activity?.SetTag(...)`), and always end the span with `using` (or `Dispose`/`Stop` in a `finally`). An activity that is never stopped is never handed to the processors, so it is never exported, and `Activity.Current` keeps pointing at it. ## Step 3: tags, status and errors | Call | Purpose | |---|---| | `activity?.SetTag("airline.flight", flight)` | adds or replaces an attribute | | `activity?.AddEvent(new ActivityEvent("seat-held"))` | a timestamped event inside the span | | `activity?.SetStatus(ActivityStatusCode.Error, ex.Message)` | marks the span as failed | | `activity?.AddException(ex)` | records the exception as a span event | | `activity?.IsAllDataRequested` | `false` when the activity only propagates context; skip expensive tags then | Keep personal data such as passenger names out of tags; traces are usually retained and searched widely. ## Step 4: subscribe the SDK to the source In `Program.cs`, the tracer provider must name the source: ```csharp builder.Services.AddOpenTelemetry() .WithTracing(t => t .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddSource("Airline.Booking")); ``` `AddSource` accepts exact names and wildcards such as `"Airline.*"`. `AddAspNetCoreInstrumentation()` does **not** cover custom sources. ## Choosing the `ActivityKind` `StartActivity` defaults to `ActivityKind.Internal`, which is right for an in-process step such as validating a fare or holding a seat in local state. The other kinds describe a boundary: - `Client` for an outgoing synchronous call your code makes without an instrumented library, for example a hand-rolled TCP protocol; - `Server` for an incoming request handled outside ASP.NET Core's hosting layer; - `Producer` and `Consumer` for messages put on or taken off a queue. Backends draw service maps from these kinds, so a wrong kind produces a misleading map: a `Client` span with no matching `Server` span on the other side looks like a call to an unknown dependency. When a library such as `HttpClient` already creates the boundary span, keep yours `Internal` and let it wrap the library's span. ## Why the span never appears: a checklist 1. **Name mismatch.** `AddSource("Airline.Bookings")` versus `new ActivitySource("Airline.Booking")`: `StartActivity` returns `null` every time and nothing fails. 2. **Not stopped.** An activity created without `using`, or stopped only on the success path, is never exported. 3. **Not sampled.** If the sampler drops the trace (for example because the caller marked it unsampled), `StartActivity` returns `null` for children too. 4. **Started before the SDK exists.** Code that runs before the host builds the tracer provider, such as early startup work, has no listener yet. 5. **Filtered or not exported.** A `Filter` on the ASP.NET Core instrumentation that rejects the request clears the request activity's recorded flag; with the SDK's default parent-based sampler, every child span in that request, yours included, is then dropped as well. And with no exporter, nothing leaves at all. ## Across the three services The custom span lives in one process. The trace continues into seat-inventory and payments because the outgoing `HttpClient` request carries the trace context in its headers and the receiving service's hosting layer continues it. The custom span therefore needs no cross-service code; it simply nests under the request span and above the `HttpClient` client span it wraps.

  • Why does the code use activity?.SetTag instead of activity.SetTag?
    `StartActivity` returns `Activity?`. It returns `null` when no listener subscribes to the source name or the sampler drops the trace, which is the normal state in tests or with tracing off. Calling `SetTag` on that `null` would throw `NullReferenceException`, so every use is null-conditional.
  • The HoldSeat span shows up but appears as its own trace instead of under the checkout request. What happened?
    It was started when `Activity.Current` was not the request activity, for example from work queued to a `BackgroundService` or started after the request ended. `StartActivity` parents to `Activity.Current`; with none it starts a new trace. Pass an explicit parent `ActivityContext` or start the span inside the request's flow.

  • StartActivity always returns an Activity, so the null-conditional operator is unnecessary.
  • AddAspNetCoreInstrumentation also exports spans from your own ActivitySource instances.
  • A span is exported as soon as StartActivity returns, so disposing it is optional.
  • A custom span must be given the request's parent explicitly to nest under it.
  • Creating a new ActivitySource per request is required to keep spans isolated.