OpenTelemetry .NET: All you need to know

Hello, TedsuoTedsuo here, back again with the Windows edition of all you need to know. Today we’re going to dig into getting started with OpenTelemetry in .NET.

In fact, we have some exciting news! OpenTelemetry .NET is now v1.0v1.0! With the v1.0 release, tracing is now marked as stable. Future releases will be fully backwards compatible for many years – we have no plans for a v2.0 release.

OpenTelemetry in .NET TL;DR

All you need to know is:

• Initialization: how to start OpenTelemetryOpenTelemetry.

• AttributesAttributes and events: how to use the Activities API to decorate your current span with application data.

• SpanSpan creation: how to add new child spans to your trace.

Seriously, that’s it. Learn the following simple patterns, and you can trace your application.

To install OpenTelemetry, you need at least two packages. OpenTelemetry itself, and the OTLP exporter. Execute this command from the project folder:

dotnet add package OpenTelemetry 
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol 

You will also need to install a package for each piece of instrumentation you want to use. See the READMEREADME for currently available instrumentation.

Getting started with Dependency Injection

If you’re using OpenTelemetry, then you’re probably using a webframework which supports dependency injection via ConfigureServices. This is the most common way to set up OpenTelemetry.

using System;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Instrumentation.AspNetCore;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

namespace WebApi
{
   public class Startup
   {
       public Startup(IConfiguration configuration)
       {
           this.Configuration = configuration;
       }

       public IConfiguration Configuration { get; }

       public void ConfigureServices(IServiceCollection services)
       {
           // We recommend using env vars to set
           // the Lightstep Access Token and the service name
           string access_token = Environment.GetEnvironmentVariable("LS_ACCESS_TOKEN");
           string service_name = Environment.GetEnvironmentVariable("LS_SERVICE_NAME");

           // Set up the OpenTelemetry SDK
           services.AddOpenTelemetryTracing((builder) => builder
               .AddAspNetCoreInstrumentation()
               .AddOtlpExporter(opt =>
               {
                   opt.Endpoint = new Uri("https://ingest.lightstep.com:443");
                   opt.Headers = "lightstep-access-token=" + access_token;
               })
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(service_name)));

       }
   }
}

Let’s break this setup code down into a couple of pieces.

services.AddOpenTelemetryTracing builds the OpenTelemetry SDK. If you aren’t using dependency injection, the following invocation is equivalent:

using OpenTelemetry
public class Program
{
   public static void Main()
   {
       // The builder takes the same setup options as the DI version
       using var tracerProvider = Sdk.CreateTracerProviderBuilder()
           .Build();
   }
}

Let’s talk about the builder options we’re using here.

AddAspCoreInstrumentation does just what it says. WIth OpenTelemetry, it is important to add instrumentation for every supported library. You can find all of the available instrumentation in the getting startedgetting started section of the README.

AddOtlpExporter sets up OpenTelemetry to send data to Lightstep using the OTLP protocolOTLP protocol. I recommend sticking with OTLP in your clients. There are other exporters, such as ZipkinZipkin and JaegerJaeger. But, if you need to change formats, I recommend doing so by running a Collector and making the format change there. This allows you to make configuration changes by redeploying your Collectors, not your applications. The same goes for connecting to Lightstep. You can set your access token as the lightstep-access-token header here in the client, or connect your client to a Collector and manage your access tokens there.

SetResourceBuilder creates resources: key-value pairs which describe your service. It’s important to add as many relevant resources as you can, as this improves the quality of your tracing data. Check out the Semantic ConventionsSemantic Conventions for ideas on what to add. The two that are most important to add are service.name and service.version. If you don’t name your service, it will get named unknown_service by default.

The above snippets are enough to start, but more configuration details can be found herehere.

Instrumenting with OpenTelemetry

The setup is pretty straightforward. But What about instrumentation? You should be able to get a good baseline of coverage just by installing instrumentation for your web framework, plus any HTTP and database clients you are using. This will also ensure that traces are propagated from service to service. Still, you are going to want to add application-specific information to your traces. Here’s how to do it.

In every language, OpenTelemetry comes with an API packageAPI package, which can be used to manually instrument your code. This is available in .NET as well. However, in addition to this, OpenTelemetry is integrated into the Activity diagnostic system. An OpenTelemetry Tracer is equivalent to an ActivitySource, and an OpenTelemetry Span is equivalent to an Activity. Using Activities is the recommended approach.

It’s simple. If the instrumentation you’ve installed is already generating traces, that means you have an active span available at every point in your code. You can access this active span at any time by calling Activity.Current.

Here are the three primary actions you will want to perform with your Activity.

// adding an attribute to a span
activity?.SetTag("http.method", "GET");

Add attributes to your span by calling activity.SetTag. Attributes are key-value pairs which add indexes and operation-level information to your span. The instrumentation you installed will use the semantic conventions to add standard attributes to your spans. But you will want to add application-specific attributes as well.

Concepts like ProjectID and AccountID are good candidates. These attributes allow you to make correlations across traces and pinpoint what may be causing errors or excessive latency. For example, let’s say your application is experiencing excessive queuing time in your Kafka clients. Seeing that those long queues highly correlate with a specific Kafka partition would give you a lot of insight into the problem. Systems like Lightstep can do this type of correlation for you automatically. This feature is one of the great things that make tracing awesome!

// adding an event to a span
activity?.AddEvent(new ActivityEvent("sample activity event."));

Add logs to your spans by calling activity.AddEvent. Events are the equivalent of structured logging. They have a message name, a timestamp, and a set of attributes (key-value pairs). Being able to see your logs on your traces saves a huge amount of time – you no longer have to search and filter to find all of the logs associated with a particular transaction.

//marking a span as an error
activity?.SetTag("otel.status_code", "ERROR");
activity?.SetTag("otel.status_description", "error status description");

Mark an operation as an error by setting the status. The error status will cause the span to create alerts, count against an error budgeterror budget, or otherwise trigger the error handling behavior in your observability system. The status concept is currently being added to Activities. In the meantime, you can set it as a tag.

Creating child spans

Last but not least, you may want to create more spans yourself. You should only do this when you have large and important operations to carve out for tracking latency and setting alerts. There is no need to create spans for every class or function. For web application logic, I recommend decorating the span created by your framework instrumentation instead of creating child spans.

If you do have a library you’d like to instrument with its own spans, there are two steps. First, create an ActivitySource to generate spans from. More details on the options available can be found herehere.

// an activity source is equivalent to a tracer
static ActivitySource activitySource = new ActivitySource(
   "companyname.product.instrumentationlibrary",
   "semver1.0.0");

Second, use the ActivitySource to create an Activity around your operation.

// start the activity before your operation
var activity = activitySource.StartActivity("ActivityName");
// stop the activity when your operation completes
activity?.Stop();

// Alternativley, wrap your operation in a using block
using (var activity = activitySource.StartActivity("ActivityName")
{

} // Activity gets stopped automatically at the end of this block during dispose.

I recommend keeping span management out of your application code if you can since it can be a little verbose.

Congratulations, you’re tracing!

And that’s it. OpenTelemetry may seem complex at first, but I hope this walkthrough showed how simple it actually is. Initialize your SDK, add attributes and events to your current span, and create child spans. That’s all you need to know!

Interested in joining our team? See our open positions herehere.