OpenTelemetry Go: All you need to know

Welcome back to the latest edition of All you need to know. Today we’ll be focusing on one of my favorite languages, Go.

All you really need to know is this:

• Initialization: How to set up instrumentation and shut down cleanly.

• Tracer methods: GetTracer, StartSpan.

• Span methods: SpanFromContext SetAttribute, AddEvent, RecordEerror, SetStatus, and End.

Seriously, that’s it. If you want to try out OpenTelelmetry in Go, follow the guide below.

A heavily commented version of the finished tutorial can be found at herehere. Feel free to use this code as a reference when you get started with instrumenting your own application. If you want to see your data easily, sign up for a free Lightstep accountfree Lightstep account to try this tutorial.

Installing OpenTelemetry

There are two types of components you will need when installing OpenTelemetryOpenTelemetry. The first contains the core components: the OpenTelemetry API and SDK. Lightstep provides an OpenTelemetry distro that configures these components for you (more on that later).

Install the launcher:

go get github.com/lightstep/otel-launcher-go/launcher

The second type of component is OpenTelemetry instrumentation. These packages need to match up with the libraries your application is using. You can find officially supported instrumentation in the go-contrib repositorygo-contrib repository. Longer-term, the OpenTelemerty RegistryOpenTelemerty Registry will provide a reference to all available libraries.

For this tutorial, we’ll be using the core HTTP package from the Go standard library. Install the HTTP instrumentation:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Setting up and tearing down with OpenTelemetry in Go

In this tutorial, we’re going to create two programs: a client and a server. Let’s start with the server. Create a server package and add the OpenTelemetry Launcher to your main function.

NOTE: The Launcher wraps up OpenTelemetry with some convenient defaults set for connecting to Lightstep. If you would like to connect to another backend, or want to see what’s going on under the hood, check out the Go installation guideGo installation guide found in the OpenTelemetry documentation.

import (
"github.com/lightstep/otel-launcher-go/launcher"
"go.opentelemetry.io/otel/semconv"
)

func main() {
   // Create an OpenTelemetry SDK using the launcher
   sdk := launcher.ConfigureOpentelemetry(
       launcher.WithServiceName("hello-server-4"),
       launcher.WithServiceVersion("1.3"),
       launcher.WithAccessToken("YOUR_ACCESS_TOKEN"),
       launcher.WithPropagators([]string{"tracecontext", "b3", "baggage"}),
       launcher.WithResourceAttributes(map[string]string{
           string(semconv.ContainerNameKey): "my-container-name",
       }),
   )
   // Shut down the SDK to flush any remaining data before the program exits
   defer sdk.Shutdown()
}

Ideally, you should start OpenTelemetry at the beginning of main, before any other services start running. When your program exits, call shutdown on the SDK to ensure the last bit of telemetry is flushed before the program exits.

Let’s have a look at the options I’ve included. There are more here than is strictly required, but I want to go over them in detail because these are my suggestions for a minimal production setup. Note that all of these options can also be set via Environment Variables. The complete list of options can be found herehere.

ServiceName: the name of the class of service this program is an instance of. For example, authentication-service or web-proxy. When you look in Lightstep, this is how service instances will be grouped together. If you don’t provide a name, unkown_service will be used.

ServiceVersion: the version number of your service. This can be in any format, though semver format is expected. Reporting the version number allows you to track regressions across deployments, and pinpoint errors to specific versions.

AccessToken: this is required for sending data to Lightstep. You can find your Lightstep Access Token under project settings in the Lightstep UI. This token is added as a header to your OTLP connection.

Note: If you are running Collectors, you can set the access token there instead of in the application itself.

Propagators: the trickiest setting. This defines which headers the OpenTelemetry tracing system will use to connect your services together. If you aren’t running a tracing system yet, I recommend using tracecontext as that is now the W3C standard headerW3C standard header. If you are adding OpenTelemetry to an existing system that is already using “b3” headers, I recommend setting both tracecontext and b3 so that you can switch over to tracecontext seamlessly. Turn the also include the “baggage” headers in case you want to use the baggage system at a later date (don’t worry about baggage right now though). If you want more details about what this is all for, you can read up on context propagation herecontext propagation here.

Resources: this set of key-value pairs defines attributesattributes which describe your service. Service name and service version are examples of resource attributes, but you can add many more. The list of official attributes can be found herehere, and the go constants for them can be found in the semversemver package. I recommend setting as many standard attributes as you can, plus any custom attributes which you believe may make a difference when root-causing a problem.

Instrumenting an HTTP server

Ok, let’s have a look at how you add tracing to your application. For an HTTP server, you do this by wrapping each handler with OpenTelemetry instrumentation.

So, if you have a basic handler function like this:

// Example HTTP Handler
func helloHandler(w http.ResponseWriter, req *http.Request) {
   time.Sleep(time.Second)
   w.Write([]byte("Hello World"))
}

You can then wrap this handler in instrumentation using the otelhttp package.

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

func startServer() {
   // create a handler wrapped in OpenTelemetry instrumentation
   handler := http.HandlerFunc(helloHandler)
   wrappedHandler := otelhttp.NewHandler(handler, "/hello")

   // serve up the wrapped handler
   http.Handle("/hello", wrappedHandler)
   http.ListenAndServe(":9000", nil)
}

That’s it! Your handler will now be traced. While this is one simple way to add tracing to your service, you can check out the Go docsGo docs for more info on the other options available.

Instrumenting an HTTP client

Let’s look at the other end of the connection. First, Create a client package and add the launcher to its main function. You can use the same launcher code you used for your server, just change the name to hello-client instead of hello-server.

To set up client-side tracing, wrap the transport of your HTTP client with OpenTelemetry instrumentation. All requests made with this client will now be traced.

func makeRequest(ctx context.Context) {
   // Trace an HTTP client by wrapping the transport
   client := http.Client{
       Transport: otelhttp.NewTransport(http.DefaultTransport),
   }

   // Make sure you pass the context to the request to avoid broken traces.
   req, err := http.NewRequestWithContext(ctx, "GET", "http://localhost:9000/hello", nil)
   if err != nil {
       panic(err)
   }

   // All requests made with this client will create spans.
   res, err := client.Do(req)
   if err != nil {
       panic(err)
   }
   res.Body.Close()
}

Besides setting up the transport, there is one other important piece: always, always, always pass the current context to the request. OpenTelemetry relies heavily on the Context object being both correct and available.

Looking at the data

Run the server and use this client to connect to it. You will see a very simple trace, with two spans: one for the client and one for the server.

Notice all of the HTTP information which has already been added to the spans as attributes. Click over to the Details tab to see all of the resource attributes you added which you started the launcher.

Adding detail

The steps above are all you actually need to get started with tracing. But once you’re set-up, you’ll want to add more application-specific detail to your traces.

Adding attributes, events, and errors to the current span

Let’s go back to the HTTP server example. Imagine that you would like to add projectID as an attribute to the server span. This would be useful information to know when looking at a trace, and would allow Lightstep to detect if problems correlated with a particular project. You may also want to log events and trigger error warnings.

import (
   "go.opentelemetry.io/otel/label"
   "go.opentelemetry.io/otel/trace"
)

func helloHandler(w http.ResponseWriter, req *http.Request) {
   // get the span from the current context
   cxt := req.Context()
   span := trace.SpanFromContext(cxt)
   // add attributes to the span
   span.SetAttributes(label.Int("ProjectID", 42))

   // Adding events is the tracing equivalent to logging.
   // These events will show up as logs on the span.
   span.AddEvent("writing response", trace.WithAttributes(
       label.String("content", "hello world"),
       label.Int("answer", 42),
   ))

   // Errors can be recorded as events
   span.RecordError(errors.New("ooops"))

   // To mark the entire operation as an error, set the status.
   // Note that recording an error does not automatically change
   // the status.
   span.SetStatus(codes.Error, "failed due to internal error")

   // …rest of function
}

Adding this information to a span has two parts. First, you need access to the span that was created by the HTTP instrumentation you set up. To access the current span, call the SpanFromContext method. When context is flowing correctly, and your server is creating spans for you, a span will always be available in the current context object.

Once you have the span, you can add attributes, events, and other options.

Setting attributes adds additional indexes to your span. This allows you to group and search for data more effectively.

Adding events is the tracing equivalent of structured logging. Each event has a message, a timestamp, and a set of attributes. This is how you record fine-grained detail about what your application is doing. I think you will discover that finding events attached to your spans is easier than finding logs that have no tracing context. It saves a lot of time when you are moving quickly and root causing a problem.

Recording errors is a special form of adding an event. It ensures that the error is formatted in a standard fashion.

Finally, Setting the status is probably the most important mechanism to know about. Tracing systems are designed to alert and trigger on failed operations. By setting the span status to Error, you are telling your tracing system that this operation has failed, and it should trigger any configured alerts and count against your error budgeterror budget. HTTP instrumentation will automatically set the span status to error for 4xx and 5xx HTTP status codes.

NOTE: You can use the Collector to modify and override the span status on any span, as well as add and remove attributes.

You might think that recording an error automatically sets the span status to error, but it doesn’t! This is because not all go errors represent a failed operation. For example, a “file not found” error may change the behavior of an operation, but not cause the entire operation to abort. Likewise, an operation may fail without an explicit error occurring that needs to be recorded.

Creating child spans

In general, you should not need to create additional spans when tracing your application. But, as you become comfortable with tracing, you may want to carve out operations into their own spans so that you can set up error and latency monitoring for that particular portion of your code. Here’s how you do it.

First, you need a handle to a tracer in order to create a span.

import "go.opentelemetry.io/otel/trace"

// Create one tracer per package
// NOTE: You only need a tracer if you are creating your own spans
var tracer trace.Tracer

func init() {
   // Name the tracer after the package, or the service if you are in main
   tracer = otel.Tracer("github.com/my/package")
}

In Go, it is recommended that you create one tracer per package. Tracers should be named after the package name, or after the service name if you are in the main package. These tracer names are handy; they tell you which package the spans originated from.

Once you have a tracer, you can create your own spans.

   // You can create child spans from the span in the current context.
   // The returned context contains the new child span
   cxt, childSpan = tracer.Start(cxt, "my-child-span")

   // The new context now contains the child span, so it can be accessed
   // in other functions simply by passing the context.
   span := trace.SpanFromContext(cxt)

   // Always end the span when the operation completes,
   // otherwise you will have a leak.
   defer childSpan.End()

Spans are started by a tracer and are automatically attached to the span in the current context as a child span. This adds the new span to the current trace. The context returned from Start now contains the child span. Pass this context to any functions that are part of the operation the child span is recording.

Once the operation is complete, call End on the span to record the operation latency. Properly ending spans is important: this is the one place where you may potentially create a leak, as spans which never end will never be sent, and will continue to consume memory. Luckily, defer makes this easy in most cases.

Thanks all, folks!

That’s it! The above guide contains everything you need to trace a production application using OpenTelemetry.