OpenTelemetry Java: All you need to know

Hi all, tedsuotedsuo here, back with our third installment of All you need to know. Today we’re going to go over Java, and how to instrument it.

Getting started with OpenTelemetry Java: TL;DR

All you need to know is:

• Initialization: How to attach the OpenTelemetry Java Agent.

• Tracer methods: getTracer, currentSpan, startSpan, and setCurrent.

• Span methods: setAttribute, addEvent, recordException, setStatus, and end. Seriously, that’s it. If you want to try it out, follow the guide below. A heavily commented version of the finished tutorial can be found at https://github.com/tedsuo/otel-java-basicshttps://github.com/tedsuo/otel-java-basics. Consider walking through this tutorial with the code open.

Hello, world

For this tutorial, we’re going to make a very, very simple application: a web servlet that responds to http://localhost:9000/hellohttp://localhost:9000/hello with “Hello World.” Let’s make the application. The hello world server has two pieces. The Jetty handler looks like this:

package com.lightstep.examples.server;

import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;

public class ApiContextHandler extends ServletContextHandler
{
 public ApiContextHandler()
 {
   addServlet(new ServletHolder(new ApiServlet()), "/hello");
 }

 static final class ApiServlet extends HttpServlet
 {
   @Override
   public void doGet(HttpServletRequest req, HttpServletResponse res)
     throws ServletException, IOException
   {
     // pretend to do work
     try {
       Thread.sleep(500);
     } catch (InterruptedException e) {
     }

     // respond
     try (PrintWriter writer = res.getWriter()) {
       writer.write("Hello World");
     }
   }
 }
}

Set up and run your jetty server:

package com.lightstep.examples.server;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.handler.ContextHandlerCollection;

public class App
{
 public static void main( String[] args )
     throws Exception
   {
     ContextHandlerCollection handlers = new ContextHandlerCollection();
     handlers.setHandlers(new Handler[] {
       new ApiContextHandler(),
     });
     Server server = new Server(9000);
     server.setHandler(handlers);

     server.start();
     server.dumpStdErr();
     server.join();
   }
}

To talk to this server, create a simple client that makes 5 calls to /hello.

package com.lightstep.examples.client;

import java.io.IOException;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;

public class App
{
 public static void main( String[] args )
     throws Exception
   {
     for (int i = 0; i < 5; i++) {
       makeRequest();
     }
   }

 static void makeRequest()
 {
    OkHttpClient client = new OkHttpClient();
    Request req = new Request.Builder()
      .url("http://localhost:9000/hello")
      .build();

    try (Response res = client.newCall(req).execute()) {
      System.out.println("make request”);
    } catch (Exception e) {
      System.out.println(String.format("Request failed: %s", e));
    }
  }
}

Boot up the server and check that it works, then try running the client against it.

OpenTelemetry Architecture in 30 seconds

Ok, I said no details, but here is one that is actually helpful. OpenTelemetry clientsOpenTelemetry clients have two major components: the SDK and the API. The SDK is the actual framework, the API is what you use to instrument your code. This separation provides loose coupling: your application code only depends on the API, which has virtually no dependencies and acts like a no-op when the SDK is not installed. This allows packages to add instrumentation without automatically pulling in the implementation’s dependency chain (think grpc, etc). This separation of concerns is especially helpful for OSS libraries that want to bake in instrumentation, but don’t want to create overhead or dependency conflicts when OpenTelemetry is not being used.

Tip: Never reference any SDK package outside of installation and setup. All other packages and application code should only depend on the API.

Pick an OpenTelemetry backend

Ok, let’s add OpenTelemetry to this application. To test our tracing, you’ll need a place to send the data.

At Lightstep, we created free-for-life community accountsfree-for-life community accounts specifically for making OpenTelemetry easy to experiment with. If you don’t already have one, please grab an account.

If you’d like to use Zipkin or Jaeger instead, this getting started guidegetting started guide will walk you through the setup. Once you’re set-up, you can come back here and follow the rest of the tutorial.

Install the Java Agent using the OpenTelemetry Launcher

Since we’re connecting to Lightstep, we’ll also be using the Lightstep Distro of OpenTelemetry, the OpenTelemetry LaunchersOpenTelemetry Launchers. Distros package up any plugins and configuration needed to talk to a particular backend. At the moment, we’re still fleshing out the full definition of a Distro (what is allowed, and what isn’t), but the basic point is to make getting started easier by reducing configuration boilerplate. If you want more detail, you can check out this blog postthis blog post where I initially proposed the concept.

For now, just download the launcher jar file from the latest release of the launcherlauncher.

Run your application with OpenTelemetry

export LS_ACCESS_TOKEN=my-access-token-etc

java -javaagent:lightstep-opentelemetry-javaagent-0.11.0.jar  \
 -Dls.service.name=hello-server \
 -Dotel.propagators=tracecontext,b3 \
 -Dotel.resource.attributes="something=else,container.name=my-container" \
 -Dotel.bsp.schedule.delay.millis=200 \
       -cp server/target/server-1.0-SNAPSHOT.jar \
       com.lightstep.examples.server.App

 java -javaagent:lightstep-opentelemetry-javaagent-0.11.0.jar  \
 -Dls.service.name=hello-client \
 -Dotel.propagators=tracecontext,b3 \
 -Dotel.bsp.schedule.delay.millis=200 \
       -cp client/target/client-1.0-SNAPSHOT.jar \
       com.lightstep.examples.client.App

Check out what automatic instrumentation gives you

Switch over to Lightstep, or your backend of choice, and confirm the spans were received:

Yup, we see spans. Click through and look at a trace:

Notice that we see a client span from hello-client, a server span from hello-server, and several internal spans representing HTTP client and server components. Also, notice that the client and server spans are already populated with HTTP, network, and other attributesattributes. All of this common information is standardized across instrumentation as semantic conventionssemantic conventions. An HTTP request will always be described with the same keys and values, regardless of what language or package it comes from.

This is a lot of really useful information. We already have a complete trace, with a lot of detail, and we haven’t written any instrumentation yet. When rolling out OpenTelemetry, this is the approach I recommend. Get OpenTelemetry installed into every service and ensure that context is propagating correctly, before adding any further detail. This will be enough information to set up error monitoring and identify latency issues.

The OpenTelemetry Java API

Ok, so the out-of-the-box experience will get you a long way, but of course, you will eventually want to add additional application data.

Spans should ideally be managed by your application framework. In this case, the servlet framework manages the span for you. In your application code, you can continue to decorate these spans with more information. There are two primary types of data you will want to add: attributes and events.

Span attributes are indexes for segmenting your data. For example, you may want to add project.id or account.id in order to understand if slow requests and errors are specific to a certain set of accounts, or affecting everyone.

Fine-grain logging can be added as span events. Events are a form of structured logging - use them like you would logs. The advantage with span events is that you can automatically find all of the logs associated with a particular transaction, rather than having to go hunting with a bunch of searches and filters. As you scale up, this becomes a lifesaver (or, at least, a big time saver).

Import the OpenTelemetry API itself, then get a new tracer in your ApiContextHandler:

import io.opentelemetry.api.trace.Tracer;

public class ApiContextHandler extends ServletContextHandler
{
 // name your tracer after the class it instruments
 // spans started with this tracer will then
 // be attributed to this package
 private static final Tracer tracer =
     OpenTelemetry.getGlobalTracer("com.lightstep.examples.server.ApiContextHandler");

The name of the tracer appears on every span as the instrumentation.name attribute. This is useful for investigating instrumentation issues. We can then get our current span from the context by requesting it by calling Span.current(), as seen in the below server code - attributes and events can be added to whatever the “current” span is after you have a handle on the span in context.

Run your server and client again, and you will see these new attributes and events show up on the same spans.

package com.lightstep.examples.server;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;

public class ApiContextHandler extends ServletContextHandler
{
 // name your tracer after the class it instruments
 // spans started with this tracer will then
 // be attributed to this package
 private static final Tracer tracer =
     OpenTelemetry.getGlobalTracer("com.lightstep.examples.server.ApiContextHandler");

 public ApiContextHandler()
 {
   addServlet(new ServletHolder(new ApiServlet()), "/hello");
 }

 static final class ApiServlet extends HttpServlet
 {
   @Override
   public void doGet(HttpServletRequest req, HttpServletResponse res)
     throws ServletException, IOException
   {
     // access the current span that has automatically been created by
     // the servlet instrumentation
     Span span = Span.current();

     // define the route name using semantic conventions
     span.setAttribute("http.route", "hello");

     // pretend to do work
     try {
       Thread.sleep(500);
     } catch (InterruptedException e) {
     }

     try (PrintWriter writer = res.getWriter()) {
       // events are structured logs, contextualized by the trace.
       childSpan.addEvent("writing response",
         Attributes.of(AttributeKey.stringKey("content"), "hello world"));

       writer.write("Hello World");
     }
 }
}

Creating your own spans

You can also create your own spans. These spans will automatically become children of the current span and added to the trace.

Span management involves three steps: starting the span, setting it as the current span, and ending the span.

When you create a new span in JavaJava, OpenTelemetry will create it as a child of the current span, if one exists. Call the spanBuilder method on your tracer to start a new one. Name the span after the operation you are measuring. Advice on naming can be found in the tracing specificationspecification.

*IMPORTANT: make sure to end the span when your operation finishes, or you will have a leak! *

After creating a new span, use a Scope to create a new block of code where the child span is the current one. Any calls to Span.current() inside this scope will return your child span, rather than the parent for the request. Other methods continue to work as normal. Once you’re finished, be sure to close your span by calling the end() method on it.

After span.end() is called, Spans are queued up to be exported in the next flush. Calls to setAttribute and addEvent become no-ops after span.end() is called.

public class ApiContextHandler extends ServletContextHandler
{
 // name your tracer after the class it instruments
 // spans started with this tracer will then
 // be attributed to this package
 private static final Tracer tracer =
     OpenTelemetry.getGlobalTracer("com.lightstep.examples.server.ApiContextHandler");

     // start a child span
     Span childSpan = tracer.spanBuilder("my-server-span").startSpan();
     try (Scope scope = childSpan.makeCurrent()) {

       // inside the new scope, getCurrentSpan returns childSpan.
       // note that span methods can be chained.
       Span.current().setAttribute("ProjectId", "456");

     } finally {
       // make sure to close the span
       childSpan.end();
     }
   }

Should you create new spans in this way? It depends on the organization and size of your service, primarily. If you have a significant amount of work that needs to be measured independently of the overall work being performed, then adding child spans to your code can be useful. However, it can often be easier and more beneficial to simply add more events or attributes to a single span rather than creating many smaller ones per-service.

Error Handling

There is one final type of event that deserves special attention: exceptions. In OpenTelemetry, exceptions are recorded as events. But, to ensure that the exception is properly formatted, the span.recordException(error) method should be used instead of addEvent.

childSpan.recordException(new RuntimeException("oops"));
childSpan.setStatus(StatusCode.ERROR);

By default, exceptions do not count as errors. In OpenTelemetry, an error means that the overall operation did not complete. Plenty of exceptions are expected, and a handled exception does not automatically mean the entire operation failed to complete. In other cases, an operation could fail without an exception being thrown. In order to declare an operation a failure, call span.setStatus() and pass in an error code. Status codes are used by analysis tools to automatically trigger alerting, measure error rates, etc.

Note: status codes will be simplified in the next version of OpenTelemetry.

That’s all, folks!

And that is that. All you need to know to get started with tracing in Java. Hopefully, that was pretty straight forward and clears up any mysteries about how to use OpenTelemetry.

If you stick with the above patterns, you can get a great deal of visibility with very little work. Of course, there are many more details and options; you can check out the documentationdocumentation for more information. I also have a more involved getting started guide for opentelemtry javaopentelemtry java; it works as a handy reference for all of the procedures described above.

OpenTelemetry is still in beta due to API changes, but it is also already in production across many organizations. If you stick to a DistroDistro and automated instrumentation, you can use OpenTelemetry today without much fear of a breaking change, as those changes will most likely involve the API. If you are writing manual instrumentation during the beta, consider creating helper functions that simplify the API for your use cases, and give you a centralized place to manage any potential breakage.

Also: consider joining our community! There are plenty of libraries left to instrument. You can find us on GitHubGitHub, or say hi on gittergitter.

Interested in joining our team? See our open positions herehere.