NodeJS Express: An OpenTelemetry deep dive

Hello, folks! This week we’re looking at Javascript and Express. Node is a popular JavaScript runtime for back-end services and Express is easily the most well-known web framework for Node.

Express is elegant enough that this walkthrough makes a great archetypal example of OpenTelemetry. The patterns explained here apply to web frameworks in general.

TL;DR

• Create a free account.
• Install OpenTelemetry with the Lightstep Launcher.
• Monitor routes to identify errors and latency.
• Pinpoint issues using data from middleware, templating, and database instrumentation.

OpenTelemetry Setup

For a detailed setup and usage guide, please check out my OpenTelemetry Node walkthrough. The well-commented example code that goes with it is makes a great handy reference for all of the basic OpenTelemetry functions.

But, if you have your express app handy – here’s the short, short version:

Create a free account

First, pick a place to send your telemetry. At Lightstep, we built free community accounts specifically for learning about and developing with OpenTelemetry. So if you don’t already have one, please grab an account.

These setup instructions presume you are using Lightstep. But if you’d like to use a different analysis tool, such as Jaeger, follow those setup instructions and skip to the Observations section below.

Install OpenTelemetry

Lightstep offers a basic distro called the OpenTelemetry Launcher which installs and configures all available OpenTelemetrymodules, including Express instrumentation. Install everything with this.

npm i lightstep-opentelemetry-launcher-node

Run Express with OpenTelemetry

OpenTelemetry must be started before any other packages are required.

For an easy way to load and run opentelemetry, copy this script from otel-node-basics into your application, and use it as your new start script.

const { lightstep } = require('lightstep-opentelemetry-launcher-node');

// This file replaces your application’s start script. 
// Replace ORIGINAL_START_SCRIPT with the path
// to your start script.
const startScript = './path/to/ORIGINAL_START_SCRIPT'

// Use the launcher to set up the OpenTelemetry SDK.
// Recommended parameters: 
// * accessToken: found in Lightstep settings.
// * serviceName: used to identify resources.
// * serviceVersion: used to identify regressions.
// * propagators: tracecontext setting recommended.
const sdk = lightstep.configureOpenTelemetry({
  accessToken: '<ACCESS TOKEN>',
  serviceName: 'hello-server',
  serviceVersion: 'v1.2.3',
  propagators: 'tracecontext,b3,baggage',
});

// For auto-instrumentation to work,
// the SDK must be started before any 
// other packages are loaded.
sdk.start().then(() => {
  // require your application once opentelemetry
  // has loaded.
  require(startScript);
});

// Shutdown flushes any remaining spans before exit.
function shutdown() {
  sdk.shutdown().then(
    () => console.log("shutdown complete"),
    (err) => console.log("error shutting down", err),
  ).finally(() => process.exit(0))
};
process.on('beforeExit', shutdown);
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);

Check for success

Run your application, click around a bunch to generate data (or cURL your API), then shut it down to flush any remaining telemetry. Refresh the Explorer in Lightstep, and confirm that you are seeing data. If nothing shows up, turn on the debug logs to check for errors.

export OTEL_LOG_LEVEL=debug

Observing Express

Ok, here’s the fun part. Let’s review all of the telemetry data an Express application generates out of the box.

OpenTelemetry currently defines three signal types: traces, metrics, and resources. Resources and metrics represent services, traces represent transactions across services.

Every object described by a signal is defined by key-value pairs, called attributes. OpenTelemetry also defines the standard attributes for describing common concepts, such as HTTP requests. These definitions are called Semantic Conventions. All instrumentation, regardless of language, reports data using these conventions, so it is worthwhile to become familiar with these conventions. We’ll be making heavy use of them below.

Service resources

Services describe themselves to OpenTelemetry by using resource conventions.

• service.name - service name is a primary index in OpenTelemerty. Almost all queries and alerts end up being scoped by service name. It should always be present.
• service.version - A version number or commit hash. When a new version is deployed, latency and error rates may shift. Without a version number, it is much harder to understand how code changes affect errors and performance. For example, Lightstep is able to analyze performance changes and automatically detect regressions, but we need a version number to be present.
• service.instance.id - useful for looking up all concurrent operations on a particular service. Without this, you associate traces with a type of service, not individual services. Some resources can be auto-detected, such as hostname, but others need to be added by hand. Before deploying to production, I recommend reviewing all of the resource conventions and confirming that every relevant convention has been added.

System metrics

An important part of observing Express is monitoring resource usage. OpenTelemetry provides standard system metrics you would expect, via the host metrics plugin. There are two obvious highlights to start with, which should always be monitored:

• system.cpu.utilization - percentage of cpu being utilized.
• system.memory.utilization - percentage of memory being utilized.
• Monitoring these (or equivalent) statistics should be a requirement for every service.

Tracing Express applications

Tracing is where the express-specific details begin to emerge.

Observing routes

There is one primary span in every Express request, called the route span.
The naming pattern for the route span is {METHOD} {ROUTE} e.g. “GET /hello”. This span measures the entire request lifecycle. When alerting on latency, this is the best span to use. The route span includes HTTP and networking information for the request.

• http.* and net.* – these conventions describe everything about the request.
• instrumentation.name – this describes the instrumentation package which generated the span.

Observing middleware

Every piece of middleware is represented by a child span of the original route span. Middleware spans are named “middleware - {MIDDLEWARE NAME}”. So, for example, compression would appear as “middleware - compression”.

Observing application code

Unless overridden, application code will always have access to the route span created by express.

const opentelemetry = require('@opentelemetry/api');
const express = require('express');

// create a tracer and name it after your package
const tracer = opentelemetry.trace.getTracer('@otel-node-basics/server');

const app = express();

app.get('/hello', (req, res) => {
// access the span created by express instrumentation
span = tracer.getCurrentSpan();
 // add an attribute to segment your data by projectID
span.setAttribute('projectID', '123');
// log an event and include some structured data.
span.addEvent('setting timeout', { sleep: 300 });

setTimeout(()=> {
  span.addEvent(responding after timeout);
  res.status(200).send('Hello World');
}, 300);
});

Rather than create child spans, I recommend you continue to decorate the route span with custom application attributes and events.

Observing databases

Last but not least, observing database interactions is critical. OpenTelemetry provides instrumentation for a variety of databases and message queues. Instrumentation for these packages will automatically be installed if they are present. Database spans will appear as child spans of the route span.

Thanks for diving deep!