In this blog post
OpenTelemetry SetupOpenTelemetry SetupCreate a free accountCreate a free accountInstall OpenTelemetryInstall OpenTelemetryRun Express with OpenTelemetryRun Express with OpenTelemetryCheck for successCheck for successObserving ExpressObserving ExpressService resourcesService resourcesSystem metricsSystem metricsTracing Express applicationsTracing Express applicationsObserving routesObserving routesObserving middlewareObserving middlewareObserving application codeObserving application codeObserving databasesObserving databasesHello, folks! This week we’re looking at JavascriptJavascript and Express. Node is a popular JavaScript runtime for back-end services and Express is easily the most well-known web framework for Node.
Express is elegant enough that this walkthrough makes a great archetypal example of OpenTelemetryOpenTelemetry. The patterns explained here apply to web frameworks in general.
TL;DR
Create a free account.
Install OpenTelemetry with the Lightstep Launcher.
Monitor routes to identify errors and latency.
Pinpoint issues using data from middleware, templating, and database instrumentation.
OpenTelemetry Setup
For a detailed setup and usage guide, please check out my OpenTelemetry Node walkthroughOpenTelemetry Node walkthrough. The well-commented example codeexample code that goes with it is makes a great handy reference for all of the basic OpenTelemetry functions.
But, if you have your express app handy – here’s the short, short version:
Create a free account
First, pick a place to send your telemetry. At Lightstep, we built free community accountsfree community accounts specifically for learning about and developing with OpenTelemetry. So if you don’t already have one, please grab an accountgrab an account.
These setup instructions presume you are using Lightstep. But if you’d like to use a different analysis tool, such as Jaeger, follow those setup instructions and skip to the Observations section below.
Install OpenTelemetry
Lightstep offers a basic distrodistro called the OpenTelemetry LauncherOpenTelemetry Launcher which installs and configures all available OpenTelemetrymodules, including Express instrumentation. Install everything with this.
npm i lightstep-opentelemetry-launcher-node
Run Express with OpenTelemetry
OpenTelemetry must be started before any other packages are required.
For an easy way to load and run opentelemetry, copy this script from otel-node-basicsotel-node-basics into your application, and use it as your new start script.
const { lightstep } = require('lightstep-opentelemetry-launcher-node');
// This file replaces your application’s start script.
// Replace ORIGINAL_START_SCRIPT with the path
// to your start script.
const startScript = './path/to/ORIGINAL_START_SCRIPT'
// Use the launcher to set up the OpenTelemetry SDK.
// Recommended parameters:
// * accessToken: found in Lightstep settings.
// * serviceName: used to identify resources.
// * serviceVersion: used to identify regressions.
// * propagators: tracecontext setting recommended.
const sdk = lightstep.configureOpenTelemetry({
accessToken: '<ACCESS TOKEN>',
serviceName: 'hello-server',
serviceVersion: 'v1.2.3',
propagators: 'tracecontext,b3,baggage',
});
// For auto-instrumentation to work,
// the SDK must be started before any
// other packages are loaded.
sdk.start().then(() => {
// require your application once opentelemetry
// has loaded.
require(startScript);
});
// Shutdown flushes any remaining spans before exit.
function shutdown() {
sdk.shutdown().then(
() => console.log("shutdown complete"),
(err) => console.log("error shutting down", err),
).finally(() => process.exit(0))
};
process.on('beforeExit', shutdown);
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);
Check for success
Run your application, click around a bunch to generate data (or cURL your API), then shut it down to flush any remaining telemetry. Refresh the Explorer in Lightstep, and confirm that you are seeing data. If nothing shows up, turn on the debug logs to check for errors.
export OTEL_LOG_LEVEL=debug
Observing Express
Ok, here’s the fun part. Let’s review all of the telemetry data an Express application generates out of the box.
OpenTelemetry currently defines three signal types: tracestraces, metricsmetrics, and resourcesresources. Resources and metrics represent services, traces represent transactions across services.
Every object described by a signal is defined by key-value pairs, called attributesattributes. OpenTelemetry also defines the standard attributes for describing common concepts, such as HTTP requests. These definitions are called Semantic Conventions. All instrumentation, regardless of language, reports data using these conventions, so it is worthwhile to become familiar with these conventions. We’ll be making heavy use of them below.
Service resources
Services describe themselves to OpenTelemetry by using resource conventions.
service.name
- service name is a primary index in OpenTelemerty. Almost all queries and alerts end up being scoped by service name. It should always be present.service.version
- A version number or commit hash. When a new version is deployed, latency and error rates may shift. Without a version number, it is much harder to understand how code changes affect errors and performance. For example, Lightstep is able to analyze performance changes and automatically detect regressions, but we need a version number to be present.service.instance.id
- useful for looking up all concurrent operations on a particular service. Without this, you associate traces with a type of service, not individual services. Some resources can be auto-detected, such ashostname
, but others need to be added by hand. Before deploying to production, I recommend reviewing all of the resource conventions and confirming that every relevant convention has been added.
System metrics
An important part of observing Express is monitoring resource usage. OpenTelemetry provides standard system metricssystem metrics you would expect, via the host metricshost metrics plugin. There are two obvious highlights to start with, which should always be monitored:
system.cpu.utilization
- percentage of cpu being utilized.system.memory.utilization
- percentage of memory being utilized.Monitoring these (or equivalent) statistics should be a requirement for every service.
Tracing Express applications
Tracing is where the express-specific details begin to emerge.
Observing routes
There is one primary span in every Express request, called the route span.The naming pattern for the route span is {METHOD} {ROUTE}
e.g. “GET /hello”. This span measures the entire request lifecycle. When alerting on latency, this is the best span to use. The route span includes HTTP and networking information for the request.
http.*
andnet.*
– these conventions describe everything about the request.instrumentation.name
– this describes the instrumentation package which generated the span.
Observing middleware
Every piece of middleware is represented by a child span of the original route span. Middleware spans are named “middleware - {MIDDLEWARE NAME}”. So, for example, compression would appear as “middleware - compression”.
Observing application code
Unless overridden, application code will always have access to the route span created by express.
const opentelemetry = require('@opentelemetry/api');
const express = require('express');
// create a tracer and name it after your package
const tracer = opentelemetry.trace.getTracer('@otel-node-basics/server');
const app = express();
app.get('/hello', (req, res) => {
// access the span created by express instrumentation
span = tracer.getCurrentSpan();
// add an attribute to segment your data by projectID
span.setAttribute('projectID', '123');
// log an event and include some structured data.
span.addEvent('setting timeout', { sleep: 300 });
setTimeout(()=> {
span.addEvent(responding after timeout);
res.status(200).send('Hello World');
}, 300);
});
Rather than create child spans, I recommend you continue to decorate the route span with custom application attributes and events.
Observing databases
Last but not least, observing database interactions is critical. OpenTelemetry provides instrumentation for a variety of databases and message queues. Instrumentation for these packages will automatically be installed if they are present. Database spans will appear as child spans of the route span.
Thanks for diving deep!
Interested in joining our team? See our open positions herehere.
In this blog post
OpenTelemetry SetupOpenTelemetry SetupCreate a free accountCreate a free accountInstall OpenTelemetryInstall OpenTelemetryRun Express with OpenTelemetryRun Express with OpenTelemetryCheck for successCheck for successObserving ExpressObserving ExpressService resourcesService resourcesSystem metricsSystem metricsTracing Express applicationsTracing Express applicationsObserving routesObserving routesObserving middlewareObserving middlewareObserving application codeObserving application codeObserving databasesObserving databasesExplore more articles

OpenTelemetry Collector in Kubernetes: Get started with autoscaling
Moh Osman | Jan 6, 2023Learn how to leverage a Horizontal Pod Autoscaler alongside the OpenTelemetry Collector in Kubernetes. This will enable a cluster to handle varying telemetry workloads as the collector pool aligns to demand.
Learn moreLearn more
Observability-Landscape-as-Code in Practice
Adriana Villela, Ana Margarita Medina | Oct 25, 2022Learn how to put Observability-Landscape-as-Code in this hands-on tutorial. In it, you'll use Terraform to create a Kubernetes cluster, configure and deploy the OTel Demo App to send Traces and Metrics to Lightstep, and create dashboards in Lightstep.
Learn moreLearn more
OpenTelemetry for Python: The Hard Way
Adriana Villela | Sep 20, 2022Learn how to instrument your Python application with OpenTelemetry through manual configuration and manual context propagation, using Lightstep as the Observability back-end.
Learn moreLearn moreLightstep sounds like a lovely idea
Monitoring and observability for the world’s most reliable systems