Instrument your Lambda Functions with OpenTelemetry (Preview)

Lightstep, one of the major contributors to OpenTelemetry, will be donating an AWS Lambda Extension to the OpenTelemetry communityAWS Lambda Extension to the OpenTelemetry community for anyone looking to use open-source vendor agnostic tools to instrument and better understand systems built with AWS Lambda Functions.

OpenTelemetry, an observability framework for cloud-native software, is one of the most active and popular projects in the Cloud Native Computing Foundation (CNCF).It began as a merger of the OpenCensus and OpenTracing projects to provide a unified open source solution to make observability a built-in feature of cloud-native software, with contributors from major players in the space such as Google, Amazon, and others.

This new extension solves a major challenge in generating observability data such as traces and metrics from serverless functions -- namely, collecting and exfiltrating that data to analysis tools while ensuring that function performance is not adversely impacted. The recent preview of AWS Lambda ExtensionsAWS Lambda Extensions provides a way to get high quality telemetry data in parallel with function execution and minimal additional overhead.

With OpenTelemetry Lambda ExtensionsOpenTelemetry Lambda Extensions developers can now more efficiently capture telemetry data with a flexible open source, vendor agnostic tool. The extension runs a binary that emits metrics and traces to any configurable backend alongside with the Lambda function. It is configured using a standard OpenTelemetry collector configuration YAML file. Telemetry data can then be collected using standard OpenTelemetry API calls and the OpenTelemetry protocol.

To find out more about the OpenTelemetry project, visit our OpenTelemetry DocumentationOpenTelemetry Documentation.

OpenTelemetry Lambda Extension

A few weeks ago, AWS announced Lambda extensionsLambda extensions, which allows users to run binaries in parallel with lambda functions. This gave us an idea for a fun experiment, what if we could run an OpenTelemetry Collector as a Lambda Extension?

The problem

Our users have been very keen on leveraging OpenTelemetry via Lambdas. This brings up a couple of different challenges. The first challenge is that collecting telemetry data every time the lambda is called adds latency to each invocation. If the destination of the telemetry data is on a different network or experiencing delays, it could severely impact the cost of invoking those lambdas. If the delay is too great and causes the functions to hit a timeout, the telemetry data will never reach its destination. A way to reduce this latency could be to deploy an OpenTelemetry collector within the same Virtual Private Cloud collector. Of course, this then is yet another component in the infrastructure that needs to be managed, monitored, and updated.

The extension

AWS Lambda Extensions were specifically built with the idea of allowing users to integrate tools necessary to improve monitoring and observability of their functions. The tools embedded within an extension can register to be notified on different events in the lifecycle of a lambda function. This is important, because in order to ensure telemetry is sent from the collector, we need an opportunity to flush any remaining data before the lambda runtime is shut down. Thankfully, there's an event for that.

The binaries are run within the same lambda runtime as the functions themselves and can be written in a different language than the runtime environment. Once the code has been compiled, the lambda extension is distributed as a lambda layerlambda layer, which can be published and made available through any of the many AWS tools.

For the experiment, in order to have a collector that can register for shutdown events, an InProcessCollectorInProcessCollector was combined with sample code available in the aws-sampleaws-sample repository. The last piece of the puzzle was figuring out how to configure the collector. The OpenTelemetry collector supports a flexible YAML configuration fileconfiguration file and we wanted to ensure users could re-use existing configuration with their lambdas. The approach we took was to allow users to embed their configuration within their lambda packages and make the OpenTelemetry extension capable of reading that file. All users need to do is to set the OPENTELEMETRY_COLLECTOR_CONFIG_FILE environment variable to the location of the file. To simplify this in the future, the configuration could live in S3.

Using the Lambda extension

Now that we had the extension layer, all we needed was to put it to use. We put together a Lambda using NodeJS. Inside the Lambda, we used the standard OpenTelemetry API to instrument our function, and configured the code to use the OpenTelemetry Protocol (OTLP) exporter with the default configuration:

const exporter = new CollectorTraceExporter({
  serviceName: 'basic-service',
})

We then added the following collector configuration file to our package. This configured the OTLP receiver for JSON over HTTP and an exporter to send data to Lightstep.

NOTE: the collector can expand environment variables when reading its configuration file, so we used an environment variables to configure our token

# collector.yaml
receivers:
  otlp:
    protocols:
      http:
exporters:
  otlp:
    endpoint: ingest.lightstep.com:443
    headers: {"lightstep-access-token":"$TOKEN"}
processors:
  batch:

service:
  extensions:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Once the Lambda was created, we added the extension layer and the environment variable configuration file:

aws lambda update-function-configuration --function-name Function --layers arn:aws:lambda:us-east-1:297975325230:layer:opentelemetry-extension:2
aws lambda update-function-configuration --function-name Function --environment Variables={OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/collector.yaml}

And that was it! We were able to start invoking our function and see traces appear in Lightstep!

Next steps

Now that we have a working prototype, we plan on doing additional performance tests to validate whether this approach is reasonable. One side effect that we observed that will likely need to be improved on is that the maximum amount of memory consumed by a lambda was increased by roughly 80MB with the new Extension layer. This could likely be tweaked by doing some work on the collector. We will be looking to donate the code for the extension layercode for the extension layer to the OpenTelemetry organization to ensure everyone can enjoy its benefits.

Interested in joining our team? See our open positions herehere.