Introducing the Lightstep Observability Toolkit for GitHub developers
by Fran Thorpe
Lightstep has made it easy for organizations to identify changes in production, answer the question "what changed", and use that information to quickly resolve issues. Lightstep is now bringing that same level of detail in front of developers inside GitHub. For developers without access to the powerful insights provided by Lightstep, it’s often a manual process to connect deployments to associated performance issues. Developers often have no way to easily connect the code or config changes they commit to source control to services running in production and pre-production environments.
Tracking what changed and how the new code impacts related services at a given point in time, creates a high cognitive overload zapping developer time and brain cycles. Developers in a complex microservice environment have to manually jump between tabs in homegrown tools and curated dashboards to piece their service topology together.
Existing toolsets provide customers with details about each service individually—but not how they fit together. APM products can alert on changes caused during a deploy. What they can’t do is deliver useful workflows to understand the change and remediate impact in the related services.
Leveraging OpenTelemetry instrumentation, GitHub Actions, and Lightstep’s API we’ve threaded workflows together to provide timely alerts, insights, and resolution paths for performance impacts before and after code changes are released.
GitHub Actions enable GitHub workflows to be aware of what happens before and after a code change is deployed. Our Pre-Deploy Check and the Services Change Report enable developers to get rich feedback on the impact of code changes inside of pull requests and issues. In addition, as founders and core contributors to the OpenTelemetry project, we’ve recently donated code that can annotate code with GitHub project metadata such as repository name and version.
First, we use an OpenTelemetry plugin to detect if code is running in a GitHub environment. The plugin automatically picks up environment information to determine if they are running code in an action workflow. The GitHub Actions Resource Detector picks up all that context, i.e. continuous integration scenarios code runs in an action workflow or building Docker images and injects information into how that image then performs in production. The action itself reports telemetry i.e. the context of what is running in GitHub and what code is running and pulling it into Lightstep telemetry.
By automatically bringing relevant Observability data directly into the development workflow, the Lightstep Pre-Deploy Check provides a context-rich view of the health of the whole system — related services, service ownership, latest error status — before code is merged, and inside a pull request. Developers can now understand the state of production, before merges and deploys happen, without having to switch between multiple screens. After they merge their code, developers have ready access to details they need to troubleshoot in the event that something went wrong.
With the Lightstep Services Change Report, we’ve built a model of production services inside of a GitHub action leveraging OpenTelemetry, Lightstep, and Partner APIs to provide context and response options to different events. The Action dynamically infers of services from a snapshot and combines Lightstep service relationships and service data with partner data about services. This means that when a software team wants to go through a workflow to make a change from GitHub to a running service in an environment - telemetry from the running services can be connected to code changes. Lightstep threads these events together so that developers can understand “what caused that change.”
Leveraging Lightstep data, the action identifies performance related issues within the affected services enabling the developer to go into investigative mode. To understand what caused the change, you can double click on attributes that are connected to the service to examine before and after status.
Downstream dependencies and key operations are highlighted so you can see what's new, what was added, and what changed with related services.
The Lightstep Observability Toolkit for GitHub developers enables Lightstep customers and community tier members to know what changed before and after an event as it relates to the service architecture or topology. OpenTelemetry instrumentation brings more detail and broadens the types of triggers or use cases developers can consider for analysis. Leveraging open standards, latent extensible capabilities within Lightstep and GitHub, we’ve built an out-of-the-box workflow designed to give developers the visibility they need so they can focus on the thing that matters most — shipping good code.