Enhancements in [x]PM Monitoring

Today, we’re announcing a suite of new features that make LightStep [x]PM monitoring both more powerful and easier to understand.

Our customers often tell us they’re able to diagnose their systems’ problems much faster with LightStep than other tools. One key reason for this is LightStep’s ability to quickly and accurately monitor latency statistics and error rates. However, many customers also ran into limits with our SLAs: only one latency SLA per saved search, and one error rate SLA.

We’re happy to announce that we’re removing those limits. SLAs are now called Conditions, and they have powerful new capabilities.

Unlimited conditions

Monitor any number of conditions for each saved searchMonitor any number of conditions for each saved search

You can now have any number of conditions in LightStep – create three latency conditions for different latency percentiles, different error conditions to notify different groups, or send notifications to different places depending on the severity of the condition. You can write monitoring conditions that are flexible and fit in with your team’s workflow.

New signals to monitor

Monitor latency, operation rate, or error rateMonitor latency, operation rate, or error rate

In addition to monitoring the latency and error rate in your saved searches, you can now monitor the operations-per-second rate for a saved search. This is useful if you need to monitor whether a service is online and the load is in line with your expectations.

You can also monitor whether any value is above or below a threshold. This is great for setting bounds on operation rates, so you get notified if traffic is significantly above or below your expectations. Operation rates, error rates, and latency percentiles can all have “greater than” and “less than” conditions.

Monitor over any time range

LightStep [x]PM monitoring has been tuned for quick response to signals, and by default, it monitors the last two minutes of data from your system. We’ve learned from you that sometimes you’d prefer to monitor the last hour, day, or even last week of performance data for some systems. LightStep [x]PM monitoring now lets you control the evaluation window of your conditions, giving you flexibility between extremely sensitive monitoring and the ability to detect more sustained trends.

Unlimited notifications

When a condition is violated, you can now send notifications to as many Slack, PagerDuty, or Webhook destinations as you like, rather than one to each.

Send notifications to PagerDuty, Slack, or WebhooksSend notifications to PagerDuty, Slack, or Webhooks

Snooze notifications

You can temporarily add a “snooze” for any condition to prevent it from sending notifications. This can be useful when experimenting with new condition settings or during a pager storm. We’ll keep track of who applied the snooze and when, so you’ll always know “why didn’t I get paged?”

Getting started with monitoring

All your existing SLAs work the same as they did before, and you can find them in the “Monitoring” section of the LightStep navigation bar, represented by the bell. You can create new Conditions for a saved search on the “Saved Search” page by opening the “Conditions” drawer on the right-hand side.

Monitoring is under the Bell icon on the left-hand sideMonitoring is under the Bell icon on the left-hand side

We’re excited to give you improved and more powerful monitoring in LightStep [x]PM, so you can go even faster from an alert to a root cause. Give it a try and share your feedback with us.

GoLang Dep: The Missing Manual

At LightStep we run a number of applications written in Go that handle data ingestion from customers, query processing, monitoring and a variety of other tasks. We’ve adopted dep to manage dependencies for these apps. As our engineering team has grown, and as the number of different applications have grown, it’s helped us stay sane when dealing with changes in external dependencies.

During the transition from a custom vendor management solution to dep, I found some of the important documentation to be scattered around. This post collects the most helpful “extra” information I wish I had known getting started.

What’s dep?

dep is a dependency management tool for the Go programming language. We elected to use dep because of its close relationship with the official Go toolchain developers and its straightforward, “unmagical” dependency management model.

It has an active and helpful development community over on gophers.slack.com#vendor and an exciting roadmap. If you start using dep you should join up and get to know the community. It’s a great time to help find edge cases and make this tool better.

Set up dep

Follow the steps in the dep README to install dep. The homebrew release is up-to-date and is the recommended way to install dep.

Where to run dep

dep should be run at the project root—the directory just above where your vendor directory sits. depassumes that any packages that cannot be reached in your GOPATH by navigating down from where it’s run are external packages that need to be added to the vendor directory.

To get started, run dep init. This will read your application code and generate a set of constraints based on its best solution for your dependencies. After running this, you may need to edit it to be more specific about constraints, especially if you depend on older versions of some projects.


Read this: Gopkg.toml README

The Gopkg.toml file describes all the dependencies required for your project. It only describes primary dependencies, not transitive dependencies, leaving the dep constraint solver free to pick transitive dependency versions as long as a version can be chosen that satisfies all constraints.

The most important fields in the file are the constraint entries. Constraints look like this:

  name = "github.com/lightstep/lightstep-tracer-go"
  version = "v0.14.0"

Every constraint needs a name, which is actually the URI you would use when go geting the project. Every constraint should also have either a versionbranch, or revision, with version being preferred if you can use it.

version fields use SemVer 2.0 [http://semver.org/ ] and dep assumes that v1.2.0 means ^v1.2.0. If you need to constrain to an exact version, use =v1.2.0.

Never edit the lock file!

Gopkg.lock is really an output of the constraint solver. Editing it does nothing.

dep ensure -v is your friend

It’s possible to describe a set of constraints that cannot be solved—you may have a primary dependency on a version of a package, and one of your primary dependencies may depend on an incompatible version. In that case, dep ensure will fail and print an error message. By running dep ensure -v you will get detailed output from the constraint solver that can help you identify the source of the problem.

dep ensure doesn’t do much if Your constraints are already satisfied!

dep ensure will “ensure” that a package that you import is also installed in your vendor directory and satisfies any described constraints… That’s it! It won’t make sure you have the latest release if your constraints are already satisfied.

Use dep ensure -update pgkname to get the latest version that satisfies constraints.

Gopkg.toml trick: required

Sometimes you need some go code included that your application doesn’t run directly. For example, https://github.com/grpc-ecosystem/grpc-gateway generates code which can then be committed to a repository, but a dep ensure will not install it, and a dep prune would remove it if installed.

The required keyword lets you depend on repositories that are not dependencies of your application like so:

required = ["github.com/grpc-ecosystem/grpc-gateway"]

Note that after ensuring it’s installed, you still need to go install your requirements:

$ dep ensure
$ go install vendor/github.com/grpc-ecosystem/grpc-gateway/...

There are other tools that can be used to make these installations project-specific as well, like virtualgo: https://github.com/GetStream/vg

Gopkg.toml trick: ignored

The ignored keyword prevents a package and its transitive dependencies from being installed in a project. Why would you want to do that? A typical use case might be to support updating to a new major version of a library that removes a package. Let’s say that you depend on github.org/foo/bar/bazsomewhere in your application, and version 2.0.0 of foo/bar drops this package.

Original Gopkg.toml:

  name = "github.com/foo/bar"
  version = "1.2.1"

Updating to 2.0.0 without changing your source code:

ignored = ["github.com/foo/bar/baz"]

  name = "github.com/foo/bar"
  version = "2.0.0"

This can help you install the new version of your library, even though it doesn’t have all the packages required by your application. You can then work to transition your application code while having the source for the library version you’re working with installed locally.

Committing vendor

If you’re writing a library, especially an open-source library, it’s not generally a good idea to commit your vendor directory. Any users who go get your code may have an impossible time compiling if they have conflicting dependencies with you. It’s a great idea, however, to commit and share your Gopkg.toml file. This will help other users of dep easily consume your library.

If you’re writing an applicaiton that emits binaries, there are some arguments for committing your vendor directory as an application that builds a binary and some arguments against it.

In favor of committing vendor

  • You have all the source and binaries needed to build your application in your repository. This can speed CI builds by avoiding a lot of downloading and dependency resolution when building. It also gives you a repeatable set of source to build from.
  • You’re protected from upstream changes breaking your builds—if a dependency unpublishes a previous release, you’ll still have a copy.

Against committing vendor

  • You may be storing and handling a lot of code that isn’t directly related to your application in your own repository.
  • Changes that include dependency upgrades will have very large diffs and may be unwieldy.

We’ve chosen, for the time being, to commit our vendor directories for our application binaries.

Making your project dep friendly

This section assumes you’re hosting your project as a git repository. Similar rules hold for bzr and hg as well.

Use annotated git tags to mark releases. git tag -a v2.1.3 -m "Release Version 2.1.3"Including release notes is a great idea. Leave off the -m argument to open your editor and add a longer tag message.

Mark them with 3-digit SemVer tags: v2.1.3.

Be honest about what a “breaking change” is

Any change that would cause a build failure if it is installed is considered a “breaking change” by SemVer and should be released with a major revision update. Changes that add new APIs can be considered minor releases, and changes that do not modify the API of the project can be considered a “patch” version.

We’ve found dep to be a straightforward, understandable tool for managing application dependencies. It fits nicely into our CI and development workflows, and it’s easy to understand which version of a given dependency is currently being used in a build. Let us know in the comments about your experiences with dep and share any information you’ve found helpful when using the tool!