At LightStep we run a number of applications written in Go that handle data ingestion from customers, query processing, monitoring and a variety of other tasks. We’ve adopted dep to manage dependencies for these apps. As our engineering team has grown, and as the number of different applications have grown, it’s helped us stay sane when dealing with changes in external dependencies.
During the transition from a custom vendor management solution to dep, I found some of the important documentation to be scattered around. This post collects the most helpful “extra” information I wish I had known getting started.
dep is a dependency management tool for the Go programming language. We elected to use dep because of its close relationship with the official Go toolchain developers and its straightforward, “unmagical” dependency management model.
It has an active and helpful development community over on gophers.slack.com#vendor and an exciting roadmap. If you start using
dep you should join up and get to know the community. It’s a great time to help find edge cases and make this tool better.
Set up dep
Follow the steps in the dep README to install
dep. The homebrew release is up-to-date and is the recommended way to install dep.
Where to run
dep should be run at the project root—the directory just above where your
vendor directory sits.
depassumes that any packages that cannot be reached in your
GOPATH by navigating down from where it’s run are external packages that need to be added to the
To get started, run
dep init. This will read your application code and generate a set of constraints based on its best solution for your dependencies. After running this, you may need to edit it to be more specific about constraints, especially if you depend on older versions of some projects.
Read this: Gopkg.toml README
The Gopkg.toml file describes all the dependencies required for your project. It only describes primary dependencies, not transitive dependencies, leaving the
dep constraint solver free to pick transitive dependency versions as long as a version can be chosen that satisfies all constraints.
The most important fields in the file are the constraint entries. Constraints look like this:
name = "github.com/lightstep/lightstep-tracer-go"
version = "v0.14.0"
Every constraint needs a name, which is actually the URI you would use when
go geting the project. Every constraint should also have either a
version being preferred if you can use it.
version fields use SemVer 2.0 [http://semver.org/ ] and
dep assumes that
^v1.2.0. If you need to constrain to an exact version, use
Never edit the lock file!
Gopkg.lock is really an output of the constraint solver. Editing it does nothing.
dep ensure -v is your friend
It’s possible to describe a set of constraints that cannot be solved—you may have a primary dependency on a version of a package, and one of your primary dependencies may depend on an incompatible version. In that case,
dep ensure will fail and print an error message. By running
dep ensure -v you will get detailed output from the constraint solver that can help you identify the source of the problem.
dep ensure doesn’t do much if Your constraints are already satisfied!
dep ensure will “ensure” that a package that you import is also installed in your
vendor directory and satisfies any described constraints… That’s it! It won’t make sure you have the latest release if your constraints are already satisfied.
dep ensure -update pgkname to get the latest version that satisfies constraints.
Sometimes you need some go code included that your application doesn’t run directly. For example, https://github.com/grpc-ecosystem/grpc-gateway generates code which can then be committed to a repository, but a
dep ensure will not install it, and a
dep prune would remove it if installed.
required keyword lets you depend on repositories that are not dependencies of your application like so:
required = ["github.com/grpc-ecosystem/grpc-gateway"]
Note that after ensuring it’s installed, you still need to
go install your requirements:
$ dep ensure
$ go install vendor/github.com/grpc-ecosystem/grpc-gateway/...
There are other tools that can be used to make these installations project-specific as well, like virtualgo: https://github.com/GetStream/vg
ignored keyword prevents a package and its transitive dependencies from being installed in a project. Why would you want to do that? A typical use case might be to support updating to a new major version of a library that removes a package. Let’s say that you depend on
github.org/foo/bar/bazsomewhere in your application, and version 2.0.0 of
foo/bar drops this package.
name = "github.com/foo/bar"
version = "1.2.1"
Updating to 2.0.0 without changing your source code:
ignored = ["github.com/foo/bar/baz"]
name = "github.com/foo/bar"
version = "2.0.0"
This can help you install the new version of your library, even though it doesn’t have all the packages required by your application. You can then work to transition your application code while having the source for the library version you’re working with installed locally.
If you’re writing a library, especially an open-source library, it’s not generally a good idea to commit your
vendor directory. Any users who
go get your code may have an impossible time compiling if they have conflicting dependencies with you. It’s a great idea, however, to commit and share your Gopkg.toml file. This will help other users of
dep easily consume your library.
If you’re writing an applicaiton that emits binaries, there are some arguments for committing your vendor directory as an application that builds a binary and some arguments against it.
In favor of committing vendor
- You have all the source and binaries needed to build your application in your repository. This can speed CI builds by avoiding a lot of downloading and dependency resolution when building. It also gives you a repeatable set of source to build from.
- You’re protected from upstream changes breaking your builds—if a dependency unpublishes a previous release, you’ll still have a copy.
Against committing vendor
- You may be storing and handling a lot of code that isn’t directly related to your application in your own repository.
- Changes that include dependency upgrades will have very large diffs and may be unwieldy.
We’ve chosen, for the time being, to commit our vendor directories for our application binaries.
Making your project
This section assumes you’re hosting your project as a git repository. Similar rules hold for bzr and hg as well.
Use annotated git tags to mark releases.
git tag -a v2.1.3 -m "Release Version 2.1.3"Including release notes is a great idea. Leave off the
-m argument to open your editor and add a longer tag message.
Mark them with 3-digit SemVer tags: v2.1.3.
Be honest about what a “breaking change” is
Any change that would cause a build failure if it is installed is considered a “breaking change” by SemVer and should be released with a major revision update. Changes that add new APIs can be considered minor releases, and changes that do not modify the API of the project can be considered a “patch” version.
dep to be a straightforward, understandable tool for managing application dependencies. It fits nicely into our CI and development workflows, and it’s easy to understand which version of a given dependency is currently being used in a build. Let us know in the comments about your experiences with dep and share any information you’ve found helpful when using the tool!