Istio Distributed Tracing: How to Get Started with Lightstep and Kubernetes

Lightstep TracingLightstep Tracing is an easy way to start using distributed tracing without deploying your own distributed tracing system. IstioIstio is a “batteries included” set of best practices for deploying and managing containerized software. Istio proxyIstio proxy provides an automatic service mesh, based on EnvoyEnvoy, so that you can understand and control how different services communicate with each other. Envoy and therefore Istio support distributed tracing out of the box.

For this walkthrough, we’ll be using Google Kubernetes Engine (GKE) for our KubernetesKubernetes cluster. We’ll assume you have a working gcloud CLI installationgcloud CLI installation that includes kubectl as well (gcloud components install kubectl).

Creating a Cluster

Step 1: Create the cluster

export CLUSTER_NAME=lst-walkthrough # name of the GKE cluster
export PROJECT_NAME= # name of the project to create cluster in
export ZONE=us-central1-a # zone to create cluster in

gcloud container clusters create $CLUSTER_NAME \
--cluster-version latest \
--machine-type n1-standard-2 \ # istio-telemetry failed to schedule with n1-standard-1
--num-nodes 3 \
--preemptible \ # to reduce costs for proof of concept
--zone $ZONE \
--project $PROJECT_NAME

Step 2: Store credentials locally for kubectl

gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT_NAME

Step 3: Grant our user cluster-admin role

kubectl create clusterrolebinding cluster-admin-binding –clusterrole=cluster-admin –user=$(gcloud config get-value core/account)

Step 4: Create the Istio system namespace for the components

kubectl create namespace istio-system

Step 5: Download and unpack an Istio distribution

export ISTIO_VERSION=1.1.5
wget https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-osx.tar.gz
tar xcvf istio-$ISTIO_VERSION-osx.tar.gz
cd istio-$ISTIO_VERSION

This is for OS X, if you’re using a different OS, refer to the Istio directionsIstio directions.

Step 6: Initialize Istio system certificates and Custom Resource Definitions (CRDsCRDs)

helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

Step 7: Check for completion of the creation of the CRDs

kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l

In version 1.1.5, when complete this should print “53”

Step 8: Sign up for Lightstep Tracing

1. Navigate to https://go.lightstep.com/tracing.htmlhttps://go.lightstep.com/tracing.html

2. Fill out the form

3. Click on the link in the email to set up your account

4. Go to Settings and copy the token

Step 9: Set up Istio with Helm

export ACCESS_TOKEN=""
helm template --set pilot.traceSampling=100 --set global.proxy.tracer="lightstep" --set global.tracer.lightstep.address="ingest.lightstep.com:443" --set global.tracer.lightstep.accessToken=$ACCESS_TOKEN --set global.tracer.lightstep.secure=true --set global.tracer.lightstep.cacertPath="/etc/lightstep/cacert.pem" install/kubernetes/helm/istio --name istio --namespace istio-system > $HOME/istio.yaml
kubectl apply -f ~/istio.yaml

Step 10: Create a cacert.pem file with the Let’s Encrypt Root CA

curl https://letsencrypt.org/certs/trustid-x3-root.pem.txt -o cacert.pem

The current version of the Istio Lightstep integration requires a custom CA cert bundle to be specified. The public Lightstep collectors at ingest.lightstep.com are from Let’s Encrypt.

Step 11: Add the file as a secret for the Lightstep integration to use

kubectl create secret generic lightstep.cacert –from-file=cacert.pem

Step 12: Label your default namespace so that Istio will inject the Istio Proxy sidecarinject the Istio Proxy sidecar automatically

kubectl label namespace default istio-injection=enabled

Step 13: Wait for all pods to show as running (this can take a few minutes)

kubectl get pods –namespace istio-system

Step 14: Create the example BookInfo app and gateway:

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Step 15: Capture the information necessary to access the BookInfo app locally and use open to load it in your web browser

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')

Step 16: Open BookInfo in your browser

open http://$INGRESS_HOST:$INGRESS_PORT/productpage

Viewing Distributed Traces

You can now go to https://app.lightstep.com/https://app.lightstep.com/ and see traces for the example application.

Yay! A distributed trace! You can now see end-to-end transactions in your system. But, you may ask, how did it actually happen? It’s a common misunderstanding that when tracing with your service mesh, there aren’t any code changes. In fact, it’s necessary for services to pass through distributed tracingdistributed tracing headers even if they are not participating in the trace. Let’s walk through those headers for Istio and where that’s implemented in the BookInfo sample applicationBookInfo sample application.

The Istio tracing documentationIstio tracing documentation lists the following headers required to be forwarded:

• x-request-id

• x-b3-traceid

• x-b3-spanId

• x-b3-parentspanid

• x-b3-sampled

• x-b3-flags

• b3

If using Lightstep also:

• x-ot-span-context

However, looking at the productpage service sourceproductpage service source, the only non-b3 header being forwarded is x-request-id. So, how is this working? Istio Proxy’s Lightstep integration supports a special tag called guidspecial tag called guid. Since the x-request-id header is being forwarded and that header is being tagged as a guid on all the Istio Proxy spans, Lightstep is able to infer the ordering and parentage of the spansspans with only that information. A change to the productpage source to properly forward the header needs to be madeneeds to be made. For the other applications here are the places where the headers are captured and forwarded:

DetailsDetails (Ruby) CapturedCaptured ForwardedForwarded ReviewsReviews (Java) CapturedCaptured ForwardedForwarded

As you can see with the above list, there may be many headers to forward if you want to support Zipkin/Jaeger B3 headers, OpenTracing headers, and Istio Proxy (Envoy) headers. With the standardization of tracing headers with W3C Trace ContextW3C Trace Context and OpenTelemetryOpenTelemetry this should be much simpler in the future.