ingress-nginx-helm/docs/user-guide/third-party-addons/opentracing.md

OpenTracing

Enables requests served by NGINX for distributed tracing via The OpenTracing Project.

Using the third party module opentracing-contrib/nginx-opentracing the NGINX ingress controller can configure NGINX to enable OpenTracing instrumentation. By default this feature is disabled.

Usage

To enable the instrumentation we must enable OpenTracing in the configuration ConfigMap:

data:
  enable-opentracing: "true"

To enable or disable instrumentation for a single Ingress, use the enable-opentracing annotation:

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentracing: "true"

We must also set the host to use when uploading traces:

zipkin-collector-host: zipkin.default.svc.cluster.local
jaeger-collector-host: jaeger-agent.default.svc.cluster.local
datadog-collector-host: datadog-agent.default.svc.cluster.local

NOTE: While the option is called jaeger-collector-host, you will need to point this to a jaeger-agent, and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent.

Next you will need to deploy a distributed tracing system which uses OpenTracing. Zipkin and Jaeger and Datadog have been tested.

Other optional configuration options:

# specifies the name to use for the server span
opentracing-operation-name

# specifies specifies the name to use for the location span
opentracing-location-operation-name

# specifies the port to use when uploading traces, Default: 9411
zipkin-collector-port

# specifies the service name to use for any traces created, Default: nginx
zipkin-service-name

# specifies sample rate for any traces created, Default: 1.0
zipkin-sample-rate

# specifies the port to use when uploading traces, Default: 6831
jaeger-collector-port

# specifies the endpoint to use when uploading traces to a collector instead of an agent
jaeger-endpoint

# specifies the service name to use for any traces created, Default: nginx
jaeger-service-name

# specifies the traceparent/tracestate propagation format
jaeger-propagation-format

# specifies the sampler to be used when sampling traces.
# The available samplers are: const, probabilistic, ratelimiting, remote, Default: const
jaeger-sampler-type

# specifies the argument to be passed to the sampler constructor, Default: 1
jaeger-sampler-param

# Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL.
# Default: http://127.0.0.1
jaeger-sampler-host

# Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. Default: 5778
jaeger-sampler-port

# Specifies the header name used for passing trace context. Must be a string. Default: uber-trace-id
jaeger-trace-context-header-name

# Specifies the header name used for force sampling. Must be a string. Default: jaeger-debug-id
jaeger-debug-header

# Specifies the header name used to submit baggage if there is no root span. Must be a string. Default: jaeger-baggage
jaeger-baggage-header

# Specifies the header prefix used to propagate baggage. Must be a string. Default: uberctx-
jaeger-tracer-baggage-header-prefix

# specifies the port to use when uploading traces, Default 8126
datadog-collector-port

# specifies the service name to use for any traces created, Default: nginx
datadog-service-name

# specifies the environment this trace belongs to, Default: prod
datadog-environment

# specifies the operation name to use for any traces collected, Default: nginx.handle
datadog-operation-name-override

# Specifies to use client-side sampling for distributed priority sampling and ignore sample rate, Default: true
datadog-priority-sampling

# specifies sample rate for any traces created, Default: 1.0
datadog-sample-rate

All these options (including host) allow environment variables, such as $HOSTNAME or $HOST_IP. In the case of Jaeger, if you have a Jaeger agent running on each machine in your cluster, you can use something like $HOST_IP (which can be 'mounted' with the status.hostIP fieldpath, as described here) to make sure traces will be sent to the local agent.

Examples

The following examples show how to deploy and test different distributed tracing systems. These example can be performed using Minikube.

Zipkin

In the rnburn/zipkin-date-server GitHub repository is an example of a dockerized date service. To install the example and Zipkin collector run:

kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml
kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml

Also we need to configure the NGINX controller ConfigMap with the required values:

$ echo '
apiVersion: v1
kind: ConfigMap
data:
  enable-opentracing: "true"
  zipkin-collector-host: zipkin.default.svc.cluster.local
metadata:
  name: ingress-nginx-controller
  namespace: kube-system
' | kubectl replace -f -

In the Zipkin interface we can see the details:

Jaeger

1. Enable Ingress addon in Minikube:

   $ minikube addons enable ingress
   

2. Add Minikube IP to /etc/hosts:

   $ echo "$(minikube ip) example.com" | sudo tee -a /etc/hosts
   

3. Apply a basic Service and Ingress Resource:

   # Create Echoheaders Deployment
   $ kubectl run echoheaders --image=k8s.gcr.io/echoserver:1.4 --replicas=1 --port=8080
   
   # Expose as a Cluster-IP
   $ kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x
   
   # Apply the Ingress Resource
   $ echo '
     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: echo-ingress
     spec:
       rules:
       - host: example.com
         http:
           paths:
           - backend:
               serviceName: echoheaders-x
               servicePort: 80
             path: /echo
     ' | kubectl apply -f -
   

4. Enable OpenTracing and set the jaeger-collector-host:

   $ echo '
     apiVersion: v1
     kind: ConfigMap
     data:
       enable-opentracing: "true"
       jaeger-collector-host: jaeger-agent.default.svc.cluster.local
     metadata:
       name: ingress-nginx-controller
       namespace: kube-system
     ' | kubectl replace -f -
   

5. Apply the Jaeger All-In-One Template:

   $ kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml
   

6. Make a few requests to the Service:

   $ curl example.com/echo -d "meow"
   
   CLIENT VALUES:
   client_address=172.17.0.5
   command=POST
   real path=/echo
   query=nil
   request_version=1.1
   request_uri=http://example.com:8080/echo
   
   SERVER VALUES:
   server_version=nginx: 1.10.0 - lua: 10001
   
   HEADERS RECEIVED:
   accept=*/*
   connection=close
   content-length=4
   content-type=application/x-www-form-urlencoded
   host=example.com
   user-agent=curl/7.54.0
   x-forwarded-for=192.168.99.1
   x-forwarded-host=example.com
   x-forwarded-port=80
   x-forwarded-proto=http
   x-original-uri=/echo
   x-real-ip=192.168.99.1
   x-scheme=http
   BODY:
   meow
   

7. View the Jaeger UI:

   $ minikube service jaeger-query --url
   
   http://192.168.99.100:30183
   

   In the Jaeger interface we can see the details: