Tracing helps gather timing data to troubleshoot latency problems in microservice architectures and helps to understand a complete transaction or workflow as it propagates through a distributed system. Every transaction may reflect performance anomalies in an early phase when new services are being introduced by independent teams.

Tracing the Che application may help analyze the execution of various operations, such as workspace creations, workspace startup, breaking down the duration of sub-operations executions, helping finding bottlenecks and improve the overall state of the platform.

Tracers live in applications. They record timing and metadata about operations that take place. They often instrument libraries, so that their use is transparent to users. For example, an instrumented web server records when it received a request and when it sent a response. The trace data collected is called a span. A span has a context that contains information such as trace and span identifiers and other kinds of data that can be propagated down the line.

Tracing API

Che utilizes OpenTracing API - a vendor-neutral framework for instrumentation. This means that if a developer wants to try a different tracing back end, then instead of repeating the whole instrumentation process for the new distributed tracing system, the developer can simply change the configuration of the tracer back end.

Tracing back end

By default, Che uses Jaeger as the tracing back end. Jaeger was inspired by Dapper and OpenZipkin, and it is a distributed tracing system released as open source by Uber Technologies. Jaeger extends a more complex architecture for a larger scale of requests and performance.

Installing the Jaeger tracing tool

The following sections describe the installation methods for the Jaeger tracing tool. Jaeger can then be used for gathering metrics in Che.

Installation methods available:

For tracing a Che instance using Jaeger, version 1.12.0 or above is required. For additional information about Jaeger, see the Jaeger website.

Installing the Jaeger tracing tool for Che on OpenShift 4

To install the Jaeger tracing tool from a Che project in OpenShift Container Platform, follow the instructions in this section.

Prerequisites

  • The user is logged in to the OpenShift Container Platform web console.

  • Che is installed in a OpenShift Container Platform cluster.

Procedure

  1. In the Che installation namespace of the OpenShift Container Platform cluster, use the oc client to create a new application for the Jaeger deployment.

    $ oc new-app -f / ${CHE_LOCAL_GIT_REPO}/deploy/openshift/templates/jaeger-all-in-one-template.yml:

--> Deploying template "<project_name>/jaeger-template-all-in-one" for "/home/user/che-projects/che/deploy/openshift/templates/jaeger-all-in-one-template.yml" to project <project_name>

     Jaeger (all-in-one)
     ---------
     Jaeger Distributed Tracing Server (all-in-one)

     * With parameters:
        * Jaeger Service Name=jaeger
        * Image version=latest
        * Jaeger Zipkin Service Name=zipkin

--> Creating resources ...
    deployment.apps "jaeger" created
    service "jaeger-query" created
    service "jaeger-collector" created
    service "jaeger-agent" created
    service "zipkin" created
    route.route.openshift.io "jaeger-query" created
--> Success
    Access your application using the route: 'jaeger-query-<project_name>.apps.ci-ln-whx0352-d5d6b.origin-ci-int-aws.dev.rhcloud.com'
    Run 'oc status' to view your app.

  2. Using the Workloads → Deployments from the left menu of main OpenShift Container Platform screen, monitor the Jaeger deployment until it finishes successfully.

  3. Select Networking → Routes from the left menu of the main OpenShift Container Platform screen, and click the URL link to access the Jaeger dashboard.

  4. Follow the steps in Enabling Che metrics collections to finish the procedure.

Installing Jaeger using OperatorHub on OpenShift 4

To install the Jaeger tracing tool from the OperatorHub interface in OpenShift Container Platform, follow the instructions below.

Prerequisites

  • The user is logged in to the OpenShift Container Platform Web Console.

  • A Che instance is deployed in a namespace.

Procedure

  1. Open the OpenShift Container Platform console.

  2. From the left menu of the main OpenShift Container Platform screen, navigate to Operators → OperatorHub.

  3. In the Search by keyword search bar, type Jaeger Operator.

  4. Click the Jaeger Operator tile.

  5. Click the Install button in the Jaeger Operator pop-up window.

  6. Select the installation method: A specific namespace on the cluster where the Che is deployed and leave the rest in its default values.

  7. Click the Subscribe button.

  8. From the left menu of the main OpenShift Container Platform screen, navigate to the Operators → Installed Operators section.

  9. Eclipse Che is displayed as an Installed Operator, as indicated by the InstallSucceeded status.

  10. Click the Jaeger Operator name in the list of installed operators.

  11. Navigate to the Overview tab.

  12. In the Conditions sections at the bottom of the page, wait for this message: install strategy completed with no errors.

  13. Jaeger Operator and additional Elasticsearch Operator is installed.

  14. Navigate to the Operators → Installed Operators section.

  15. Click Jaeger Operator in the list of installed operators.

  16. The Jaeger Cluster page is displayed.

  17. In the bottom-left corner of the window, click Create Instance

  18. Click Save.

  19. The Jaeger cluster jaeger-all-in-one-inmemory is created.

  20. Follow the steps in Enabling Che metrics collections to finish the procedure.

Installing Jaeger locally using a prebuilt container image

To install the Jaeger tracing tool locally using a prebuilt container image, follow the instructions in this section.

Prerequisites
Procedure

  1. Start Podman:

    $ sudo service podman start

  2. Install Jaeger using the pre-built container image.

     $ podman run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.15

  3. Navigate to http://localhost:16686 to access the Jaeger UI.

  4. Follow the steps in Enabling Che metrics collections to finish the procedure.

Enabling Che metrics collections

Prerequisites
Procedure

For Jaeger tracing to work, enable the following environment variables in your Che deployment:

# Activating Che tracing modules
CHE_TRACING_ENABLED=true

# Following variables are the basic Jaeger client library configuration.
JAEGER_ENDPOINT="http://jaeger-collector:14268/api/traces"

# Service name
JAEGER_SERVICE_NAME="che-server"

# URL to remote sampler
JAEGER_SAMPLER_MANAGER_HOST_PORT="jaeger:5778"

# Type and param of sampler (constant sampler for all traces)
JAEGER_SAMPLER_TYPE="const"
JAEGER_SAMPLER_PARAM="1"

# Maximum queue size of reporter
JAEGER_REPORTER_MAX_QUEUE_SIZE="10000"

To enable the following environment variables:

  1. In the yaml source code of the Che deployment, add the following configuration variables under spec.server.customCheProperties.

    customCheProperties:
      CHE_TRACING_ENABLED: 'true'
      JAEGER_SAMPLER_TYPE: const
      DEFAULT_JAEGER_REPORTER_MAX_QUEUE_SIZE: '10000'
      JAEGER_SERVICE_NAME: che-server
      JAEGER_ENDPOINT: 'http://jaeger-collector:14268/api/traces'
      JAEGER_SAMPLER_MANAGER_HOST_PORT: 'jaeger:5778'
      JAEGER_SAMPLER_PARAM: '1'

  2. Edit the JAEGER_ENDPOINT value to match the name of the Jaeger collector service in your deployment.

    From the left menu of the main OpenShift Container Platform screen, obtain the value of JAEGER_ENDPOINT by navigation to Networking → Services. Alternatively, execute the following oc command:

    $ oc get services

    The requested value is included in the service name that contains the collector string.

Additional resources

Viewing Che traces in Jaeger UI

This section demonstrates how to utilize the Jaeger UI to overview traces of Che operations.

Procedure

In this example, the Che instance has been running for some time and one workspace start has occurred.

To inspect the trace of the workspace start:

  1. In the Search panel on the left, filter spans by the operation name (span name), tags, or time and duration.

    trace search
    Figure 1. Using Jaeger UI to trace Che

  2. Select the trace to expand it and show the tree of nested spans and additional information about the highlighted span, such as tags or durations.

    trace tree expanded
    Figure 2. Expanded tracing tree

Che tracing codebase overview and extension guide

The core of the tracing implementation for Che is in the che-core-tracing-core and che-core-tracing-web modules.

All HTTP requests to the tracing API have their own trace. This is done by TracingFilter from the OpenTracing library, which is bound for the whole server application. Adding a @Traced annotation to methods causes the TracingInterceptor to add tracing spans for them.

Tagging

Spans may contain standard tags, such as operation name, span origin, error, and other tags that may help users with querying and filtering spans. Workspace-related operations (such as starting or stopping workspaces) have additional tags, including userId, workspaceID, and stackId. Spans created by TracingFilter also have an HTTP status code tag.

Declaring tags in a traced method is done statically by setting fields from the TracingTags class:

TracingTags.WORKSPACE_ID.set(workspace.getId());

TracingTags is a class where all commonly used tags are declared, as respective AnnotationAware tag implementations.

Additional resources

For more information about how to use Jaeger UI, visit Jaeger documentation: Jaeger Getting Started Guide.

Tags: