Che can expose certain data as metrics, that can be processed by Prometheus and Grafana stack. Prometheus is a monitoring system, that maintains the collection of metrics - time series key-value data which can represent consumption of resources like CPU and memory, amount of processed HTTP queries and their execution time, and Che specific resources, such as number of users and workspaces, the start and shutdown of workspaces, information about JsonRPC stack.

Prometheus is powered with a special query language, that allows manipulating the collected data, and perform various binary, vector and aggregation operations with it, to help create a more refined view on data.

While Prometheus is the central piece, responsible for scraping and storing the metrics, while Grafana offers a front-end "facade" with tools to create a various visual representation in the form of dashboards with various panels and graph types.

Note that this monitoring stack is not an official production-ready solution, but rather has an introduction purpose.

monitoring che stack structure
Figure 1. The structure of Che monitoring stack

Enabling Che metrics collections

Prerequisites
Procedure

  1. Set the CHE_METRICS_ENABLED=true environment variable

  2. Expose the 8087 port as a service on the che-master host

  3. Configure Prometheus to scrape metrics from the 8087 port

  4. Configure a Prometheus data source on Grafana

  5. Deploy Che-specific dashboards on Grafana

Collecting Che metrics with Prometheus

Prometheus is a monitoring system that collects metrics in real time and stores them in a time series database.

Prometheus comes with a console accessible at the 9090 port of the application pod. By default, a template provides an existing service and a route to access it. It can be used to query and view metrics.

monitoring che prometheus console

Prometheus terminology

Prometheus offers:

counter

the simplest numerical type of metric whose value can be only increased. A typical example is counting the amount of HTTP requests that go through the system.

gauge

numerical value that can be increased or decreased. Best suited for representing values of objects.

histogram

a more complex metric that is suited for performing observations. Metrics are collected and grouped in configurable buckets, which allwos to present the results, for instance, in a form of a heatmap.

Configuring Prometheus

Prometheus configuration
- apiVersion: v1
  data:
    prometheus.yml: |-
      global:
        scrape_interval:     5s           (1)
        evaluation_interval: 5s           (2)
      scrape_configs:
        - job_name: 'che'
          static_configs:
            - targets: ['che-host:8087']
  kind: ConfigMap
  metadata:
    name: prometheus-config
1 rate, at which a target is scraped
2 rate, at which recording and alerting rules are re-checked (not used in our system at the moment)

Viewing Che metrics on Grafana dashboards

Grafana is used for informative representation of Prometheus metrics. Providing visibility for Openshift, Grafana’s deployment configuration and ConfigMaps are located in the che-monitoring.yaml configuration file.

Configuring and deploying Grafana

Grafana is run on port 3000 with a corresponding service and route.

Three ConfigMaps are used to configure Grafana:

  • grafana-datasources — configuration for Grafana datasource, a Prometheus endpoint

  • grafana-dashboards — configuration of Grafana dashboards and panels

  • grafana-dashboard-provider  — configuration of the Grafana dashboard provider API object, which tells Grafana where to look in the file system for pre-provisioned dashboards

Grafana dashboards overview

Che provides several types of dashboards.

Che server dashboard

Use case: Che server-specific metrics related to Che components, such as workspaces or users.

monitoring che che server dashboard general panel
Figure 2. The General panel

The General panel contains basic information, such as the total number of users and workspaces in the Che database.

monitoring che che server dashboard workspace panel
Figure 3. The Workspaces panel

  • Workspace start rate — the ratio between successful and failed started workspaces

  • Workspace stop rate — the ratio between successful and failed stopped workspaces

  • Workspace Failures — the number of workspace failures shown on the graph

  • Starting Workspaces — the gauge that shows the number of currently starting workspaces

  • Average Workspace Start Time — 1-hour average of workspace starts or fails

  • Average Workspace Stop Time — 1-hour average of workspace stops

  • Running Workspaces — the gauge that shows the number of currently running workspaces

  • Stopping Workspaces — the gauge that shows the number of currently stopping workspaces

  • Workspaces started under 60 seconds — the percentage of workspaces started under 60 seconds

  • Number of Workspaces — the number of workspaces created over time

monitoring che che server dashboard users panel
Figure 4. The Users panel

  • Number of Users — the number of users known to Che over time

monitoring che che server dashboard tomcat panel
Figure 5. The Tomcat panel

  • Max number of active sessions — the max number of active sessions that have been active at the same time

  • Number of current active sessions — the number of currently active sessions

  • Total sessions — the total number of sessions

  • Expired sessions — the number of sessions that have expired

  • Rejected sessions — the number of sessions that were not created because the maximum number of active sessions was reached

  • Longest time of an expired session — the longest time (in seconds) that an expired session had been alive

monitoring che che server dashboard requests panel
Figure 6. The Request panel

The Requests panel displays HTTP requests in a graph that shows the average number of requests per minute.

monitoring che che server dashboard jsonrpc panel
Figure 7. The JsonRPC panel

The JsonRPC panel show how the JsonRPC stack is used for sending asynchronous messages between the Che server and clients (for example, the GWT IDE).

There are two endpoints - major and minor. The major endpoint is the primary endpoint. It is used for sending important messages, such as workspace statuses. The minor endpoint is typically used by workspace starting logs.

  • JsonRpcQueue — the number of queued tasks on executors

  • Rejected message/minute — the number of messages that will be rejected if executors and queue are full

  • Seconds Max — the maximum time of executing a task through the JsonRPC stack in the given moment of time

  • Seconds Sum — the summary time of executing a task through the JsonRPC stack in the given moment of time

  • Completed Task/minute — the number of completed tasks per minute

The Traces panel

This panel is functional only if Che tracing has been enabled. image::monitoring/monitoring-che-che-server-dashboard-traces-panel.png[link="/che/docs/images/monitoring/monitoring-che-che-server-dashboard-traces-panel.png"]

  • Workspace start Max — the max time of traces execution, involved in workspace start

  • Workspace start Avg — 1-hour average time of traces execution, involved in workspace start

  • OpenshiftInternalRuntime#startMachines Max — the max time of traces execution, involved in machine starting phase during the workspace start

  • OpenshiftInternalRuntime#startMachines Avg — 1-hour average time of trace, involved in machine starting phase during the workspace start

  • Workspace stop Max — the max time of traces execution, involved in workspace start

  • Workspace stop Avg — 1-hour average time of traces execution, involved in workspace stop

monitoring che che server dashboard workspace detailed panel
Figure 8. The Workspace detailed panel

The Workspace Detailed panel contains heat maps, which illustrate the average time of workspace starts or fails. The row shows some period of time.

Che server JVM dashboard

Use case: JVM metrics of the Che server, such as JVM memory or classloading.

monitoring che che server jvm dashboard
Figure 9. Che server JVM dashboard
monitoring che che server jvm dashboard quick facts
Figure 10. Quick Facts
monitoring che che server jvm dashboard jvm memory
Figure 11. JVM Memory
monitoring che che server jvm dashboard jvm misc
Figure 12. JVM Misc
monitoring che che server jvm dashboard jvm memory pools heap
Figure 13. JVM Memory Pools (heap)
monitoring che che server jvm dashboard jvm memory pools non heap
Figure 14. JVM Memory Pools (Non-Heap)
monitoring che che server jvm dashboard garbage collection
Figure 15. Garbage Collection
monitoring che che server jvm dashboard classloading
Figure 16. Classloading
monitoring che che server jvm dashboard buffer pools
Figure 17. Buffer Pools

Developing Grafana dashboards

Grafana offers the possibility to add custom panels.

Procedure

To add a custom panel, use the New dashboard view.

  1. In the first section, define Queries to. Use the Prometheus Query Language to construct a specific metric, as well as to modify it with various aggregation operators.

    monitoring che new grafana dashboard queries
    Figure 18. New Grafana dashboard: Queries to

  2. In the Visualisation section, choose a metric to be shown in the following visual in the form of a graph, gauge, heatmap, or others.

    monitoring che new grafana dashboard visualization
    Figure 19. New Grafana dashboard: Visualization

  3. Save changes to the dashboard by clicking the Save button, and copy and paste the JSON code to the deployment.

  4. Load changes in the configuration of a running Grafana deployment. First remove the deployment:

    $ oc process -f che-monitoring.yaml | oc delete -f -

    Then redeploy your Grafana with the new configuration:

    $ oc process -f che-monitoring.yaml | oc apply -f - | oc rollout latest grafana

Extending Che monitoring metrics

There are two major modules for metrics:

  • che-core-metrics-core — contains core metrics module

  • che-core-api-metrics — contains metrics that are dependent on core Che components, such as workspace or user managers

Procedure

To create a metric or a group of metrics, you need a class that extends the MeterBinder class. This allows to register the created metric in the overriden bindTo(MeterRegistry registry) method.

The following is an example of a metric that has a function that supplies the value for it:

Example metric
public class UserMeterBinder implements MeterBinder {

  private final UserManager userManager;

  @Inject
  public UserMeterBinder(UserManager userManager) {
    this.userManager = userManager;
  }

  @Override
  public void bindTo(MeterRegistry registry) {
    Gauge.builder("che.user.total", this::count)
        .description("Total amount of users")
        .register(registry);
  }

  private double count() {
    try {
      return userManager.getTotalCount();
    } catch (ServerException e) {
      return Double.NaN;
    }
  }

Alternatively, the metric can be stored with a reference and updated manually in some other place in the code.

Additional resources

For more information about the types of metrics and naming conventions, visit Prometheus documentation:

Tags: