Metrics collection with Prometheus#
We collect operational metrics about all the components of mybinder.org and create dashboards from them. This document details the components involved in collecting, storing and querying the metrics.
This is only for operational metrics - not for analytics on repositories built or traffic.
Metrics Storage + Querying#
We use Prometheus to store and query our metrics.
What is Prometheus?#
Prometheus is a Time Series Database optimized for storing operational metrics. It stores all data as streams of timestamped values belonging to the same metric and the same set of labels.
The metric name specifies the general feature of a system that is
http_requests_total - the total number of HTTP requests received).
A set of labels for the same metric name identifies a particular
dimensional instantiation of that metric (for example: all HTTP requests
that used the method
POST to the
/api/tracks handler would be represented
as the time series
Prometheus has its own query language called PromQL, optimized for time series queries.
The prometheus documentation has fairly clear and thorough documentation on PromQL - basics, operators and functions. You do not need to become an expert, but a basic understanding is useful. There are also examples to pick up and play with!
prometheus.mybinder.org is our public prometheus installation, and you can practice your queries there!
Prometheus uses a pull model for metrics. It has a list of targets, and constantly polls them for their current state, and records what it gets back. The targets are supposed to respond to these HTTP requests with data in the prometheus format.
Our data is currently sourced from the following targets.
The node_exporter exports
information about each node we run - CPU usage, memory left, disk space,
etc. It provides fairly detailed info, usually prefixed with
This is not kubernetes specific.
exposes information about the kubernetes cluster - such as number of pods
and the states they are in, number of nodes, etc. These are usually
These only contain information from kubernetes API itself. For example,
‘how much RAM are these containers using’ is not recorded by
since that is not information that is available to the Kubernetes API.
‘how much RAM are these pods requesting’ is, however, available.
cadvisor provides detailed runtime
information about all the containers running in the cluster. This is information
mostly not available from
kube-state-metrics - such as ‘how much RAM are
these containers using right now’, etc. These are usually prefixed with
HTTP request information#
We use the nginx-ingress helm chart
to let all HTTP traffic into our cluster. This allows us to use
the nginx VTS exporter
to collect information in prometheus about requests / responses.
These metrics are prefixed with
Prometheus is installed using the prometheus helm chart. This installs the following components:
Prometheus server (storage + querying)
node_exporteron every node
cadvisor is already present on all nodes (it ships with the
kubernetes component), and the prometheus helm chart has configuration
that adds those as targets.
You can see the available options for configuring the prometheus
helm chart in its values.yaml
file. You can see the current configuration we have under the