# Configure DataSource Prometheus monitoring

This page describes how to publish metrics in [Prometheus](https://prometheus.io) format.

## Publishing DataSource metrics in Prometheus format 

Version requirements:

* DataSource for C 7.1.16+ (Liberator 7.1.15+, Transformer 7.1.10+, TREP Adapter 7.1.3+)
* DataSource for Java 7.1.14+

DataSource metrics are disabled by default. To publish DataSource metrics, follow the steps below:

1. Set [`prometheus-port`](datasource-datasource-monitoring-configuration-part-1.md#prometheus-port) to a free port on the DataSource’s host:

   ```
   prometheus-port 48557
   ```
2. Restart the DataSource
3. Open a web browser and navigate to <span>http://</span>&lt;datasource_host>:&lt;prometheus_port>/metrics. You will see output similar to the excerpt below:

   **Example Liberator DataSource metrics in Prometheus format**

   ```
   # HELP rttpd_user_global_login_fail counter
   # TYPE rttpd_user_global_login_fail counter
   rttpd_user_global_login_fail 0

   # HELP rttpd_session_global_discards gauge
   # TYPE rttpd_session_global_discards gauge
   rttpd_session_global_discards 0
   ```

For the full list of published metrics, follow the links below:

* [DataSource for C Prometheus metrics](datasource-prometheus-metrics.md#datasource-for-c-dsdk-metrics)
* [DataSource for Java Prometheus metrics](datasource-prometheus-metrics.md#datasource-for-java-metrics)
* [Liberator Prometheus metrics](../liberator/liberator-prometheus-metrics.md)
* [Transformer Prometheus metrics](../transformer/transformer-prometheus-metrics.md)

### Publishing custom metrics

The Java DataSource API uses [Micrometer](https://micrometer.io/) to publish Prometheus metrics. You can supplement the standard DataSource metrics with your own custom metrics by adding custom [Meter](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.0/io/micrometer/core/instrument/Meter.html) objects to Micrometer’s global meter registry.

The Micrometer [`Metrics`](https://javadoc.io/doc/io.micrometer/micrometer-core/latest/io/micrometer/core/instrument/Metrics.html) class provides static methods to create [`Counter`](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.0/io/micrometer/core/instrument/Counter.html) and [`Gauge`](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.0/io/micrometer/core/instrument/Gauge.html) meters in the global meter registry. To create more advanced meters, access the global meter registry object directly via the static reference [`Metrics.globalRegistry`](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.0/io/micrometer/core/instrument/Metrics.html#globalRegistry).

**Adding a simple counter to a DataSource application’s metrics**

To add a [`Counter`](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.0/io/micrometer/core/instrument/Counter.html) to the global meter registry, use the [`Metrics.counter`](https://javadoc.io/static/io.micrometer/micrometer-core/1.11.2/io/micrometer/core/instrument/Metrics.html#counter(java.lang.String,java.lang.Iterable)) builder:

```java
Counter myEventsCounter = Metrics.counter("my_events", Tags.empty());
```

The metrics published on the DataSource’s Prometheus interface (<span>http://</span>&lt;datasource_host>:&lt;prometheus_port>/metrics) now include a `my_events_total` metric:

```
# HELP my_events_total  
# TYPE my_events_total counter
my_events_total 0.0
```

For more information on Prometheus and Micrometer, follow the links below:

* [Prometheus metric types](https://prometheus.io/docs/tutorials/understanding_metric_types/)
* [Micrometer Documentation](https://micrometer.io/docs)

## Publishing JVM metrics in Prometheus format

DataSource metrics do not include metrics on the JVM associated with a DataSource. To publish JVM metrics, we recommend that you configure the DataSource’s JVM to load the Prometheus project’s [JMX Exporter](https://github.com/prometheus/jmx_exporter) as a JVM Java agent.

### Configuring a C DataSource to run the Prometheus JMX Exporter

This section describes how to configure a C DataSource’s embedded JVM to load the Prometheus JMX Exporter as a JVM Java agent.

The instructions in this section use Liberator as an example and assume that you have already deployed or activated a blade that enables Liberator’s embedded JVM. For example, the Liberator’s LiberatorJMX blade or Caplin’s PermissioningService module.

Follow the steps below:

1. Download the latest [Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) Java Agent Jar file to your Deployment Framework’s `tools` directory.
2. In `global_config/environment.conf`, define a convenience variable for the location of the JMX Exporter Jar file:

   **File: global_config/environment.conf**

   ```
   *define* JMX_EXPORTER_JAR [navy]##${BASE}##/tools/jmx_prometheus_javaagent-0.17.2.jar
   ```

   The latest version of JMX Exporter at the time of writing is 0.17.2.
3. Add the following line of configuration to Liberator’s `java.conf` override file:

   **File: global_config/overrides/servers/Liberator/etc/java.conf**

   ```
   *jvm-options* -javaagent:[navy]##${JMX_EXPORTER_JAR}##=__jmx_exporter_port__:[navy]##${ccd}##/jmx-exporter.yaml
   ```

   where __jmx_exporter_port__ is a free port on the host. If you are running multiple DataSources on the host, then assign each JMX Exporter a unique port.
4. Create a configuration file for the JMX Exporter in the overrides directory for Liberator:

   **File: global_config/overrides/servers/Liberator/etc/jmx-exporter.yaml**

   ```yaml
   ---
   rules:
   - pattern: ".*"
   ```
5. (Re)start Liberator

   ```
   $ ./dfw start
   ```
6. Open a web browser and navigate to http://__liberator_host__:__jmx_exporter_port__. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

   ```
   # HELP jvm_threads_current Current thread count of a JVM
   # TYPE jvm_threads_current gauge
   jvm_threads_current 22.0
   # HELP jvm_threads_daemon Daemon thread count of a JVM
   # TYPE jvm_threads_daemon gauge
   jvm_threads_daemon 19.0
   # HELP jvm_threads_peak Peak thread count of a JVM
   # TYPE jvm_threads_peak gauge
   jvm_threads_peak 28.0
   ```

### Configuring a Java DataSource to run the Prometheus JMX Exporter

This section describes how to configure a Java DataSource’s JVM to load the Prometheus JMX Exporter as a JVM Java agent.

Follow the steps below:

1. Download the latest [Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) Java Agent jar file to your Deployment Framework’s `tools` directory.
2. Make the following modifications (highlighted) to the Java DataSource’s startup script: `kits/__bladename__/__bladename__-__version__/DataSource/bin/start-jar.sh`. The example script below is adapted from Caplin’s Pricing Adapter Template (see [start-jar.sh](https://github.com/caplin/project-templates/blob/master/pricing-template/blade/DataSource/bin/start-jar.sh)):

   ```bash
   #!/bin/bash

   BLADENAME=MyPricingAdapter

   # Prometheus JMX Exporter jar location, port, and config file location
   PROM_AGENT=${CONFIG_BASE}../tools/jmx_prometheus_javaagent-0.17.2.jar ①
   PROM_AGENT_PORT=5580 ②
   PROM_AGENT_CONFIG=${CONFIG_BASE}overrides/${BLADENAME}/etc/jmx-exporter.yaml

   if [ "$1" = "CONFREADER" ]; then
      shift
      confreading=1
      jar=`ls "$BINARY_ROOT"/lib/datasource*.jar|head -1`
   else
      confreading=0
      jar=`ls "$BINARY_ROOT"/lib/$BLADENAME*.jar|head -1`
      classpath="${BINARY_ROOT}/lib/*"

      echo "Classpath: $jar"
   fi

   if [ $confreading = 1 ]; then
      java -jar "$jar" "$@"
      exit $?
   else
      java \
       -javaagent:${PROM_AGENT}=${PROM_AGENT_PORT}:${PROM_AGENT_CONFIG} \
       -cp "$classpath" \
       -jar "$jar" "$@" 2> "$LOGDIR"/java-$BLADENAME.log >/dev/null &
      echo $!
   fi
   ```
   1. The latest version of JMX Exporter at the time of writing is 0.17.2
   2. Set `PROM_AGENT_PORT` to a free port on the host. Each JMX Exporter running on the host requires its own unique port.
3. Create a new configuration file for the JMX Exporter in the overrides directory for the adapter:

   **File: global_config/overrides/&lt;bladename>/etc/jmx-exporter.yaml**

   ```yaml
   ---
   rules:
   - pattern: ".*"
   ```
4. (Re)start the Java DataSource:

   ```
   $ ./dfw start
   ```
5. Open a web browser and navigate to <span>http://</span>&lt;liberator-host>:5580/. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

   ```
   # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
   # TYPE jvm_gc_collection_seconds summary
   jvm_gc_collection_seconds_count{gc="PS Scavenge",} 1.0
   jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.005
   jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 1.0
   jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.013
   # HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
   # TYPE jvm_classes_loaded gauge
   jvm_classes_loaded 2533.0
   ```