# Publishing alerts from Prometheus

This page describes how you can configure [Prometheus](https://prometheus.io/docs/introduction/overview/) to monitor Liberator and publish alerts to Slack, raise alerts when Liberator is down, and publish those alerts to Slack via the [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) alert message router.

## Introduction to Prometheus in Caplin Platform

All Caplin Platform applications export a core set of Prometheus metrics supplemented by metrics specific to the application. See, for example, [Liberator’s Prometheus metrics](../liberator/liberator-prometheus-metrics.md).

### Enabling Prometheus endpoints

To enable the Prometheus endpoint in a Caplin application, set the application’s [`prometheus-port`](datasource-datasource-monitoring-configuration-part-1.md#prometheus-port) configuration item to a free port on the host.

Liberator 8, when deployed to the Caplin Deployment Framework, is configured by default to publish Prometheus metrics on port 8083:

**global_config/overrides/servers/Liberator/etc/rttpd.conf**

```
prometheus-port 8083
```

The Prometheus endpoint is available at the URI `/metrics`:

```
http://<host>:<prometheus-port>**/metrics**
```

### Instrumenting your own Java DataSource adapters

If you write your own Java DataSource adapters, you can supplement the default [Caplin Java DataSource metrics](../datasource/datasource-prometheus-metrics.md#java-datasource) with metrics specific to your application.

Java DataSources use Micrometer for instrumentation. To add a custom metric to an adapter’s Prometheus endpoint, use Micrometer’s [`Metrics`](https://javadoc.io/doc/io.micrometer/micrometer-core/latest/io/micrometer/core/instrument/Metrics.html) class:

**Creating a Prometheus gauge metric**

```java
import io.micrometer.core.instrument.Metrics;

requestCountGauge = Metrics.globalRegistry.gauge(appName + "request.count", new AtomicInteger(0));
```

For more information on creating your own custom metrics, see the following resources:

* [micrometer.io](https://micrometer.io/)
* [micrometer-core JavaDoc](https://javadoc.io/doc/io.micrometer/micrometer-core/latest/index.html)

## Walkthrough

In this walkthrough, we will deploy Liberator, Prometheus, and Alertmanager, and configure Prometheus and Alertmanager to publish alerts to Slack.

### Deploy Liberator 8

Liberator 8 can be deployed in three different ways:

* Deploy to the Caplin Deployment Framework
* Deploy as a standalone server
* Deploy as a Docker image

For the purposes of this walkthrough, we’ll keep things simple and run Liberator in the [Platform Core](../platform-architecture/platform-core-docker-image.md) Docker image, available from the docker-release.caplin.com registry (Caplin account required).

1. Open a new terminal and run the command below to start Liberator in a [Platform Core](../platform-architecture/platform-core-docker-image.md) 8 container:

   ```
   docker run --rm --name liberator --interactive --tty --publish 8083:8083 --publish 18080:18080 docker-release.caplin.com/platform/core:8.0.0 /app/DeploymentFramework/dfw start-fg Liberator
   ```

   The command above runs Liberator in an interactive foreground container. When you want to stop Liberator, press Ctrl-C in the terminal window.
2. To view the metrics exported by Liberator, navigate to http://localhost:8083/metrics. You will see output resembling the sample below:

   ```prometheus
   # HELP rttpd_session_connections_permitted gauge
   # TYPE rttpd_session_connections_permitted gauge
   rttpd_session_connections_permitted 1
   # HELP rttpd_process_fds gauge
   # TYPE rttpd_process_fds gauge
   rttpd_process_fds{type="non_socket_fds"} 27
   rttpd_process_fds{type="socket_fds"} 535
   # HELP rttpd_data_service_messages_read_total counter
   # TYPE rttpd_data_service_messages_read_total counter
   rttpd_data_service_messages_read_total{service_name="broadcast",message_type="nodata"} 0
   rttpd_data_service_messages_read_total{service_name="ContainerTestsService",message_type="map"} 0
   ```

### Install Prometheus

In this section, we’ll install Prometheus and configure an alert when Liberator is down.

Follow the steps below:

1. Download the latest version of Prometheus from the Prometheus [Download](https://prometheus.io/download/) page, and extract it to a local directory:

   **Example directory structure following download and extraction**

   ```plantuml
   @startsalt
   scale 1.25
   {
   {T
    + <color:goldenrod><&folder></color> ~   
    ++ <color:goldenrod><&folder></color> prometheus-2.49.1.linux-amd64
    +++ <color:cornflowerblue><&terminal></color> prometheus
    +++ <color:cornflowerblue><&cog></color> prometheus.yml
   }
   }
   @endsalt
   ```
2. In the Prometheus root directory, edit `prometheus.yml` and add a scrape job for Liberator in the `scrape_configs` sequence:

   **prometheus.yml**

   ```yaml
   scrape_configs:
     - job_name: "liberator"
       scrape_timeout: 30s
       scrape_interval: 40s
       static_configs:
         - targets: ["localhost:8083"]
   ```
3. In the Prometheus root directory, create a `rules.yml` file containing the content below:

   **rules.yml**

   ```yaml
   groups:
     - name: liberator
       rules:
         - alert: InstanceDown
           expr: up == 0
           for: 1m
   ```

   For more information on writing alerting rules, see [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
4. In the Prometheus root directory, edit the `prometheus.yml` file and add `rules.yml` to the `rule_files` sequence:

   ```yaml
   rule_files:
     # - "first_rules.yml"
     # - "second_rules.yml"
     - "rules.yml"
   ```
5. If Liberator is not already running, open a new terminal and start Liberator now:

   ```
   docker run --rm --name liberator --interactive --tty --publish 8083:8083 --publish 18080:18080 docker-release.caplin.com/platform/core:8.0.0 /app/DeploymentFramework/dfw start-fg Liberator
   ```
6. In the Prometheus root directory, start Prometheus:

   ```
   ./prometheus
   ```
7. In a web browser, navigate to http://localhost:9090 to confirm that Prometheus is running:

   ![prometheus_home](../../images/prometheus_home.png)
8. On the Prometheus home page, click **Status > Targets** to confirm that Prometheus has detected that the Liberator is 'up':

   ![liberator_prometheus_up](../../images/liberator_prometheus_up.png)
9. In the Liberator terminal window, press Ctrl-C to stop the Liberator.
10. In Prometheus, click **Alerts**. Notice that Prometheus has fired an alert for Liberator’s status change:

   ![liberator_prometheus_down](../../images/liberator_prometheus_down.png)

### Install Alertmanager and route alerts to Slack

In the previous section we configured an alerting rule for a Liberator status change. At this point, the alert is visible only in the Prometheus GUI. To publish the alert to Slack, we use Prometheus’s sister application: Alertmanager.

Follow the steps below:

1. Download Alertmanager from the Prometheus [Download](https://prometheus.io/download/) page and extract to a local directory:

   **Example directory structure following download and extraction**

   ```plantuml
   @startsalt
   scale 1.25
   {
   {T
    + <color:goldenrod><&folder></color> ~
    ++ <color:goldenrod><&folder></color> **alertmanager-0.26.0-linux-amd64**
    +++ <color:cornflowerblue><&terminal></color> **alertmanager**
    +++ <color:cornflowerblue><&cog></color> **alertmanager.yml**
    ++ <color:goldenrod><&folder></color> prometheus-2.49.1.linux-amd64
    +++ <color:cornflowerblue><&terminal></color> prometheus
    +++ <color:cornflowerblue><&cog></color> prometheus.yml
   }
   }
   @endsalt
   ```

1. In Slack’s **Manage Apps** settings, create a webhook URL for the Slack channel you want to post alerts to.
2. Add the following configuration to Alertmanager’s `alertmanager.yml` configuration file:

   **alertmanager.yml**

   ```yaml
   global:
     resolve_timeout: 1m
     slack_api_url: '<your_webhook_url>'

   route:
     receiver: 'slack-notifications'

   receivers:
     - name: 'slack-notifications'
       slack_configs:
         - channel: '#<your_slack_channel>'
           send_resolved: true
   ```
3. Add the configuration below to your `prometheus.yml` file to provide Prometheus with the location of your Alertmanager. By default, Alertmanager runs on port 9093.

   **prometheus.yml**

   ```yaml
   # Alertmanager configuration
   alerting:
     alertmanagers:
       - static_configs:
         - targets:
           - localhost:9093
   ```
4. If Liberator is not already running, start Liberator. Navigate to http://localhost:18080 to confirm Liberator has started.

   ```
   docker run --rm --name liberator --interactive --tty --publish 8083:8083 --publish 18080:18080 docker-release.caplin.com/platform/core:8.0.0 /app/DeploymentFramework/dfw start-fg Liberator
   ```
5. Start Alertmanager.  Navigate to localhost:9093 to confirm Alertmanager has started.

   ```
   ./alertmanager
   ```
6. Start Prometheus.  Navigate to localhost:9090 to confirm Prometheus has started.

   ```
   ./prometheus
   ```
7. In the Liberator terminal window, press Ctrl-C to stop the Liberator.

   Alertmanager sends a message to your configured Slack channel. For example:

   ![prometheus_alertmanager_basic_alerts](../../images/prometheus_alertmanager_basic_alerts.png)

More informative and interactive alerts can be created by using the Alertmanager’s templating syntax to produce alerts similar to the example below:

![alertmanager_styled_down](../../images/alertmanager_styled_down.png)

For a detailed reference on configuring Alertmanager, including how to customise the presentation of alert notifications, see [Alerting](https://prometheus.io/docs/alerting/latest/overview/) in the Prometheus documentation.