devkuma – Grafana

Grafana - Open-source data monitoring tool

kc@example.com (kc kim) — Mon, 05 Sep 2022 16:32:00 +0900

What is Grafana?

Grafana is an open-source data visualization and monitoring tool. It visualizes data from Prometheus and many other data sources. It is developed by Grafana Labs.

Differences from Kibana

Kibana is mainly used to analyze log messages.
Grafana is specialized for visualizing system metrics such as CPU, memory, and disk usage.
Kibana is tied to Elasticsearch, while Grafana can work with many different databases.
Grafana provides alerting features for free.

Installing Prometheus + Grafana with Docker Compose

kc@example.com (kc kim) — Sun, 04 Dec 2022 06:49:14 +0900

Overview

This article explains how to install and run Prometheus and Grafana with Docker Compose.

Docker Configuration

Prometheus

Let’s configure the files and directories needed to install Prometheus with Docker.

First, create an installation directory in an appropriate location.

mkdir prometheus-grafana
cd prometheus-grafana

Create a docker-compose.yml file to run Docker. prometheus-grafana/docker-compose.yml

version: '3.7'  # File format version
services:       # Define the containers to run under this item
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/config:/etc/prometheus
      - ./prometheus/volume:/prometheus
    ports:
      - 9090:9090 # Access port setting (outside container:inside container)
    command: # web.enable-lifecycle allows config files to be reloaded without restarting the API
      - '--web.enable-lifecycle'
      - '--config.file=/etc/prometheus/prometheus.yml'
    restart: always
    networks:
      - promnet

networks:
  promnet:
    driver: bridge

Prometheus DockerHub image
- https://hub.docker.com/r/prom/prometheus
The Prometheus storage directory is set to ./prometheus/volume.
The Prometheus configuration directory is set to ./prometheus/config.

Create the directory where Prometheus configuration files will be placed.

mkdir prometheus
mkdir prometheus/config

Create configuration files (prometheus.yml, rule.yml) inside the /prometheus/config directory.

Reference: Prometheus Configuration

prometheus-grafana/prometheus/config/prometheus.yml

global:
  scrape_interval: 15s     # Change the default interval for scrape targets to 15 seconds / default = 1m
  scrape_timeout: 15s      # Length before a scrape request times out / default = 10s
  evaluation_interval: 2m  # How often rules are evaluated / default = 1m

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'       # Label to attach by default
  query_log_file: query_log_file.log # Records Prometheus query logs; if omitted, logs are not recorded

# Loads rules and evaluates them periodically according to the evaluation_interval setting.
rule_files:
  - "rule.yml"  # The file location is the same as prometheus.yml
  - "rule2.yml" # Multiple files are possible

# Endpoint for collecting metrics. Here it points to the Prometheus server itself.
scrape_configs:
  # Sets the job name as `job=<job_name>` on time series collected by this setting.
  # The default metrics_path is '/metrics', and the default scheme is `http`.
  - job_name: 'monitoring-item' # job_name must be unique across all scrapes
    scrape_interval: 10s      # Can be omitted because the default value is defined globally
    scrape_timeout: 10s       # Can be omitted because the default value is defined globally
    metrics_path: '/asdf'     # Optional - changes the URI Prometheus references to get metrics | default = /metrics
    honor_labels: false       # Optional - whether to change labels if label conflicts occur | default = false
    honor_timestamps: false   # Optional - if honor_labels is true, metrics timestamps are exposed | default = false
    scheme: 'http'            # Optional - scheme used to send requests | default = http
    params:                   # Optional - params sent with requests
      user-id: ['myemail@email.com']

    # Additional authorization settings
    # service discovery settings (sd)

    # Settings for actual scrape targets
    static_configs:
      - targets: ['localhost:9090', 'localhost:9100', 'localhost:80'] ## prometheus, node-exporter, cadvisor  
        labels: # Optional - labels to attach to all scraped metrics
          service : 'monitor-1'
    
    # relabel_config - modifies labels before scraping
    # metric_relabel_configs - dynamically rewrites labels of collected targets (drop, replace, labeldrop)

prometheus-grafana/prometheus/config/rule.yml

groups:
- name: example # Must be unique within the file
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

  # Alert for any instance that has a median request latency >1s.
  - alert: APIHighRequestLatency
    expr: api_http_request_latencies_second{quantile="0.5"} > 1
    for: 10m
    annotations:
      summary: "High request latency on {{ $labels.instance }}"
      description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"

Change the permissions of the /prometheus directory so Docker can modify it.

sudo chmod -R 777 ./prometheus

After creating all final files, the structure is as follows.

.
├── docker-compose.yml
└── prometheus
    └── config
        ├── prometheus.yml
        └── rule.yml

Grafana

Grafana is an open-source platform for data visualization, monitoring, and analysis.

Add the Grafana settings under services: in docker-compose.yml.

  grafana:
    image: grafana/grafana
    container_name: grafana
    # user: "$GRA_UID:$GRA_GID"
    ports:
      - 3000:3000 # Access port setting (outside container:inside container)
    volumes:
      - ./grafana/volume:/var/lib/grafana
    restart: always
    networks:
      - promnet

Grafana DockerHub Image
- https://hub.docker.com/r/grafana/grafana
Here, the Grafana storage directory is set to ./grafana/volume.

Running Docker

Run the command that creates Docker images and starts containers all at once.

docker compose up -d

To stop the containers, run the following command.

docker compose stop

To restart the containers, run the following command.

docker compose start

After running it, you can see that the following volume directories are created.

.
├── docker-compose.yml
├── grafana
│   └── volume
│       ├── grafana.db
│       ├── grafana.db-journal
│       └── plugins
└── prometheus
    ├── config
    │   ├── prometheus.yml
    │   ├── query_log_file.log
    │   └── rule.yml
    └── volume
        └── data
            ├── chunks_head
            ├── lock
            ├── queries.active
            └── wal
                └── 00000000

Access

Prometheus
- http://localhost:9090
Grafana
- http://localhost:3000
  - Default account ID/PW: admin/admin

References

The configuration files have been uploaded to GitHub.

Spring Boot default metrics

kc@example.com (kc kim) — Sat, 10 Dec 2022 04:41:00 +0900

https://tomgregory.com/spring-boot-default-metrics/

This page looks at the most important default metrics provided by Spring Boot and how to use them to highlight problems in an application more effectively.

Overview of Spring Boot Actuator and Micrometer

Spring Boot Actuator provides various monitoring and management endpoints through HTTP and JMX. By integrating with the Micrometer application monitoring framework, it includes very important metrics features.

The first page of the Micrometer site says:

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.

Micrometer is a vendor-independent metrics facade. In other words, metrics are collected in one common way and exposed in the format required by various monitoring systems. If SLF4J exists for logging systems, Micrometer exists for monitoring systems.

Popular supported monitoring frameworks include Graphite, Prometheus, and StatsD. Here, the focus is Prometheus, a standalone service that periodically fetches metrics from an application.

Add Actuator metrics to a Spring Boot application

To configure a Spring Boot application to publish metrics in Prometheus format, follow these steps.

Include additional dependencies

To include the Spring Boot Starter Actuator module, add the following to the Gradle dependency list.

implementation 'org.springframework.boot:spring-boot-starter-actuator'

The Maven equivalent is:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Then add the Micrometer Registry Prometheus module, so metrics can be scraped from the application in Prometheus format.

Gradle:

implementation 'io.micrometer:micrometer-registry-prometheus:1.5.1'

Maven:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.5.1</version>
</dependency>

Enable metrics in configuration

By default, almost all Spring Boot Actuator HTTP endpoints are disabled, but they can be enabled through configuration. Add the following setting to the application.properties file in the project’s src/main/resources folder.

management.endpoints.web.exposure.include=metrics,prometheus

The Actuator module has many endpoints that can be enabled, but this configuration enables only the following two.

/actuator/metrics: an endpoint that provides a JSON API for exploring metrics and viewing their values.
/actuator/prometheus: an endpoint that returns metrics in a custom format required for collection by Prometheus.
- This is the part that will be explained continuously below.

1. Spring MVC metrics

For any web application, the default Spring MVC metrics provide an excellent starting point for monitoring inbound HTTP traffic. Whether you need to track errors, traffic volume, or request latency, these metrics help.

Inbound HTTP request duration

For each endpoint exposed by a Spring Boot application, the http_server_requests_seconds summary metric provides information about request count and request duration. It consists of two metrics exposed by the /actuator/prometheus endpoint.

# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/hello",} 12.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/hello",} 0.083579374

http_server_requests_seconds_count is the total number of requests received by the application at that endpoint.
http_server_requests_seconds_sum is the sum of the durations of all requests received by the application at that endpoint.

These metrics include the following tags.

Tag	Description	Example
`exception`	Class name of the exception that occurred	None, NullPointerException
`method`	HTTP request method	GET, POST, PUT, PATCH, etc.
`outcome`	String description of the HTTP response status	SUCCESS, SERVER_ERROR
`status`	HTTP response status code	200, 500, etc.
`uri`	Request URI	`/hello`

What is a tag?

A tag is useful for measuring variations in a metric. For example, with `http_server_requests_seconds_count`, you can measure the number of requests for a specific URI. If the application starts returning a different error response status, the metric is separated into a metric with a different status tag. Those metrics can be queried together or separately.

In Prometheus, you can create a simple query that provides the average inbound request duration across all tags.

rate(http_server_requests_seconds_sum[1m]) / rate(http_server_requests_seconds_count[1m])

Why use the rate function?

You may wonder why the rate function is included in the Prometheus query above. Why not just divide the request count by the total request duration? The [Micrometer website](https://micrometer.io/docs/registry/prometheus#_timers) explains: "Representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service." "Since the representation is a function of both the speed at which the counter increases and the lifetime of the service, representing a counter without rate normalization over a period of time is rarely useful."

Inbound HTTP request quantiles and percentiles

Spring MVC metrics can also calculate quantiles and percentiles, which can be useful when evaluating how slow API request durations are while ignoring the slowest requests.

For example, the 95th percentile is the value below which 95% of observed values fall and above which 5% fall. In other words, it provides the slowest request duration seen by 95% of requests.

To enable quantiles, add an additional configuration property to application.properties, replacing it with the quantiles of interest, which are not percentiles despite the name.

management.metrics.web.server.request.autotime.percentiles=<comma-separated list of quantiles>

If the property is configured as 0.95, the following metric is generated at /actuator/prometheus.

http_server_requests_seconds{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/hello",quantile="0.95",} 0.023068672

The tags used with this metric are the same as above, but now there is also a quantile tag that can be used to query the metric.

The query http_server_requests_seconds{quantile="0.95"} generates a graph in Prometheus.

Maximum inbound HTTP request duration

For each endpoint exposed by a Spring Boot application, the http_server_requests_seconds_max gauge metric provides the maximum duration for each inbound HTTP request type.

# HELP http_server_requests_seconds_max Duration of HTTP server request handling
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/hello",} 0.005831

http_server_requests_seconds_max is the maximum request duration during a period. When a new time window starts, the value is reset to 0. The default time window is two minutes.

The tags used with this metric are the same as above.

This graph is generated in Prometheus from a regular http_server_requests_seconds_max query.

All metrics described in this document can be started and run immediately using the accompanying GitHub repository. It includes a Spring Boot application that generates metrics and a Prometheus instance where queries can be executed.

2. HTTP client RestTemplate and WebClient outbound request metrics

If you perform outbound HTTP requests using the RestTemplate or WebClient classes, request metrics similar to inbound HTTP requests can be used. For these metrics to work, the HTTP client must be created using the injected RestTemplateBuilder or WebClient.Builder class.

Outbound HTTP request duration

Each outbound endpoint gets the http_client_requests_seconds metric. It consists of two metrics exposed by the /actuator/prometheus endpoint.

# HELP http_client_requests_seconds Timer of RestTemplate operation
# TYPE http_client_requests_seconds summary
http_client_requests_seconds_count{clientName="google.com",method="GET",outcome="SUCCESS",status="200",uri="/https://google.com",} 3.0
http_client_requests_seconds_sum{clientName="google.com",method="GET",outcome="SUCCESS",status="200",uri="/https://google.com",} 0.465022459

http_client_requests_seconds_count is the total number of requests the application made to this endpoint.
http_client_requests_seconds_sum is the sum of the durations of all requests the application made to this endpoint.

These metrics also include the following tags.

Tag	Description
`clientName`	Name of the endpoint being called, using the host of the URI
`exception`	Class name of the exception that occurred
`method`	HTTP request method
`outcome`	String description of the HTTP response status
`status`	HTTP response status code
`uri`	Request URI

In Prometheus, you can create a simple query that provides average outbound request duration over time.

rate(http_client_requests_seconds_sum[1m]) / rate(http_client_requests_seconds_count[1m])

Maximum outbound HTTP request duration

Each outbound endpoint gets the http_client_requests_seconds_max gauge metric, which provides the maximum duration of each outbound HTTP request type.

# HELP http_client_requests_seconds_max Timer of RestTemplate operation
# TYPE http_client_requests_seconds_max gauge
http_client_requests_seconds_max{clientName="google.com",method="GET",outcome="SUCCESS",status="200",uri="/https://google.com",} 0.205564498

http_client_requests_seconds_max is the maximum request duration during a period. When a new time window starts, the value is reset to 0. The default time window is two minutes.

The tags used with this metric are the same as in “Outbound HTTP request duration” above.

3. JVM metrics

Micrometer includes three types of metrics that help monitor what is happening in the JVM (Java Virtual Machine).

JVM memory metrics

For each memory area, you can check the amount of memory used with jvm_memory_used_bytes and the available memory with jvm_memory_max_bytes.

# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",} 8388608.0
jvm_memory_used_bytes{area="heap",id="G1 Old Gen",} 3938936.0
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 4.2306152E7

...

# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",id="G1 Survivor Space",} -1.0
jvm_memory_max_bytes{area="heap",id="G1 Old Gen",} 8.589934592E9

There is a lot of data here, but for example, you can display the total amount of used heap memory in Prometheus with the following query.

sum(jvm_memory_used_bytes{area="heap"})

This uses the Prometheus sum function to add the memory used in all heap memory areas visible in the id tag above, such as G1 Survivor Space, G1 Old Gen, and G1 Eden Space.

JVM garbage collection metrics

There are many garbage collection metrics available to gain deep insight into how the JVM manages memory. They can largely be divided into the following areas.

Pause time

The jvm_gc_pause_seconds and jvm_gc_pause_seconds_max metrics provide information about the time spent in garbage collection.

# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Metadata GC Threshold",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Metadata GC Threshold",} 0.005
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Metadata GC Threshold",} 0.0

Memory pool size increase

The jvm_gc_memory_allocated_bytes_total metric tells you about increases in the size of the young-generation memory pool, while jvm_gc_memory_promoted_bytes_total is for the old generation.

# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total 2.66338304E8
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total 1.4841448E7

Live old-generation pool size

The jvm_gc_live_data_size_bytes metric tells you the old-generation pool size. jvm_gc_max_data_size_bytes tells you the maximum size that can be allocated to the old-generation pool.

# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 9039328.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes 5.22190848E8

JVM thread metrics

These metrics let you check the threads in the JVM.

# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{state="runnable",} 7.0
jvm_threads_states_threads{state="blocked",} 0.0
jvm_threads_states_threads{state="waiting",} 11.0
jvm_threads_states_threads{state="timed-waiting",} 3.0
jvm_threads_states_threads{state="new",} 0.0
jvm_threads_states_threads{state="terminated",} 0.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 21.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 17.0
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 23.0

jvm_threads_states_threads shows the number of threads in each thread state.
jvm_threads_live_threads shows the total number of live threads, including daemon and non-daemon threads.
jvm_threads_daemon_threads shows the total number of daemon threads.
jvm_threads_peak_threads shows the maximum total number of threads since the JVM started.

Daemon thread

A daemon thread is a low-priority thread that performs background tasks such as garbage collection.

If you run jvm_threads_states_threads in Prometheus, you can see all thread states in the graph.

Quick Guide to Application Monitoring with Spring Boot + Prometheus + Grafana + Docker

kc@example.com (kc kim) — Sat, 10 Dec 2022 04:41:00 +0900

Overview

This guide configures Spring Boot Actuator to expose Metrics, sends those metrics to Prometheus, and visualizes them in Grafana.

Create a Spring Boot Project

First, create a Spring Boot project that will expose metrics for Prometheus.

Command to create a new project

Use the curl command below to create a new Spring Boot project.

curl https://start.spring.io/starter.tgz \
-d bootVersion=2.7.6 \
-d dependencies=web,actuator,prometheus \
-d baseDir=spring-actuator-prometheus \
-d groupId=com.devkuma \
-d artifactId=spring-actuator-prometheus \
-d packageName=com.devkuma.prometheus \
-d applicationName=ActuatorPrometheusApplication \
-d javaVersion=11 \
-d packaging=jar \
-d type=gradle-project | tar -xzvf -

This command uses Java 11 and Spring Boot 2.7.6, and adds the web, actuator, and prometheus dependencies.

Check dependencies in the build file

build.gradle

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-actuator'
	implementation 'org.springframework.boot:spring-boot-starter-web'
	runtimeOnly 'io.micrometer:micrometer-registry-prometheus'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
}

spring-boot-starter-actuator provides access to Spring Boot application information.
- Since Spring Boot 2.0, Actuator includes Micrometer support for measuring application metrics.
micrometer-registry-prometheus provides metrics that Prometheus can read.
- This library converts metrics produced by Micrometer into a metric format that a Prometheus server can use.

Configure the Actuator endpoint

In Spring Web, Actuator provides an additional endpoint so that a Prometheus server can pull application metric information generated through Micrometer.

Actuator can configure the list of endpoints exposed under /actuator through the management.endpoints.web.exposure.include option in application.yml.

/src/main/resources/application.yml

management:
  endpoints:
    web:
      exposure:
        include: health, prometheus

Here, the default health endpoint and the Prometheus endpoint are exposed.

Check the endpoint list provided by the application

Run the application and open http://localhost:8080/actuator to check the list of endpoints provided by Actuator.

{
    "_links": {
        "self": {
            "href": "http://localhost:8080/actuator",
            "templated": false
        },
        "health-path": {
            "href": "http://localhost:8080/actuator/health/{*path}",
            "templated": true
        },
        "health": {
            "href": "http://localhost:8080/actuator/health",
            "templated": false
        },
        "prometheus": {
            "href": "http://localhost:8080/actuator/prometheus",
            "templated": false
        }
    }
}

At http://localhost:8080/actuator/prometheus, you can check the metrics collected through Micrometer.

# HELP tomcat_sessions_active_current_sessions  
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions 0.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Metadata GC Threshold",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Metadata GC Threshold",} 0.006
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Metadata GC Threshold",} 0.006

... omitted ...

Each entry is displayed as a metric that Prometheus can recognize, such as tomcat_sessions_active_current_sessions and jvm_gc_pause_seconds_count, and you can also check the HELP and TYPE information above each metric.

Prometheus Server

Now that the application is ready, prepare a Prometheus server to collect the metrics generated by the application.

This guide uses Prometheus installed in Docker. For Prometheus Docker installation, refer to this page.

Configure the Prometheus server

When starting the Prometheus server, configure the settings file (prometheus.yml) as follows.

/etc/prometheus/prometheus.yml

global:
  scrape_interval: 10s # Pull metrics every 10 seconds
  evaluation_interval: 10s # Configure how often rules are evaluated
scrape_configs:
  - job_name: 'spring-actuator-prometheus'
    metrics_path: '/actuator/prometheus' # Application prometheus endpoint
    static_configs:
      - targets: ['host.docker.internal:8080'] # Application host:port

Warn: Application Host

In this configuration file, the Application Host is set to `host.docker.internal`. This host can be used in Docker environments on Mac or Windows. On Linux, enter `localhost` instead. **Reference**: [Collect Docker metrics with Prometheus](https://docs.docker.com/config/daemon/prometheus/)

Check the Prometheus server

Now start the Prometheus Docker container and connect to Prometheus.

If everything above is correct, open Prometheus (http://localhost:9090) and you will see the Prometheus main screen as shown below.

In the Status > Configuration menu, you can check whether the file configured for Prometheus has been applied.

In the Status > Targets menu, you can check the status of the connected application.

Finally, return to the main screen, enter jvm_memory_used_bytes in Expression, execute it, and select Graph to see a chart like the one below.

Grafana Server

Now let’s look at how to visualize the metrics collected by Prometheus in Grafana.

You can run queries on the Prometheus web page and visualize the desired metrics as graphs. However, manually running queries every time for monitoring is inefficient, and the built-in dashboard is limited to simple graph viewing.
Because Prometheus alone has limitations for visualization, metrics are usually monitored with a separate visualization tool.

If you followed this page, Grafana should also already be installed together with Prometheus.

First, open Grafana (http://localhost:9090) and you will see the Grafana login screen as shown below.
You can log in with the default account information (ID/password), admin/admin.

Add a Grafana data source

After logging in, the first screen below appears. Click the icon under DATA SOURCES.

Click Prometheus in the list of data sources to add.

When the screen for adding a Prometheus data source appears, enter Name and URL.
For Name, enter any name you want, and for URL, enter http://host.docker.internal:9090.

Scroll down and click the Save & test button. A success message, Data source is working, will be displayed.

Click the Configuration icon to confirm the data source you just registered.

Add a Grafana metric graph chart

Next, click the Explore icon, select jvm_memory_max_bytes for Metric, and click the Run query button. Then you will see a graph chart like the one below.

Click the Add to dashboard button, and the Add panel to dashboard screen appears.
Select New dashboard, then click the Open dashboard button.

On the Save dashboard screen, enter a Dashboard name and click Save to save it.

Now click the Dashboard icon on the left and select the JVM Memory used item with the name you entered earlier.

Then you can see the chart as shown below.

You can drag this chart sideways to resize it and save the layout.

References

The complete project code is available here.
Micrometer Application Monitoring | concepts
Building a monitoring system for a Spring Boot application

Application Monitoring with Spring Batch + Prometheus + Pushgateway + Grafana + Docker

kc@example.com (kc kim) — Tue, 27 Dec 2022 13:53:49 +0900

Overview

Prometheus basically collects metrics by periodically requesting, or pulling, them from servers that provide metric indicators.

However, there are cases like Spring Batch, where a CLI-style process is simply run periodically. In these cases, there is no separate IP, so metrics must be pushed back to Prometheus rather than collected periodically by polling. Pushgateway is what supports pushing metrics.

Pushgateway

Pushgateway, provided by Prometheus, supports pushing metrics and acts as an intermediary so that Prometheus can pull the pushed metrics. With this structure, Prometheus can retrieve metrics pushed to Pushgateway.

Create a Spring Batch project

Command to create a new project

Create a new Spring Boot project with the curl command as follows.

curl https://start.spring.io/starter.tgz \
-d bootVersion=2.7.6 \
-d dependencies=batch,h2 \
-d baseDir=spring-batch-prometheus \
-d groupId=com.devkuma \
-d artifactId=spring-batch-prometheus \
-d packageName=com.devkuma.batch.prometheus \
-d applicationName=BatchPrometheusApplication \
-d javaVersion=11 \
-d packaging=jar \
-d type=gradle-project | tar -xzvf -

Running the command above adds Spring Batch and H2 Database.

Spring Batch configuration

/src/main/java/com/devkuma/batch/prometheus/BatchPrometheusApplication.java

package com.devkuma.batch.prometheus;

import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;

@EnableScheduling
@SpringBootApplication
@EnableBatchProcessing
public class BatchPrometheusApplication {

	public static void main(String[] args) {
		SpringApplication.run(BatchPrometheusApplication.class, args);
	}
}

Add the @EnableBatchProcessing annotation to enable batch features, and add @EnableScheduling to enable scheduler features.

/build.gradle

// ... omitted ...

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-batch'
	runtimeOnly 'com.h2database:h2'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.batch:spring-batch-test'

    // Add Prometheus-related libraries
	implementation 'io.micrometer:micrometer-registry-prometheus'
	implementation 'io.prometheus:simpleclient_pushgateway'
}

// ... omitted ...

Looking at the dependency libraries in the file, you can see that Spring Batch-related libraries and Prometheus and Pushgateway libraries have been added.

Implement a Tasklet-style Job configuration

Add a simple Tasklet-style Job configuration file.

/src/main/java/com/devkuma/batch/prometheus/batch/TaskletStepJobConfiguration.java

package com.devkuma.batch.prometheus.batch;

import java.util.Random;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class TaskletStepJobConfiguration {

    private static final Logger LOGGER = LoggerFactory.getLogger(TaskletStepJobConfiguration.class);

    private Random random;

    public TaskletStepJobConfiguration() {
        this.random = new Random();
    }

    @Bean
    public Job taskletStepJob(JobBuilderFactory jobBuilderFactory, Step taskletStep1, Step taskletStep2) {
        return jobBuilderFactory.get("taskletStepJob")
                                .start(taskletStep1)
                                .next(taskletStep2)
                                .build();
    }

    @Bean
    public Step taskletStep1(StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory.get("taskletStep1")
                                 .tasklet((contribution, chunkContext) -> {
                                     LOGGER.info("taskletStep1");
                                     // simulate processing time
                                     Thread.sleep(random.nextInt(3000));
                                     return RepeatStatus.FINISHED;
                                 })
                                 .build();
    }

    @Bean
    public Step taskletStep2(StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory.get("taskletStep2")
                                 .tasklet((contribution, chunkContext) -> {
                                     LOGGER.info("taskletStep2");
                                     // simulate step failure
                                     int nextInt = random.nextInt(3000);
                                     Thread.sleep(nextInt);
                                     if (nextInt % 5 == 0) {
                                         throw new Exception("Boom!");
                                     }
                                     return RepeatStatus.FINISHED;
                                 })
                                 .build();
    }

}

Implement an itemReader and itemWriter-style Job configuration

Create a Job configuration file with a simple itemReader and itemWriter processing style.

/src/main/java/com/devkuma/batch/prometheus/batch/ItemStepJobConfiguration.java

package com.devkuma.batch.prometheus.batch;

import java.util.LinkedList;
import java.util.List;
import java.util.Random;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ItemStepJobConfiguration {

    private static final Logger LOGGER = LoggerFactory.getLogger(ItemStepJobConfiguration.class);

    private Random random;

    public ItemStepJobConfiguration() {
        this.random = new Random();
    }

    @Bean
    public Job itemStepJob(JobBuilderFactory jobBuilderFactory, Step itemStep) {
        return jobBuilderFactory.get("itemStepJob")
                                .start(itemStep)
                                .build();
    }

    @Bean
    public Step itemStep(StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory.get("itemStep").<Integer, Integer>chunk(3)
                                 .reader(itemReader())
                                 .writer(itemWriter())
                                 .build();
    }

    @Bean
    @StepScope
    public ListItemReader<Integer> itemReader() {
        List<Integer> items = new LinkedList<>();
        // read a random number of items in each run
        for (int i = 0; i < random.nextInt(100); i++) {
            items.add(i);
        }
        return new ListItemReader<>(items);
    }

    @Bean
    public ItemWriter<Integer> itemWriter() {
        return items -> {
            for (Integer item : items) {
                int nextInt = random.nextInt(1000);
                Thread.sleep(nextInt);
                // simulate write failure
                if (nextInt % 57 == 0) {
                    throw new Exception("Boom!");
                }
                LOGGER.info("item = " + item);
            }
        };
    }
}

Create a Job scheduler

A scheduler is not strictly needed here, but to make the graph easier to observe, create a Job scheduler that sends data periodically.

/src/main/java/com/devkuma/batch/prometheus/batch/JobScheduler.java

package com.devkuma.batch.prometheus.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class JobScheduler {

	private final Job taskletStepJob;

	private final Job itemStepJob;

	private final JobLauncher jobLauncher;

	@Autowired
	public JobScheduler(Job taskletStepJob, Job itemStepJob, JobLauncher jobLauncher) {
		this.taskletStepJob = taskletStepJob;
		this.itemStepJob = itemStepJob;
		this.jobLauncher = jobLauncher;
	}

	@Scheduled(cron = "*/10 * * * * *")
	public void launchJob1() throws Exception {
		JobParameters jobParameters = new JobParametersBuilder().addLong("time", System.currentTimeMillis())
																.toJobParameters();

		jobLauncher.run(taskletStepJob, jobParameters);
	}

	@Scheduled(cron = "*/15 * * * * *")
	public void launchJob2() throws Exception {
		JobParameters jobParameters = new JobParametersBuilder().addLong("time", System.currentTimeMillis())
																.toJobParameters();

		jobLauncher.run(itemStepJob, jobParameters);
	}

}

Next, add thread.pool.size to the configuration file.
/src/main/resources/application.properties

thread.pool.size=3

Use the thread.pool.size added earlier to create a configuration file for ThreadPoolTaskScheduler.
/src/main/java/com/devkuma/batch/prometheus/batch/SchedulerConfiguration.java

package com.devkuma.batch.prometheus.config;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler;

@Configuration
public class SchedulerConfiguration {

    @Bean(destroyMethod = "shutdown")
    public ThreadPoolTaskScheduler taskScheduler(@Value("${thread.pool.size}") int threadPoolSize) {
        ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
        threadPoolTaskScheduler.setPoolSize(threadPoolSize);
        return threadPoolTaskScheduler;
    }
}

Prometheus configuration

Now configure Prometheus.
/src/main/resources/application.properties

prometheus.push.rate=5000
prometheus.job.name=springbatch
prometheus.grouping.key=appname
prometheus.pushgateway.url=localhost:9091

Add the Prometheus push interval, Job name, and grouping key. Then configure the Pushgateway URL.

Reflect the configured Prometheus settings in a configuration object.
/src/main/java/com/devkuma/batch/prometheus/config/PrometheusConfiguration.java

package com.devkuma.batch.prometheus.config;

import java.util.HashMap;
import java.util.Map;

import javax.annotation.PostConstruct;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.Scheduled;

import io.micrometer.core.instrument.Metrics;
import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.PushGateway;

@Configuration
public class PrometheusConfiguration {

    private static final Logger LOGGER = LoggerFactory.getLogger(PrometheusConfiguration.class);

    @Value("${prometheus.job.name}")
    private String prometheusJobName;

    @Value("${prometheus.grouping.key}")
    private String prometheusGroupingKey;

    @Value("${prometheus.pushgateway.url}")
    private String prometheusPushGatewayUrl;

    private Map<String, String> groupingKey = new HashMap<>();

    private PushGateway pushGateway;

    private CollectorRegistry collectorRegistry;

    @PostConstruct
    public void init() {
        pushGateway = new PushGateway(prometheusPushGatewayUrl);
        groupingKey.put(prometheusGroupingKey, prometheusJobName);
        PrometheusMeterRegistry prometheusMeterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
        collectorRegistry = prometheusMeterRegistry.getPrometheusRegistry();
        Metrics.globalRegistry.add(prometheusMeterRegistry);
    }

    @Scheduled(fixedRateString = "${prometheus.push.rate}")
    public void pushMetrics() {
        try {
            pushGateway.pushAdd(collectorRegistry, prometheusJobName, groupingKey);
            LOGGER.info("Push Metrics");
        }
        catch (Throwable ex) {
            LOGGER.error("Unable to push metrics to Prometheus Push Gateway", ex);
        }
    }

}

Start Prometheus + Pushgateway + Grafana servers with Docker

Now create docker-compose.yml and configure Prometheus to start Prometheus + Pushgateway + Grafana servers with Docker.

docker-compose.yml configuration file

/src/prometheus/docker-compose.yml

version: '3.7'
services:

  prometheus:
    image: prom/prometheus
    container_name: 'prometheus'
    ports:
      - '9090:9090'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

  pushgateway:
    image: prom/pushgateway
    container_name: 'pushgateway'
    ports:
      - '9091:9091'

  grafana:
    image: grafana/grafana
    container_name: 'grafana'
    ports:
      - '3000:3000'

prometheus.yml configuration file

/src/docker/prometheus/prometheus.yml

global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'springbatch'
    honor_labels: true
    static_configs:
      - targets: ['host.docker.internal:9091'] # pushgateway

Because the test environment is macOS, the Pushgateway URL is written as host.docker.internal:9091; it may differ on Linux and other environments. Enter the URL appropriate for your test environment.

Start Docker

% cd src/prometheus
% docker-compose up -d

Access URLs

Pushgateway
- http://localhost:9091
Prometheus
- http://localhost:9090
Grafana
- http://localhost:3000
  - Default account ID/password: admin/admin

Check Pushgateway collection information

If you access Pushgateway at http://localhost:9091, you can check the information collected from spring-batch.

Check Prometheus collection information

If you access Prometheus at http://localhost:9090, you can check the information collected from spring-batch and delivered through Pushgateway.

Grafana configuration

Now display the information collected from spring-batch more visually through Grafana.

Configure the Grafana data source

First, add a data source.

Select Prometheus as the data source.

When the screen for adding a Prometheus data source appears, enter Name and URL.
Here, the implementation environment is macOS, so http://host.docker.internal:9090 is entered for the URL. Be aware that it may differ in Linux environments.

Configure and check the Grafana dashboard

Next, add the dashboard with Import.

Select the prepared import JSON file, spring-batch-dashboard.json.

Check the contents and click Import.

Now move to the dashboard screen and confirm that spring-batch metric information is displayed as graphs.

References

The example code above is available on GitHub.