If we start our application we can see that some endpoints like health and info are already exposed to the /actuator endpoint per default.
Triggering the /actuator/health endpoint gives us a metric if the service is up and running:
▶ http GET "http://localhost:8080/actuator/health"
HTTP/1.1 200
Connection: keep-alive
Content-Type: application/vnd.spring-boot.actuator.v3+json
Date: Wed, 21 Oct 2020 18:11:35 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked
{
"status": "UP"
}
Spring Boot Actuator can be integrated into Spring Boot Admin which provides a visual admin interface for your application. But this approach is not very popular and has some limitations. Therefore, we use Prometheus instead of Spring Boot Actuator and Grafana instead of Spring Boot Admin to have a more popular and framework/language-independent solution. This solution approach needs vendor-neutral metrics and Micrometer is a popular tool for this use case. Micrometer
- Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.
Micrometer is an open-source project and provides a metric facade that exposes metric data in a vendor-neutral format that a monitoring system can understand. These monitoring systems are supported: - AppOptics
- Azure Monitor
- Netflix Atlas
- CloudWatch
- Datadog
- Dynatrace
- Elastic
- Ganglia
- Graphite
- Humio
- Influx/Telegraf
- JMX
- KairosDB
- New Relic
- Prometheus
- SignalFx
- Google Stackdriver
- StatsD
- Wavefront
Micrometer is not part of the Spring ecosystem and needs to be added as a dependency. In our demo application, this was already done in the Spring Initializr configuration. Next step is to expose the Prometheus metrics in application.properties:
management.endpoints.web.exposure.include=prometheus,health,info,metric
Now we can trigger this endpoint and see the Prometheus metrics:
Custom Metrics
We can also define some custom metrics, which I will demonstrate in this section. The demo contains a Scheduler class which periodically runs the included schedulingTask method.
To be able to send custom metrics we need to import MeterRegistry from the Micrometer library and inject it into our class. For more detail please check the official documentation. It is possible to instantiate these types of meters from MeterRegistry:
- Counter: reports merely a count over a specified property of an application
- Gauge: shows the current value of a meter
- Timers: measures latencies or frequency of events
- DistributionSummary: provides distribution of events and a simple summary
I implemented a counter and a gauge for demonstration purposes:
@Component
public class Scheduler {
private final AtomicInteger testGauge;
private final Counter testCounter;
public Scheduler(MeterRegistry meterRegistry) {
// Counter vs. gauge, summary vs. histogram
// https://prometheus.io/docs/practices/instrumentation/#counter-vs-gauge-summary-vs-histogram
testGauge = meterRegistry.gauge("custom_gauge", new AtomicInteger(0));
testCounter = meterRegistry.counter("custom_counter");
}
@Scheduled(fixedRateString = "1000", initialDelayString = "0")
public void schedulingTask() {
testGauge.set(Scheduler.getRandomNumberInRange(0, 100));
testCounter.increment();
}
private static int getRandomNumberInRange(int min, int max) {
if (min >= max) {
throw new IllegalArgumentException("max must be greater than min");
}
Random r = new Random();
return r.nextInt((max - min) + 1) + min;
}
}
If we run the application we can see that our custom metrics are exposed via the actuatuor/prometheus endpoint:
▶ http GET "http://localhost:8080/actuator/prometheus" | grep custom
# HELP custom_gauge
# TYPE custom_gauge gauge
custom_gauge 29.0
# HELP custom_counter_total
# TYPE custom_counter_total counter
custom_counter_total 722.0
As we now have the metrics available in a format that Prometheus can understand, we will look at how to set up Prometheus.
Prometheus
Prometheus stores our metric data in time series in memory by periodically pulling it via HTTP. The data can be visualized by a console template language, a built-in expression browser, or by integrating Grafana (which we will do after setting up Prometheus). In this demo, we will run Prometheus locally in a Docker container and we, therefore, need some configurations in a prometheus.yml file that you can place anywhere on your hard drive:
global:
scrape_interval: 10s # How frequently to scrape targets by default
scrape_configs:
- job_name: 'spring_micrometer' # The job name is assigned to scraped metrics by default.
metrics_path: '/actuator/prometheus' # The HTTP resource path on which to fetch metrics from targets.
scrape_interval: 5s # How frequently to scrape targets from this job.
static_configs: # A static_config allows specifying a list of targets and a common label set for them
- targets: ['192.168.178.22:8080']
Now we are ready to run Prometheus:
docker run -d -p 9090:9090 -v <path-to-your-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus
<path-to-your-prometheus.yml> should be the path where you placed the prometheus.yml configuration file described above.
Finally, we can open the Prometheus on http://localhost:9090 in the web browser and search for our custom metric named custom_gauge:
Prometheus Graph
To check that Prometheus is correctly listening to our locally running Spring Boot application we can navigate to Status -> Targets in the top main navigation bar:
Prometheus Targets
Grafana
The included Prometheus browser graph is nice for basic visualization of our metrics but we will use Grafana instead. Grafana provides a rich UI where you create, explore and share dashboards that contain multiple graphs. Grafana can pull data from various data sources like Prometheus, Elasticsearch, InfluxDB, etc. It also allows you to set rule-based alerts, which then can notify you over Slack, Email, Hipchat, and similar.
We start Grafana also locally in a Docker container:
docker run -d -p 3000:3000 grafana/grafana
Opening http://localhost:3000 in a browser should now show the following login page:
Grafana Login
You can log in using the default username admin and the default password admin. After login, you should change these default passwords by visiting http://localhost:3000/profile/password.
The first step is to add our local Prometheus as our data source:
Grafana Add Datasource
Grafana Select Prometheus
Grafana Prometheus Config
Community Dashboard
Grafana Import JVM Dashboard
After loading the URL we can see the imported dashboard:
Grafana Imported JVM Dashboard
Custom Metric Dashboard
Finally, we want to create a new dashboard where we show our custom metrics. The first step is to create a new dashboard:
Grafana Imported JVM Dashboard
Now we see a new dashboard where we can create a new panel:
Grafana New Dashboard Panel
In the first panel we add a visualization for our custom_gauge metric. I use the Stat visualization as it shows the current value and a simple graph:
Grafana Stat Gauge Vizualization
Additionally, a new panel for the custom_counter metric is added to our dashboard:
Grafana Graph Counter Vizualization
In the end, the dashboard looks like this:
Grafana Custom Dashboard
Conclusion
It is important to monitor an application’s metrics and health which helps us to improve performance, manage the app in a better way and notice unoptimized behavior. Monitoring each service is important to be able to maintain a system that consists of many microservices.
In this article, I showed how a Spring Boot web application can be monitored using Micrometer which exposes metrics from our application, Prometheus which stores the metric data and Grafana to visualize the data in graphs.
Also, Since the this architecture use pull-based metric system we don’t need to define any endpoints from the application side and it is more effective to use with containerized, replica environments as well.
