Health Checks, Metrics & More with Spring Boot Actuator

I've been working with Spring Boot for a few years now and I'm a big fan. There are lots of things to like about Spring Boot, but one thing that really stands out for me is the emphasis on production readiness.

Production Readiness

Production readiness is about looking beyond functional requirements and ensuring your application can be properly managed and monitored in production. Some key things to consider when thinking about production readiness are...
  • Health checks
  • Viewing application configuration - application properties, Environment variables etc. 
  • Viewing and altering log configuration
  • Viewing application metrics - JVM, classloader, threading and garbage collection. 
  • Audibility of key application events
Spring provides all of this functionality out of the box via Spring Boot Actuator, a sub project of Spring Boot. A range of RESTful management and monitoring endpoints are provided so you don't have to implement these features each time you build an application. To enable these endpoints, simply add the Actuator starter POM to your project and you're ready to go. The sections that follow will describe some of the key Actuator endpoints and how they can be configured and used. Actuator exposes quite a few endpoints so I'm going to focus on the ones I think are most useful.

Endpoint Security

By default most Actuator endpoints are considered sensitive. Only the health and info endpoints are non sensitive (by default) and can be accessed without authentication. All other endpoints must be accessed using HTTP basic authentication. The sensitivity of individual endpoints can be configured in application.properties. For example, you could change the default security setting and make the metrics endpoint accessible without basic authentication.
  

endpoints.metrics.sensitive=false
  
Its also possible to disable security on all Actuator endpoints as follows.


management.security.enabled=false
  
Actuator endpoints expose lots of data about the internals of your application so you need to be very careful when disabling security. If your application sits behind a firewall it may be acceptable to disable security, but if the application is public facing then securing the Actuator endpoints is an absolute must. My personal opinion would be to use basic auth in all cases. Even if you're application is sitting behind a firewall it does no harm at all to have an extra layer of security in place.

Health Checks

A health check is an endpoint that returns a message describing the health of an application. An application is considered healthy if its capable of successfully servicing client requests. To do that its core components or dependencies should be up and running. Consider an application that interacts with a database and a JMS message broker.
The health check for such an application might consist of a simple query against the database and creating a connection with the JMS broker. By proving these core components are up and running we can say with a fair degree of confidence that our application is healthy.
Actuator provides a health check endpoint out of the box, which can be accessed at /health using a simple HTTP GET.  Using curl for example we can call the health check endpoint as follows.


curl --user user:password localhost:8080/health

The JSON response below is made up of statusdiskSpace and db. Status describes the overall health of the application, in this case UP indicates that the application is healthy. diskSpace describes total disk space, remaining disk space and a minimum threshold. The threshold value is configurable in application.properties using management.health.diskspace.threshold. If this value is greater than the free value, the status of the diskSpace check will be DOWN.
db describes the health of the database and is derived by running a simple query. By default Boot will look up the validation query for the specific database engine, for example on Oracle the validation query will be select 1 from dual. You can also override the default query by writing your own health check implementation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
    "status": "UP",
    "diskSpace": {
        "status": "UP",
        "total": 1983077322752,
        "free": 1732231667712,
        "threshold": 10485760
     },
     "db": {
        "status": "UP",
        "database": "HSQL Database Engine",
        "hello": 1
    }
}

When the status UP is returned, a HTTP 200 response code is used. If the health check returns DOWN the response code is HTTP 503 (Service Unavailable). This is useful for load balancers or monitoring tools that may be calling the health check. They can derive the health status using the HTTP response code rather than the response message, which can be simply logged for information. If the format of the response message happens to change it wont break the tools that are calling the health endpoint.

Automatic Health Checks 

The sample response above consists of 2 separate checks, disk space and database. Spring Boot automatically detects components used by an application and runs the appropriate health checks. In the sample app it detects a DataSource and therefore runs the database health check. If a JmsConnectionFactory was configured in the application, it would also run a JMS health check. As well as the examples shown here, Actuator also provides automatic health checks for SMTP, Mongo, Redis, ElasticSearch, RabbitMQ and more. If you'd prefer to disable these automatic health checks you can do so by setting management.health.defaults.enabled=false in application.properties.

Health Check Security 

By default when the health check is called without basic authentication, only the status (UP or DOWN) is returned in the response body. This ensures that sensitive information such as the name of the database engine aren't exposed to the world. When basic authentication is used a detailed JSON response is returned.

Custom Health Checks

If the standard health checks aren't sufficient and you need something bespoke, you can override the default health check with your own implementation. Simply create a class that implements HealthIndicator and override the health method as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class HealthCheck implements HealthIndicator {
 
    @Override
    public Health health() {
 
        if(isRemoteServiceUp()){
            return Health.up()
                         .withDetail("remote service", "online")
                         .build();  
        }
  
        return Health.down()
                     .withDetail("remote service", "offline")
                     .build();    
    }
 
    private boolean isRemoteServiceUp(){
  
        // perform call out to remote service to check if its up
        return true;
    }
}

In the example above I've created a dummy method isRemoteServiceUp to check the status of some fictitious remote service. In reality this method would perform a remote call to check that the service is up and the resulting status used to build a health response.

Info Endpoint

The info endpoint provides general information about your application. The type of information returned can be customised, but one area that is particularly useful is the application versioning information. I've seen situations in the past where there's been confusion around the version of a running application. Being able to check application info on a running instance is a great way to confirm that a deployment has gone as expected and the new version of the application is indeed up and running. The info endpoint is can be called with curl as follows


curl --user user:password localhost:8080/info

The sample response below includes useful information about the GIT branch and commit that's been deployed. This information is read from your local Git configuration using the git commit plugin.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
    "java": {
        "version": "1.8.0_73"
    },
    "git": {
        "commit": {
            "time": "2017-08-31T06:28:56.000+0000",
            "id": "d0ca400"
            },
        "branch": "master"
    },
    "build": {
        "version": "0.1.0",
        "artifact": "boot-actuator-demo",
        "name": "boot-actuator-demo",
        "group": "org.briansjavablog",
        "time": "2017-09-04T06:47:40.000+0000"
    },
    "remoteServiceVersion": "1.2.3"
}

Simply add the following to the plugins section of your POM and Boot will automatically pick up the branch and commit used to build the artifact.

1
2
3
4
 <plugin>
    <groupId>pl.project13.maven</groupId>
    <artifactId>git-commit-id-plugin</artifactId>
</plugin>     

The build section is provided by the boot maven plugin. It derives Maven related information for the artefact at build time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>build-info</goal>
            </goals>
        </execution>
    </executions>
</plugin> 

Custom Application Information

You can add custom information to the info endpoint response via application.properties. In the sample response above, java.version is populated using the following configuration in application.properties. The @...@ notation allows us to access any value from the maven project.properties at build time.


info.java.version=@java.version@

There may be data you want to expose on the info endpoint that isn't available until runtime. You can do this by creating a class that implements the InfoContributor interface. This class allows you to add whatever information you want to the info endpoint. The example below exposes the version of some fictitious remote service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
@Component
public class CustomInfoContributor implements InfoContributor {

   @Override
   public void contribute(Builder builder) {
      // add whatever data you want to expose on info endpoint... 
      builder.withDetail("remoteServiceVersion", getRemoteServiceVersion());
   }
   // service would check version of some remote service being used 
   private String getRemoteServiceVersion(){
      return "1.2.3";
   }
}

Metrics Endpoint

The metrics endpoint exposes a range of application metrics that are extremely useful when it comes to monitoring and trouble shooting poorly performing applications. Data exposed by the metrics endpoint includes
  • System Memory - Total and free system memory in KB
  • System Uptime - Server uptime and application context uptime
  • JVM Memory Usage - Initial, used and max available heap and non heap size
  • Thread Statistics - total thread count, number of threads started and number of daemon threads
  • Classloader - total classes loaded, number currently loaded and number unloaded
  • Garbage Collector - GC algorithm run, number of times it has run and time taken for last GC
  • HTTP Sessions - Current and peak number of HTTP sessions
  • Data Source - total number of connections and total active connections 
  • Endpoints - number of times each endpoint has been called and the last response time for each endpoint
Using curl you can access the metrics endpoint with


curl --user user:password localhost:8080/metrics

Most of the data in the sample response is fairly intuitive. One section worth describing further is the use of gauge and counter for endpoint metrics. A gauge is used to record a single value, for example on line 31 gauge.response.metrics: 43.0 shows the last response time in milliseconds for the metrics endpoint. A counter records the increment or decrement of a value, for example on line 36 counter.status.metrics: 6 shows the number of times the metrics endpoint has been called since the app started.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
{
    "mem": 702657,
    "mem.free": 451758,
    "processors": 4,
    "instance.uptime": 32291251,
    "uptime": 32343883,
    "systemload.average": -1.0,
    "heap.committed": 601600,
    "heap.init": 245760,
    "heap.used": 149841,
    "heap": 3484672,
    "nonheap.committed": 103408,
    "nonheap.init": 2496,
    "nonheap.used": 101057,
    "nonheap": 0,
    "threads.peak": 24,
    "threads.daemon": 21,
    "threads.totalStarted": 30,
    "threads": 23,
    "classes": 13285,
    "classes.loaded": 13285,
    "classes.unloaded": 0,
    "gc.ps_scavenge.count": 10,
    "gc.ps_scavenge.time": 354,
    "gc.ps_marksweep.count": 3,
    "gc.ps_marksweep.time": 1037,
    "httpsessions.max": -1,
    "httpsessions.active": 0,
    "datasource.primary.active": 0,
    "datasource.primary.usage": 0.0,
    "gauge.response.metrics": 43.0,
    "gauge.response.unmapped": 0.0,
    "gauge.response.info": 414.0,
    "gauge.response.star-star": 22.0,
    "counter.status.200.info": 1,
    "counter.status.200.metrics": 6,
    "counter.status.404.star-star": 1,
    "counter.status.401.unmapped": 1
}

Logger Endpoint

The loggers endpoint exposes a detailed view of your applications log configuration. Using curl you can access the loggers endpoint with


curl --user user:password localhost:8080/loggers

The ROOT logger and per package configuration is included, as defined in your applications logback.xml. If a package has no explicit log level defined in your logback.xml, its configuredLevel will be null and its effectiveLevel will be inherited from the ROOT logger.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
    "levels": ["OFF", "ERROR", "WARN", "INFO", "DEBUG", "TRACE"],
    "loggers": {
        "ROOT": {
            "configuredLevel": "INFO",
            "effectiveLevel": "INFO"
        },
        "com": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog.samples": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog.samples.Application": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        }
    }
}

Env Endpoint
The env endpoint exposes data from the application Environment, the object that encapsulates all configuration available to the running application. Using curl you can access the env endpoint with


curl --user user:password localhost:8080/env

Below is a sample response that includes profiles, server ports, Servlet initialising parameters, system properties, environment variables and application properties.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
   "profiles": [prod],
   "server.ports": {
      "local.server.port": 8080
   },
   "servletContextInitParams": {},
   "systemProperties": {
      "java.runtime.name": "Java(TM) SE Runtime Environment",
      "sun.boot.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\jre\\bin",
      "java.vm.version": "25.73-b02",
      "java.vm.specification.name": "Java Virtual Machine Specification",
      "user.dir": "C:\\blogs\\samples\\boot-actuator-demo",
      "java.runtime.version": "1.8.0_73-b02",
      "line.separator": "\r\n",
      "java.vm.specification.vendor": "Oracle Corporation",
      "user.variant": "",
      "os.name": "Windows 10",
      "classworlds.conf": "C:/apps/apache-maven-3.3.3/bin/m2.conf",
      "sun.jnu.encoding": "Cp1252",
      "spring.beaninfo.ignore": "true",
      "java.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\bin;C:\\WINDOWS\\Sun\\Java\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\Users\\brianh\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\local\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Users\\brianh\\bin;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Program Files\\Broadcom\\Broadcom 802.11 Network Adapter;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files (x86)\\ATI Technologies\\ATI.ACE\\Core-Static;C:\\Program Files (x86)\\Skype\\Phone;C:\\Program Files\\nodejs;C:\\apps\\apache-maven-3.3.3\\bin;C:\\Program Files\\Docker Toolbox;C:\\Users\\brianh\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\brianh\\AppData\\Roaming\\npm;C:\\Program Files\\Git\\usr\\bin\\vendor_perl;C:\\Program Files\\Git\\usr\\bin\\core_perl;.",
      "java.specification.name": "Java Platform API Specification",
      "java.class.version": "52.0",
      "java.vendor": "Oracle Corporation",
      "catalina.base": "C:\\Users\\brianh\\AppData\\Local\\Temp\\tomcat.7213012654997851459.8080",
      "maven.home": "C:\\apps\\apache-maven-3.3.3",
      "file.separator": "\\",
   },
   "systemEnvironment": {
      "TEMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
      "HOSTNAME": "DESKTOP-867SP8L",
      "HOMEDRIVE": "C:",
      "USERPROFILE": "C:\\Users\\brianh",
      "TMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
      "CommonProgramFiles(x86)": "C:\\Program Files (x86)\\Common Files",
      "NUMBER_OF_PROCESSORS": "4",
      "DOCKER_TOOLBOX_INSTALL_PATH": "C:\\Program Files\\Docker Toolbox",
      "HOME": "C:\\Users\\brianh"
   },
   "applicationConfig: [classpath:/application.properties]": {
      "spring.jpa.show-sql": "true",
      "endpoints.health.sensitive": "false",
      "security.user": "user",
      "security.user.password": "******",
      "info.java.version": "1.8.0_73",
      "spring.jpa.database-platform": "org.hibernate.dialect.HSQLDialect"
   }
}

This information is very useful for checking that an application is using the expected configuration. It's particularly useful when an application is deployed to a test or prod environment and using externalised configuration such as an external properties file or environment variables.

Heap Dump Endpoint

The heapdump endpoint triggers a heap dump of the application JVM and returns a hprof file. This can be very useful when trying to get to the bottom of memory or threading issues in your application. Being able to trigger and retrieve the heap dump via HTTP is nice as it saves you having to remote onto the box directly.  Its worth mentioning that heap dumps are expensive, so the heapdump endpoint should be triggered with care in a production environment.

Visualising Actuator Endpoint Data

The actuator endpoints we've looked at in this post are certainly useful, but dealing with raw JSON responses makes some of the information difficult to understand at a glance. What we'd really like is some sort of visual representation of the endpoint data. Luckily the Spring Boot Admin project gives us exactly that, by taking the data returned from the various Actuator endpoints and displaying it in an admin console.  The screen shot below shows how Spring Boot Admin displays health and metric data in a nice user friendly format.

Spring Boot Admin
In a follow up post I'll show you how to use Spring Boot Admin to display health and monitoring data for your Spring Boot applications.

Wrapping Up

In this post we looked at some of the most useful Spring Boot Actuator endpoints and how they can provide a detailed view into the internals of a running application. The sample code that accompanies this post is available on Github, so feel free to pull it down and have a play around. As always if you have any questions or comments please leave a note below.

Comments

Popular posts from this blog

Spring Web Services Tutorial

An Introduction to Wiremock

Spring JMS Tutorial with ActiveMQ

REST Endpoint Testing With MockMvc

Spring Batch Tutorial

Externalising Spring Configuration

Axis2 Web Service Client Tutorial

Spring Boot & Amazon Web Services (EC2, RDS & S3)