Health Checks, Metrics & More with Spring Boot Actuator

I've been working with Spring Boot for a few years now and I'm a big fan. There are lots of things to like about Spring Boot, but one thing that really stands out for me is the emphasis on production readiness.

Production Readiness

Production readiness is about looking beyond functional requirements and ensuring your application can be properly managed and monitored in production. Some key things to consider when thinking about production readiness are...
  • Health checks
  • Viewing application configuration - application properties, Environment variables etc. 
  • Viewing and altering log configuration
  • Viewing application metrics - JVM, classloader, threading and garbage collection. 
  • Audibility of key application events
Spring provides all of this functionality out of the box via Spring Boot Actuator, a sub project of Spring Boot. A range of RESTful management and monitoring endpoints are provided so you don't have to implement these features each time you build an application. To enable these endpoints, simply add the Actuator starter POM to your project and you're ready to go. The sections that follow will describe some of the key Actuator endpoints and how they can be configured and used. Actuator exposes quite a few endpoints so I'm going to focus on the ones I think are most useful.

Endpoint Security

By default most Actuator endpoints are considered sensitive. Only the health and info endpoints are non sensitive (by default) and can be accessed without authentication. All other endpoints must be accessed using HTTP basic authentication. The sensitivity of individual endpoints can be configured in application.properties. For example, you could change the default security setting and make the metrics endpoint accessible without basic authentication.
  


endpoints.metrics.sensitive=false







  

Its also possible to disable security on all Actuator endpoints as follows.











management.security.enabled=false










  

Actuator endpoints expose lots of data about the internals of your application so you need to be very careful when disabling security. If your application sits behind a firewall it may be acceptable to disable security, but if the application is public facing then securing the Actuator endpoints is an absolute must. My personal opinion would be to use basic auth in all cases. Even if you're application is sitting behind a firewall it does no harm at all to have an extra layer of security in place.



Health Checks

A health check is an endpoint that returns a message describing the health of an application. An application is considered healthy if its capable of successfully servicing client requests. To do that its core components or dependencies should be up and running. Consider an application that interacts with a database and a JMS message broker.

The health check for such an application might consist of a simple query against the database and creating a connection with the JMS broker. By proving these core components are up and running we can say with a fair degree of confidence that our application is healthy.

Actuator provides a health check endpoint out of the box, which can be accessed at /health using a simple HTTP GET.  Using curl for example we can call the health check endpoint as follows.











curl --user user:password localhost:8080/health












The JSON response below is made up of status, diskSpace and db. Status describes the overall health of the application, in this case UP indicates that the application is healthy. diskSpace describes total disk space, remaining disk space and a minimum threshold. The threshold value is configurable in application.properties using management.health.diskspace.threshold. If this value is greater than the free value, the status of the diskSpace check will be DOWN.

db describes the health of the database and is derived by running a simple query. By default Boot will look up the validation query for the specific database engine, for example on Oracle the validation query will be select 1 from dual. You can also override the default query by writing your own health check implementation.










 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
    "status": "UP",
    "diskSpace": {
        "status": "UP",
        "total": 1983077322752,
        "free": 1732231667712,
        "threshold": 10485760
     },
     "db": {
        "status": "UP",
        "database": "HSQL Database Engine",
        "hello": 1
    }
}










When the status UP is returned, a HTTP 200 response code is used. If the health check returns DOWN the response code is HTTP 503 (Service Unavailable). This is useful for load balancers or monitoring tools that may be calling the health check. They can derive the health status using the HTTP response code rather than the response message, which can be simply logged for information. If the format of the response message happens to change it wont break the tools that are calling the health endpoint.



Automatic Health Checks 

The sample response above consists of 2 separate checks, disk space and database. Spring Boot automatically detects components used by an application and runs the appropriate health checks. In the sample app it detects a DataSource and therefore runs the database health check. If a JmsConnectionFactory was configured in the application, it would also run a JMS health check. As well as the examples shown here, Actuator also provides automatic health checks for SMTP, Mongo, Redis, ElasticSearch, RabbitMQ and more. If you'd prefer to disable these automatic health checks you can do so by setting management.health.defaults.enabled=false in application.properties.



Health Check Security 

By default when the health check is called without basic authentication, only the status (UP or DOWN) is returned in the response body. This ensures that sensitive information such as the name of the database engine aren't exposed to the world. When basic authentication is used a detailed JSON response is returned.



Custom Health Checks

If the standard health checks aren't sufficient and you need something bespoke, you can override the default health check with your own implementation. Simply create a class that implements HealthIndicator and override the health method as follows.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class HealthCheck implements HealthIndicator {
 
    @Override
    public Health health() {
 
        if(isRemoteServiceUp()){
            return Health.up()
                         .withDetail("remote service", "online")
                         .build();  
        }
  
        return Health.down()
                     .withDetail("remote service", "offline")
                     .build();    
    }
 
    private boolean isRemoteServiceUp(){
  
        // perform call out to remote service to check if its up
        return true;
    }
}










In the example above I've created a dummy method isRemoteServiceUp to check the status of some fictitious remote service. In reality this method would perform a remote call to check that the service is up and the resulting status used to build a health response.



Info Endpoint

The info endpoint provides general information about your application. The type of information returned can be customised, but one area that is particularly useful is the application versioning information. I've seen situations in the past where there's been confusion around the version of a running application. Being able to check application info on a running instance is a great way to confirm that a deployment has gone as expected and the new version of the application is indeed up and running. The info endpoint is can be called with curl as follows










curl --user user:password localhost:8080/info












The sample response below includes useful information about the GIT branch and commit that's been deployed. This information is read from your local Git configuration using the git commit plugin.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
    "java": {
        "version": "1.8.0_73"
    },
    "git": {
        "commit": {
            "time": "2017-08-31T06:28:56.000+0000",
            "id": "d0ca400"
            },
        "branch": "master"
    },
    "build": {
        "version": "0.1.0",
        "artifact": "boot-actuator-demo",
        "name": "boot-actuator-demo",
        "group": "org.briansjavablog",
        "time": "2017-09-04T06:47:40.000+0000"
    },
    "remoteServiceVersion": "1.2.3"
}









Simply add the following to the plugins section of your POM and Boot will automatically pick up the branch and commit used to build the artifact.








1
2
3
4


 <plugin>
    <groupId>pl.project13.maven</groupId>
    <artifactId>git-commit-id-plugin</artifactId>
</plugin>     









The build section is provided by the boot maven plugin. It derives Maven related information for the artefact at build time.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>build-info</goal>
            </goals>
        </execution>
    </executions>
</plugin> 









Custom Application Information

You can add custom information to the info endpoint response via application.properties. In the sample response above, java.version is populated using the following configuration in application.properties. The @...@ notation allows us to access any value from the maven project.properties at build time.










info.java.version=@java.version@









There may be data you want to expose on the info endpoint that isn't available until runtime. You can do this by creating a class that implements the InfoContributor interface. This class allows you to add whatever information you want to the info endpoint. The example below exposes the version of some fictitious remote service.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@Component
public class CustomInfoContributor implements InfoContributor {

   @Override
   public void contribute(Builder builder) {
      // add whatever data you want to expose on info endpoint... 
      builder.withDetail("remoteServiceVersion", getRemoteServiceVersion());
   }
   // service would check version of some remote service being used 
   private String getRemoteServiceVersion(){
      return "1.2.3";
   }
}









Metrics Endpoint

The metrics endpoint exposes a range of application metrics that are extremely useful when it comes to monitoring and trouble shooting poorly performing applications. Data exposed by the metrics endpoint includes


    

  • System Memory - Total and free system memory in KB
    • 

  • System Uptime - Server uptime and application context uptime
    • 

  • JVM Memory Usage - Initial, used and max available heap and non heap size
    • 

  • Thread Statistics - total thread count, number of threads started and number of daemon threads
    • 

  • Classloader - total classes loaded, number currently loaded and number unloaded
    • 

  • Garbage Collector - GC algorithm run, number of times it has run and time taken for last GC
    • 

  • HTTP Sessions - Current and peak number of HTTP sessions
    • 

  • Data Source - total number of connections and total active connections 
    • 

  • Endpoints - number of times each endpoint has been called and the last response time for each endpoint
    • 




Using curl you can access the metrics endpoint with












curl --user user:password localhost:8080/metrics















Most of the data in the sample response is fairly intuitive. One section worth describing further is the use of gauge and counter for endpoint metrics. A gauge is used to record a single value, for example on line 31 gauge.response.metrics: 43.0 shows the last response time in milliseconds for the metrics endpoint. A counter records the increment or decrement of a value, for example on line 36 counter.status.metrics: 6 shows the number of times the metrics endpoint has been called since the app started.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
    "mem": 702657,
    "mem.free": 451758,
    "processors": 4,
    "instance.uptime": 32291251,
    "uptime": 32343883,
    "systemload.average": -1.0,
    "heap.committed": 601600,
    "heap.init": 245760,
    "heap.used": 149841,
    "heap": 3484672,
    "nonheap.committed": 103408,
    "nonheap.init": 2496,
    "nonheap.used": 101057,
    "nonheap": 0,
    "threads.peak": 24,
    "threads.daemon": 21,
    "threads.totalStarted": 30,
    "threads": 23,
    "classes": 13285,
    "classes.loaded": 13285,
    "classes.unloaded": 0,
    "gc.ps_scavenge.count": 10,
    "gc.ps_scavenge.time": 354,
    "gc.ps_marksweep.count": 3,
    "gc.ps_marksweep.time": 1037,
    "httpsessions.max": -1,
    "httpsessions.active": 0,
    "datasource.primary.active": 0,
    "datasource.primary.usage": 0.0,
    "gauge.response.metrics": 43.0,
    "gauge.response.unmapped": 0.0,
    "gauge.response.info": 414.0,
    "gauge.response.star-star": 22.0,
    "counter.status.200.info": 1,
    "counter.status.200.metrics": 6,
    "counter.status.404.star-star": 1,
    "counter.status.401.unmapped": 1
}









Logger Endpoint

The loggers endpoint exposes a detailed view of your applications log configuration. Using curl you can access the loggers endpoint with










curl --user user:password localhost:8080/loggers












The ROOT logger and per package configuration is included, as defined in your applications logback.xml. If a package has no explicit log level defined in your logback.xml, its configuredLevel will be null and its effectiveLevel will be inherited from the ROOT logger.








 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


{
    "levels": ["OFF", "ERROR", "WARN", "INFO", "DEBUG", "TRACE"],
    "loggers": {
        "ROOT": {
            "configuredLevel": "INFO",
            "effectiveLevel": "INFO"
        },
        "com": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog.samples": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.blog.samples.Application": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        }
    }
}










Env Endpoint

The env endpoint exposes data from the application Environment, the object that encapsulates all configuration available to the running application. Using curl you can access the env endpoint with












curl --user user:password localhost:8080/env









Below is a sample response that includes profiles, server ports, Servlet initialising parameters, system properties, environment variables and application properties. 










 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
   "profiles": [prod],
   "server.ports": {
      "local.server.port": 8080
   },
   "servletContextInitParams": {},
   "systemProperties": {
      "java.runtime.name": "Java(TM) SE Runtime Environment",
      "sun.boot.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\jre\\bin",
      "java.vm.version": "25.73-b02",
      "java.vm.specification.name": "Java Virtual Machine Specification",
      "user.dir": "C:\\blogs\\samples\\boot-actuator-demo",
      "java.runtime.version": "1.8.0_73-b02",
      "line.separator": "\r\n",
      "java.vm.specification.vendor": "Oracle Corporation",
      "user.variant": "",
      "os.name": "Windows 10",
      "classworlds.conf": "C:/apps/apache-maven-3.3.3/bin/m2.conf",
      "sun.jnu.encoding": "Cp1252",
      "spring.beaninfo.ignore": "true",
      "java.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\bin;C:\\WINDOWS\\Sun\\Java\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\Users\\brianh\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\local\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Users\\brianh\\bin;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Program Files\\Broadcom\\Broadcom 802.11 Network Adapter;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files (x86)\\ATI Technologies\\ATI.ACE\\Core-Static;C:\\Program Files (x86)\\Skype\\Phone;C:\\Program Files\\nodejs;C:\\apps\\apache-maven-3.3.3\\bin;C:\\Program Files\\Docker Toolbox;C:\\Users\\brianh\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\brianh\\AppData\\Roaming\\npm;C:\\Program Files\\Git\\usr\\bin\\vendor_perl;C:\\Program Files\\Git\\usr\\bin\\core_perl;.",
      "java.specification.name": "Java Platform API Specification",
      "java.class.version": "52.0",
      "java.vendor": "Oracle Corporation",
      "catalina.base": "C:\\Users\\brianh\\AppData\\Local\\Temp\\tomcat.7213012654997851459.8080",
      "maven.home": "C:\\apps\\apache-maven-3.3.3",
      "file.separator": "\\",
   },
   "systemEnvironment": {
      "TEMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
      "HOSTNAME": "DESKTOP-867SP8L",
      "HOMEDRIVE": "C:",
      "USERPROFILE": "C:\\Users\\brianh",
      "TMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
      "CommonProgramFiles(x86)": "C:\\Program Files (x86)\\Common Files",
      "NUMBER_OF_PROCESSORS": "4",
      "DOCKER_TOOLBOX_INSTALL_PATH": "C:\\Program Files\\Docker Toolbox",
      "HOME": "C:\\Users\\brianh"
   },
   "applicationConfig: [classpath:/application.properties]": {
      "spring.jpa.show-sql": "true",
      "endpoints.health.sensitive": "false",
      "security.user": "user",
      "security.user.password": "******",
      "info.java.version": "1.8.0_73",
      "spring.jpa.database-platform": "org.hibernate.dialect.HSQLDialect"
   }
}










This information is very useful for checking that an application is using the expected configuration. It's particularly useful when an application is deployed to a test or prod environment and using externalised configuration such as an external properties file or environment variables.



Heap Dump Endpoint


The heapdump endpoint triggers a heap dump of the application JVM and returns a hprof file. This can be very useful when trying to get to the bottom of memory or threading issues in your application. Being able to trigger and retrieve the heap dump via HTTP is nice as it saves you having to remote onto the box directly.  Its worth mentioning that heap dumps are expensive, so the heapdump endpoint should be triggered with care in a production environment.



Visualising Actuator Endpoint Data

The actuator endpoints we've looked at in this post are certainly useful, but dealing with raw JSON responses makes some of the information difficult to understand at a glance. What we'd really like is some sort of visual representation of the endpoint data. Luckily the Spring Boot Admin project gives us exactly that, by taking the data returned from the various Actuator endpoints and displaying it in an admin console.  The screen shot below shows how Spring Boot Admin displays health and metric data in a nice user friendly format.








Spring Boot Admin



In a follow up post I'll show you how to use Spring Boot Admin to display health and monitoring data for your Spring Boot applications.



Wrapping Up

In this post we looked at some of the most useful Spring Boot Actuator endpoints and how they can provide a detailed view into the internals of a running application. The sample code that accompanies this post is available on Github, so feel free to pull it down and have a play around. As always if you have any questions or comments please leave a note below.











































Comments











Post a Comment































Popular posts from this blog









Spring Web Services Tutorial




















Image







Update

I've recently written a more up to date version of this post which describes how to build a contract first web service using Apache CXF. Check it out here.

Introduction

Modern enterprise applications are rarely stand alone and often rely on data and services provided by external systems. In order for different types of systems to communicate there must be a  communication protocol of some sort, a standard way of sending and receiving messages in a format that is recognised and supported by all major platforms. SOAP (Simple Object Application Protocol) is such a protocol, and allows applications to communicate by exchanging messages in a standard XML format.

SOAP Web Services provide a platform agnostic integration mechanism that allows disparate systems to exchange data regardless of the platform they are running on. For example, SOAP web services are commonly used to integrate .NET applications with applications running on the Java platform. Almost all modern platforms and …








Read more










REST Endpoint Testing With MockMvc





















In this post I'm going to show you how to test a Spring MVC Rest endpoint without deploying your application to a server. In the past, full integration tests were the only meaningful way to test a Spring REST endpoint. This involved spinning up a test server like Tomcat or Jetty, deploying the application, calling the test endpoint, running some assertions and then terminating the server. While this is an effective way to test an endpoint, it isn't particularly fast. We're forced to wait while the entire application is stood up, just to test a single endpoint.

An alternative approach is to unit test by manually instantiating the Controller and mocking out any required dependencies.  While such a test will run much faster than a full integration test, its of limited value. The problem is that by manually instantiating the Controller we're bypassing Springs Dispatcher Servlet, and as a result missing out on core features like
request URL mapping request deserialization re…








Read more










Axis2 Web Service Client Tutorial




















Image







Introduction

Axis2 is a Java framework that provides comprehensive support for exposing and consuming web services. This short post will look at its SOAP client support and how it can be used to get a simple web service client up and running.

For convenience I'm going to be calling a web service that I recently built as part of another blog post.  If you don't already have a web service to call you can grab the full source code for my sample service from github. Simply run a Maven build and deploy the WAR to your Servlet container.

What is a Web Service Client

This post doesn't attempt to explain the detailed inner workings of a web service client, but its still pretty useful to have an idea of what's going on under the hood. Most web service clients provide the following
A client side proxy for the remote service we want to call, that allows our application to invoke a SOAP service using a simple method call. The proxy insulates our application from the intricacies of s…








Read more










An Introduction to Wiremock 





















This post provides a brief introduction to Wiremock, showing how it can be used to to quickly and easily mock remote API calls. We'll use Wiremock to write some integration tests for a simple Dropwizard app and show you how it can be put to use in a real world scenario.

Why would I need to mock external API calls?
There are a number of scenarios where it makes sense to mock an external API rather than call a live service.
The external API may still be in development and not yet available for integration. In this instance as long as a data contract has been defined (e.g. Swagger spec, WSDL), the remote API can be stubbed based on the data contract. Stubbed endpoints allows a team to continue development even when external APIs isn't fully implemented.You may have little or no control over the external API uptime in development or test.  As a result you cannot guarantee it will be available to call when running integration tests. In this instance it makes sense to use mocked respo…








Read more










Spring Batch Tutorial




















Image







Introduction
Spring batch is a lightweight framework that provides a solid foundation on which to build robust and scalable batch applications. It provides developers with a set of tried and tested patterns that solve common batch problems and allows developers to focus more on the business requirement and less on complex batch infrastructure. Spring batch contains a variety of out of the box configurable components that can be used to satisfy many of the most common batch use cases. Extensive XML configuration and an extensible programming model mean that these components can be customised and used as building blocks to quickly deliver common batch functionality.

This tutorial will show you how to build a very simple batch application to read fixed length data from a flat file and write it to a database table. This is a common batch use case and should be sufficient to demonstrate some of the fundamental concepts of Spring batch and provide you with a foundation on which to build mor…








Read more










Externalising Spring Configuration





















Enterprise applications typically contain a number of  properties and  runtime resources that must be configured before the application is deployed. Property and resource values can either be baked into the artefact at build time using something like Maven Profiles or derived at runtime using Spring Profiles. My preferred approach is to externalise all configuration so that the artefact taken from the build server is environment agnostic and can be deployed onto any environment . For example, we could take a WAR file from our build server and  deploy it to our development, UAT or production environments without changing any configuration in the WAR file itself. 

Spring Property Configuration
Spring applications use the PropertyPlaceholderConfigurer to load application properties. Property files are typically loaded from the class path but the example below uses a slightly different approach, loading the properties file from a directory outside of the application itself. This allows the…








Read more











Docker & Spring Boot





















Docker allows you to package an application with its dependencies, into a light weight, portable container that can run on almost any environment.  You can think of a Docker container as a run time, a mini virtual machine that encapsulates your application and its dependencies.

In order to run a container you need a Docker image. An image is like a template that defines everything that will exist within the container. You can almost think of an container as a run time instance of the image it was created from. In this post we'll define and build 3 slightly different Docker images that run a simple Java app.

 Installing Docker
If you haven't already done so you'll need to install Docker. The official documentation is pretty good so following it step by step should see you up and running in about 15 minutes.

I've installed Docker on Windows and Ubuntu but to be honest I prefer running it on Ubuntu and have found it a bit more reliable than with Docker Toolbox on Windows.��…








Read more










Spring Boot & Amazon Web Services (EC2, RDS & S3)




















Image







This post will take you through a step by step guide to building and deploying a simple Java application to the AWS cloud platform. The application will use a few well known AWS services which I'll describe along the way. There is quite a bit of material to cover in this post so the overview of the AWS services will be light. For those interested in finding out more I'll link to the appropriate section of the AWS documentation. Amazon have done a fine job documenting their platform so I'd encourage you to have a read if time permits.       

Prerequisites
In order to get the sample application up and running you'll need access to AWS. If you don't already have access you can register for a free account which includes access to a bunch of great services and some pretty generous allowances. I'd encourage you to get an account set up now before going any further.

What will the sample application look like?
The application we're going to build is very simple custo…








Read more