I've been working with Spring Boot for a few years now and I'm a big fan. There are lots of things to like about Spring Boot, but one thing that really stands out for me is the emphasis on production readiness.
Production Readiness
Production readiness is about looking beyond functional requirements and ensuring your application can be properly managed and monitored in production. Some key things to consider when thinking about production readiness are...
- Health checks
- Viewing application configuration - application properties, Environment variables etc.
- Viewing and altering log configuration
- Viewing application metrics - JVM, classloader, threading and garbage collection.
- Audibility of key application events
Spring provides all of this functionality out of the box via Spring Boot Actuator, a sub project of Spring Boot. A range of RESTful management and monitoring endpoints are provided so you don't have to implement these features each time you build an application. To enable these endpoints, simply add the Actuator starter POM to your project and you're ready to go. The sections that follow will describe some of the key Actuator endpoints and how they can be configured and used. Actuator exposes quite a few endpoints so I'm going to focus on the ones I think are most useful.
Endpoint Security
By default most Actuator endpoints are considered sensitive. Only the
health and
info endpoints are non sensitive (by default) and can be accessed without authentication. All other endpoints must be accessed using HTTP basic authentication. The sensitivity of individual endpoints can be configured in
application.properties. For example, you could change the default security setting and make the
metrics endpoint accessible without basic authentication.
|
| endpoints.metrics.sensitive=false
|
Its also possible to disable security on all Actuator endpoints as follows.
|
| management.security.enabled=false
|
Actuator endpoints expose lots of data about the internals of your application so you need to be
very careful when disabling security. If your application sits behind a firewall it may be acceptable to disable security, but if the application is public facing then securing the Actuator endpoints is an absolute must. My personal opinion would be to use basic auth in all cases. Even if you're application is sitting behind a firewall it does no harm at all to have an extra layer of security in place.
Health Checks
A health check is an endpoint that returns a message describing the health of an application. An application is considered healthy if its capable of successfully servicing client requests. To do that its core components or dependencies should be up and running. Consider an application that interacts with a database and a JMS message broker.
The health check for such an application might consist of a simple query against the database and creating a connection with the JMS broker. By proving these core components are up and running we can say with a fair degree of confidence that our application is healthy.
Actuator provides a health check endpoint out of the box, which can be accessed at
/health using a simple HTTP GET. Using
curl for example we can call the health check endpoint as follows.
|
| curl --user user:password localhost:8080/health
|
The JSON response below is made up of
status,
diskSpace and
db. Status describes the overall health of the application, in this case
UP indicates that the application is healthy.
diskSpace describes total disk space, remaining disk space and a minimum threshold. The threshold value is configurable in application.properties using
management.health.diskspace.threshold. If this value is greater than the
free value, the status of the
diskSpace check will be
DOWN.
db describes the health of the database and is derived by running a simple query. By default Boot will look up the validation query for the specific database engine, for example on Oracle the validation query will be
select 1 from dual. You can also override the default query by writing your own health check implementation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| {
"status": "UP",
"diskSpace": {
"status": "UP",
"total": 1983077322752,
"free": 1732231667712,
"threshold": 10485760
},
"db": {
"status": "UP",
"database": "HSQL Database Engine",
"hello": 1
}
}
|
When the status
UP is returned, a HTTP 200 response code is used. If the health check returns
DOWN the response code is HTTP 503 (Service Unavailable). This is useful for load balancers or monitoring tools that may be calling the health check. They can derive the health status using the HTTP response code rather than the response message, which can be simply logged for information. If the format of the response message happens to change it wont break the tools that are calling the health endpoint.
Automatic Health Checks
The sample response above consists of 2 separate checks, disk space and database. Spring Boot automatically detects components used by an application and runs the appropriate health checks. In the
sample app it detects a
DataSource and therefore runs the database health check. If a
JmsConnectionFactory was configured in the application, it would also run a JMS health check. As well as the examples shown here, Actuator also provides automatic health checks for SMTP, Mongo, Redis, ElasticSearch, RabbitMQ and more. If you'd prefer to disable these automatic health checks you can do so by setting
management.health.defaults.enabled=false in application.properties.
Health Check Security
By default when the health check is called without basic authentication, only the status (
UP or
DOWN) is returned in the response body. This ensures that sensitive information such as the name of the database engine aren't exposed to the world. When basic authentication is used a detailed JSON response is returned.
Custom Health Checks
If the standard health checks aren't sufficient and you need something bespoke, you can override the default health check with your own implementation. Simply create a class that implements
HealthIndicator and override the
health method as follows.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
@Component
public class HealthCheck implements HealthIndicator {
@Override
public Health health() {
if(isRemoteServiceUp()){
return Health.up()
.withDetail("remote service", "online")
.build();
}
return Health.down()
.withDetail("remote service", "offline")
.build();
}
private boolean isRemoteServiceUp(){
// perform call out to remote service to check if its up
return true;
}
}
|
In the example above I've created a dummy method
isRemoteServiceUp to check the status of some fictitious remote service. In reality this method would perform a remote call to check that the service is up and the resulting status used to build a health response.
Info Endpoint
The
info endpoint provides general information about your application. The type of information returned can be customised, but one area that is particularly useful is the application versioning information. I've seen situations in the past where there's been confusion around the version of a running application. Being able to check application info on a running instance is a great way to confirm that a deployment has gone as expected and the new version of the application is indeed up and running. The
info endpoint is can be called with
curl as follows
|
| curl --user user:password localhost:8080/info
|
The sample response below includes useful information about the GIT branch and commit that's been deployed. This information is read from your local Git configuration using the git commit plugin.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| {
"java": {
"version": "1.8.0_73"
},
"git": {
"commit": {
"time": "2017-08-31T06:28:56.000+0000",
"id": "d0ca400"
},
"branch": "master"
},
"build": {
"version": "0.1.0",
"artifact": "boot-actuator-demo",
"name": "boot-actuator-demo",
"group": "org.briansjavablog",
"time": "2017-09-04T06:47:40.000+0000"
},
"remoteServiceVersion": "1.2.3"
}
|
Simply add the following to the plugins section of your POM and Boot will automatically pick up the branch and commit used to build the artifact.
1
2
3
4
| <plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
</plugin>
|
The
build section is provided by the
boot maven plugin. It derives Maven related information for the artefact at build time.
1
2
3
4
5
6
7
8
9
10
11
| <plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
</plugin>
|
Custom Application Information
You can add custom information to the
info endpoint response via
application.properties. In the sample response above,
java.version is populated using the following configuration in
application.properties. The @...@ notation allows us to access any value from the maven
project.properties at build time.
|
| info.java.version=@java.version@
|
There may be data you want to expose on the
info endpoint that isn't available until runtime. You can do this by creating a class that implements the
InfoContributor interface. This class allows you to add whatever information you want to the
info endpoint. The example below exposes the version of some fictitious remote service.
1
2
3
4
5
6
7
8
9
10
11
12
13
| @Component
public class CustomInfoContributor implements InfoContributor {
@Override
public void contribute(Builder builder) {
// add whatever data you want to expose on info endpoint...
builder.withDetail("remoteServiceVersion", getRemoteServiceVersion());
}
// service would check version of some remote service being used
private String getRemoteServiceVersion(){
return "1.2.3";
}
}
|
Metrics Endpoint
The
metrics endpoint exposes a range of application metrics that are extremely useful when it comes to monitoring and trouble shooting poorly performing applications. Data exposed by the
metrics endpoint includes
- System Memory - Total and free system memory in KB
- System Uptime - Server uptime and application context uptime
- JVM Memory Usage - Initial, used and max available heap and non heap size
- Thread Statistics - total thread count, number of threads started and number of daemon threads
- Classloader - total classes loaded, number currently loaded and number unloaded
- Garbage Collector - GC algorithm run, number of times it has run and time taken for last GC
- HTTP Sessions - Current and peak number of HTTP sessions
- Data Source - total number of connections and total active connections
- Endpoints - number of times each endpoint has been called and the last response time for each endpoint
Using
curl you can access the
metrics endpoint with
|
| curl --user user:password localhost:8080/metrics
|
Most of the data in the sample response is fairly intuitive. One section worth describing further is the use of gauge and counter for endpoint metrics. A gauge is used to record a single value, for example on line 31 gauge.response.metrics: 43.0 shows the last response time in milliseconds for the metrics endpoint. A counter records the increment or decrement of a value, for example on line 36 counter.status.metrics: 6 shows the number of times the metrics endpoint has been called since the app started.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| {
"mem": 702657,
"mem.free": 451758,
"processors": 4,
"instance.uptime": 32291251,
"uptime": 32343883,
"systemload.average": -1.0,
"heap.committed": 601600,
"heap.init": 245760,
"heap.used": 149841,
"heap": 3484672,
"nonheap.committed": 103408,
"nonheap.init": 2496,
"nonheap.used": 101057,
"nonheap": 0,
"threads.peak": 24,
"threads.daemon": 21,
"threads.totalStarted": 30,
"threads": 23,
"classes": 13285,
"classes.loaded": 13285,
"classes.unloaded": 0,
"gc.ps_scavenge.count": 10,
"gc.ps_scavenge.time": 354,
"gc.ps_marksweep.count": 3,
"gc.ps_marksweep.time": 1037,
"httpsessions.max": -1,
"httpsessions.active": 0,
"datasource.primary.active": 0,
"datasource.primary.usage": 0.0,
"gauge.response.metrics": 43.0,
"gauge.response.unmapped": 0.0,
"gauge.response.info": 414.0,
"gauge.response.star-star": 22.0,
"counter.status.200.info": 1,
"counter.status.200.metrics": 6,
"counter.status.404.star-star": 1,
"counter.status.401.unmapped": 1
}
|
Logger Endpoint
The
loggers endpoint exposes a detailed view of your applications log configuration. Using
curl you can access the
loggers endpoint with
|
| curl --user user:password localhost:8080/loggers
|
The ROOT logger and per package configuration is included, as defined in your applications
logback.xml. If a package has no explicit log level defined in your
logback.xml, its
configuredLevel will be null and its
effectiveLevel will be inherited from the ROOT logger.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| {
"levels": ["OFF", "ERROR", "WARN", "INFO", "DEBUG", "TRACE"],
"loggers": {
"ROOT": {
"configuredLevel": "INFO",
"effectiveLevel": "INFO"
},
"com": {
"configuredLevel": null,
"effectiveLevel": "INFO"
},
"com.blog": {
"configuredLevel": null,
"effectiveLevel": "INFO"
},
"com.blog.samples": {
"configuredLevel": null,
"effectiveLevel": "INFO"
},
"com.blog.samples.Application": {
"configuredLevel": null,
"effectiveLevel": "INFO"
}
}
}
|
Env Endpoint
The
env endpoint exposes data from the application
Environment, the object that encapsulates all configuration available to the running application. Using
curl you can access the
env endpoint with
|
| curl --user user:password localhost:8080/env
|
Below is a sample response that includes profiles, server ports, Servlet initialising parameters, system properties, environment variables and application properties.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
| {
"profiles": [prod],
"server.ports": {
"local.server.port": 8080
},
"servletContextInitParams": {},
"systemProperties": {
"java.runtime.name": "Java(TM) SE Runtime Environment",
"sun.boot.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\jre\\bin",
"java.vm.version": "25.73-b02",
"java.vm.specification.name": "Java Virtual Machine Specification",
"user.dir": "C:\\blogs\\samples\\boot-actuator-demo",
"java.runtime.version": "1.8.0_73-b02",
"line.separator": "\r\n",
"java.vm.specification.vendor": "Oracle Corporation",
"user.variant": "",
"os.name": "Windows 10",
"classworlds.conf": "C:/apps/apache-maven-3.3.3/bin/m2.conf",
"sun.jnu.encoding": "Cp1252",
"spring.beaninfo.ignore": "true",
"java.library.path": "C:\\Program Files\\Java\\jdk1.8.0_73\\bin;C:\\WINDOWS\\Sun\\Java\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\Users\\brianh\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\local\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Users\\brianh\\bin;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Program Files\\Broadcom\\Broadcom 802.11 Network Adapter;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files (x86)\\ATI Technologies\\ATI.ACE\\Core-Static;C:\\Program Files (x86)\\Skype\\Phone;C:\\Program Files\\nodejs;C:\\apps\\apache-maven-3.3.3\\bin;C:\\Program Files\\Docker Toolbox;C:\\Users\\brianh\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\brianh\\AppData\\Roaming\\npm;C:\\Program Files\\Git\\usr\\bin\\vendor_perl;C:\\Program Files\\Git\\usr\\bin\\core_perl;.",
"java.specification.name": "Java Platform API Specification",
"java.class.version": "52.0",
"java.vendor": "Oracle Corporation",
"catalina.base": "C:\\Users\\brianh\\AppData\\Local\\Temp\\tomcat.7213012654997851459.8080",
"maven.home": "C:\\apps\\apache-maven-3.3.3",
"file.separator": "\\",
},
"systemEnvironment": {
"TEMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
"HOSTNAME": "DESKTOP-867SP8L",
"HOMEDRIVE": "C:",
"USERPROFILE": "C:\\Users\\brianh",
"TMP": "C:\\Users\\brianh\\AppData\\Local\\Temp",
"CommonProgramFiles(x86)": "C:\\Program Files (x86)\\Common Files",
"NUMBER_OF_PROCESSORS": "4",
"DOCKER_TOOLBOX_INSTALL_PATH": "C:\\Program Files\\Docker Toolbox",
"HOME": "C:\\Users\\brianh"
},
"applicationConfig: [classpath:/application.properties]": {
"spring.jpa.show-sql": "true",
"endpoints.health.sensitive": "false",
"security.user": "user",
"security.user.password": "******",
"info.java.version": "1.8.0_73",
"spring.jpa.database-platform": "org.hibernate.dialect.HSQLDialect"
}
}
|
This information is very useful for checking that an application is using the expected configuration. It's particularly useful when an application is deployed to a test or prod environment and using externalised configuration such as an external properties file or environment variables.
Heap Dump Endpoint
The
heapdump endpoint triggers a heap dump of the application JVM and returns a
hprof file. This can be very useful when trying to get to the bottom of memory or threading issues in your application. Being able to trigger and retrieve the heap dump via HTTP is nice as it saves you having to remote onto the box directly. Its worth mentioning that heap dumps are expensive, so the
heapdump endpoint should be triggered with care in a production environment.
Visualising Actuator Endpoint Data
The actuator endpoints we've looked at in this post are certainly useful, but dealing with raw JSON responses makes some of the information difficult to understand at a glance. What we'd really like is some sort of visual representation of the endpoint data. Luckily the Spring Boot Admin project gives us exactly that, by taking the data returned from the various Actuator endpoints and displaying it in an admin console. The screen shot below shows how Spring Boot Admin displays health and metric data in a nice user friendly format.
 |
| Spring Boot Admin |
In a follow up post I'll show you how to use Spring Boot Admin to display health and monitoring data for your Spring Boot applications.
Wrapping Up
In this post we looked at some of the most useful Spring Boot Actuator endpoints and how they can provide a detailed view into the internals of a running application. The
sample code that accompanies this post is available on Github, so feel free to pull it down and have a play around. As always if you have any questions or comments please leave a note below.
Comments
Post a Comment