Configuring Micro Services - Spring Cloud Config Server

Managing application configuration in a traditional monolith is pretty straight forward. Configuration is typically externalised to one or more property files that sit on the same server as the application. Externalising the configuration makes sense as it allows you to update configuration without having to rebuild and redeploy the application. It also means the build artifact is environment agnostic, allowing you to deploy the same physical WAR to dev, uat, prod etc.

Configuring Micro Services

While the approach described above is valid for many applications, the move toward micro services has created a number of new challenges. Micro services by definition are more granular, meaning many distinct components to deploy and manage. Increasing the number of components creates a maintenance challenge when it comes to managing application config. Consider managing the configuration for a solution that is broken into 8 distinct micro services (which is not many by the way). Updating some common configuration in a development environment could mean logging onto 8 different boxes and amending property files. In a production environment the overhead is even greater, as there will likely be multiple instances of each service. You can see how a simple config change could require a lot of effort.
Manually updating configuration across many servers also means there is a decent chance of human error. A situation where some services have updated configuration and some don't, can lead to all sorts of unpredictable behaviour. 
While micro services can run on premise, most micro service architectures run in the cloud.  Cloud environments tend to be quite fluid with server instances can be brought up and down regularly, as as the result of some auto scaling activity for example. This fluidity makes it very difficult to manually manage application config across multiple services.

Introducing Centralised Configuration

Centralised configuration helps address the challenges mentioned above. Rather than services having their own local configuration, each service can instead retrieve its configuration from a central source. Spring provides a basis for implementing this pattern via Spring Cloud Config, a sub project of Spring Cloud.
Spring Cloud Config allows you to create a Spring Boot application that exposes application properties via a REST API.  Other services can consume their application properties from the REST API rather than loading them from a local source. Property files containing the application configuration are not actually stored in the Cloud Config Server, but are instead pulled from Git.  This allows you to manage your application properties in Git with all the benefits of a source control system. Spring Cloud Config can be configured to use either a local git repository (useful during dev) or a remote repository. In a production environment you'd want the Config Server to access the properties from a remote repository on Github.

Sample Application

In this post I'm going to show you how to setup a Configuration Service and configure it to pull application properties from GitHub. I'll also show you how those properties can be consumed by another service, in this case a simple bank account service.  The diagram below describes the main components involved.

Centralised Configuration Architecture

Creating the Cloud Config Service 

We'll begin by creating a simple Spring Boot app for the Config Service. Inside the main application class add @EnableConfigServer to enable the config server functionality.

@EnableConfigServer
@SpringBootApplication
public class ConfigServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConfigServerApplication.class, args);
    }
}

Configuring the POM

In order to make the required Spring Cloud dependencies available you'll need to add the following dependencies to the POM.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                       http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.briansjavablog.microservices</groupId>
 <artifactId>config-server</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>
 <name>config-server</name>
 <description>Demo config server provides centralised configuration for various micro services</description>
 <parent>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-parent</artifactId>
     <version>2.0.3.RELEASE</version> 
 </parent>
 <properties>
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
     <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
     <java.version>1.8</java.version>
     <spring-cloud.version>Finchley.RELEASE</spring-cloud.version>
 </properties>
 <dependencies>
     <dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-config-server</artifactId>
     </dependency>
     <dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-devtools</artifactId>
  <scope>runtime</scope>
     </dependency> 
 </dependencies>
 <dependencyManagement>
     <dependencies>
  <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring-cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
  </dependency>
     </dependencies>
 </dependencyManagement>
 <build>
     <plugins>
  <plugin>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-maven-plugin</artifactId>
  </plugin>
     </plugins>
 </build>
</project>

Configuring the Config Server

Next you'll need to configure the Config Service via application.properties.

1
2
3
4
5
6
7
8
9
spring.application.name=config-server
server.port=8888

# URI of GIT repo containing properties
spring.cloud.config.server.git.uri=https://github.com/briansjavablog/micro-services-spring-cloud-config
# path to properties from root of repo 
spring.cloud.config.server.git.searchPaths: configuration

logging.level.org.springframework.web=INFO
  • Line 1 - spring.application.name specifies the name of the application. This isn't essential but its good practice to name your Boot applications as this name will appear on the actuator /info endpoint and will be displayed in Spring Boot Admin if you use it.   
  • Line 2 - server.port specifies the port that the admin app will run on, in this instance port 8888.
  • Line 5 - spring.cloud.config.server.git.uri specifies the URL of the remote repository containing the property files that will be served up by the Config Service. In this example I am pointing at my own repository on GitHub, but this can also point to a local Git repo. This is useful for local development and can be specified with the file prefix as follows file:///c:/dev/micor-services/git-local-config-repo.    
  • Line 7 - spring.cloud.config.server.git.searchPaths specifies the path to the properties files from the root of the repository. So in the example above I want to access property files in the configuration directory of the repository. 

Running the Cloud Config Service 

Now that we have the Config Server configured, its time to fire it up and do a quick test. You can start the application in Eclipse or on the command line the same way you'd start any other Boot app.  You should see the application start on port 8888 as follows.
Cloud Config Service Startup Port 8888

Testing the Config Service

You can test the service by calling http://localhost:8888/bank-account-service/default.  This GET request contains the name of the properties and the profile of the properties we want to load. In this instance we are looking for the properties belonging to the bank-account-service and we want the properties associated with the default profile.

When this request is received, the Config Service uses the GIT URI specified in application.properties to perform a git clone of the remote repository. The screenshot below shows the repository being cloned to the a local temp directory and includes the following line.

Adding property source: file:/C:/Users/BRIANS~1/AppData/Local/Temp/config-repo-6545303057095204707/configuration/bank-account-service.properties

Cloud Config Server - Cloning Repository
The properties in bank-account-service.properties are then read by the Config Service and returned to the client in JSON format.

Config Service Response

The following JSON response is returned from the Config Service when we call http://localhost:8888/bank-account-service/default.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
   "name": "bank-account-service",
   "profiles": ["default"],
   "label": null,
   "version": "7b0732778b442726f8dd0bf7d1a36fc00f15c5b8",
   "state": null,
   "propertySources": [{
      "name": "https://github.com/briansjavablog/micro-services-spring-cloud-config/configuration/bank-account-service.properties",
      "source": {
          "bank-account-service.minBalance": "99",
          "bank-account-service.maxBalance": "200"
      }
   }]
}

The name attribute contains the name of the properties we requested. In this instance we requested properties for the bank-account-service. Note that this matches the name of the default properties file in GitHub.

The profiles attribute is the Spring profile used to load the required properties. In this instance we used the default profile, but we can specify any valid profile we want here. If you look at the property files on GitHub you'll see 3 files.
  • bank-account-service.properties - this file contains the default properties and is used as the property source when the default profile is specified on the request.
  • bank-account-service-dev.properties - this file has a '-dev' post fix and contains the properties returned when the dev profile is specified on the request.
  • bank-account-service-uat.properties - this file has a '-uat' post fix and contains the properties returned when the uat profile is specified on the request.
If we specify dev or uat as the profile, the Config Service will return properties from the file matching that profile.

The version attribute is the current commit sha of the properties being returned. If you check GitHub this will match the commit sha of the latest commit. Finally, the propertySources attribute contains the GitHub source URI of the properties being returned along with the actual property values.

Creating a Bank Account Service

Now that the Config Service up and running, its time to put it to work. We'll create a simple Bank Account Service that will call the Config Service on start up to retrieve its properties. The Bank Account Service has two REST endpoints, one to create a bank account and one to retrieve a bank account.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
@RestController
@Slf4j
public class BankAccountController {

    @Autowired
    public BankAccountService bankAccountService; 
 
 
    @PostMapping("/bank-account")
    public ResponseEntity<?> createBankAccount(@RequestBody BankAccount bankAccount, HttpServletRequest request) throws URISyntaxException {
  
        bankAccountService.createBankAccount(bankAccount);
  
        log.info("created bank account {}", bankAccount);
  
        URI uri = new URI(request.getRequestURL() + "bank-account/" + bankAccount.getAccountId());
  
        return ResponseEntity.created(uri).build();    
    }
 
 
    @GetMapping("/bank-account/{accountId}")
    public ResponseEntity<BankAccount> getBankAccount(@PathVariable("accountId") String accountId){
  
        BankAccount account = bankAccountService.retrieveBankAccount(accountId);
  
        log.info("retrieved bank account {}", account);
  
        return ResponseEntity.ok(account);    
    }
 
}

The REST controller uses a simple BankAccountService to create and retrieve bank account details. When creating a new account, the service performs a check to see if the balance of the new account is between a set of minimum and maximum values.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
    /**
     * Add account to cache
     * 
     * @param account
     */
    public void createBankAccount(BankAccount account) {
  
        /* check balance is within allowed limits */
        if(account.getAccountBlance().doubleValue() >= config.getMinBalance() && 
           account.getAccountBlance().doubleValue() <= config.getMaxBalance()) {
   
            log.info("Account balance [{}] is is greater than lower bound [{}] and less than upper bound [{}]", 
                     account.getAccountBlance(), config.getMinBalance(), config.getMaxBalance());
   
            accountCache.put(account.getAccountId(), account);
        }
        else {
   
            log.info("Account balance [{}] is outside of lower bound [{}] and upper bound [{}]", 
                     account.getAccountBlance(), config.getMinBalance(), config.getMaxBalance());
             throw new InvalidAccountBalanceException("Bank Account Balance is outside of allowed thresholds");
        }
    }

Bank Account Service Config

These minimum and maximum values are configurable and will be read from an injected Configuration object.

1
2
3
4
5
6
@Service
@Slf4j
public class BankAccountService {                                                     

    @Autowired
    private Configuration config;

The Configuration object is defined as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
@Component
@ConfigurationProperties(prefix="bank-account-service")
public class Configuration {                                                          
 
    @Setter
    @Getter
    private Double minBalance;
 
    @Setter
    @Getter
    private Double maxBalance;

}

There are two important things to note.
  • the @ConfigurationProperties prefix matches the name of the properties file in GitHub
  • the key for each property in the property file is <fileName>.<propertyName> where the property name is the same as the instance variable name in the @Configuration class.  
A screenshot from GitHub shows the file name and property names corresponding to the values in the @Configuration class.

bank-account-service.properties in GitHub

Bank Account Service Local Configuration

We are going to use the Config Service to retrieve the Bank Account Service Configuration, however there are a few properties which must be set locally.

1
2
3
4
5
6
 spring.application.name=bank-account-service
 server.port=8080

 spring.config.cloud.uri=htp://localhost:8888
 spring.cloud.config.profile=uat
 management.endpoints.web.exposure.include=*

Lines 1 and 2 are standard Boot config and define the application name and the port it will run on. Line 4 defines the URL of the Config Service. This is the URL that the Bank Account Service will call on start up to retrieve the minBalance and maxBalance properties.  Line 5 defines the profile that will be used to call the Config Service. So given the configuration above, the Config Service will be called for the bank-account-service with the uat profile on start up as follows.

http://localhost:8888/bank-account-service/uat.

Testing the Bank Account Service

Start the Bank Account Service on the command line or in Eclipse. In the log you should see a call to the Config Service on http://localhost:8888 using the uat profile as configured in application.properties.   

Calling Config Service on startup

We can now test the application and confirm that it was indeed the uat configuration that was retrieved from the Config Service. Run the following cURL command to create a new bank account.

1
curl -i -H "Content-Type: application/json" -X POST -d '{"accountId":"B12345","accountName":"Joe Bloggs","accountType":"CURRENT_ACCOUNT","accountBlance":1250.38}' localhost:8080/bank-account

Note that we specify an account balance of £1250.38. This should be inside the allowed limits given that the uat properties are defined as follows.

bank account service uat profile properties

The log extract below shows the account object being created successfully and the min and max values being logged as 501.0 and 15002.0 respectively. These values match with those defined in the uat properties in GitHub above.

Updating Service Configuration

One of the main benefits of centralised configuration is that properties can be updated in one place (GIT) and then those changes can be picked up by the Config Service right away.  Every time an application requests properties from the Config Service, the Config Service will check if its locally cloned copy of the remote repository is up to date. If the local copy isn't up to date the Config Service will pull the latest properties from the remote repository and serve them back to the client. This allows applications that are starting up to retrieve the latest properties from the remote repository, but what about updating properties for applications that are already running? 

Thankfully Spring Boot provides a mechanism for reloading application properties in a running application. The reload can be triggered with a HTTP POST to the /refresh actuator endpoint as follows.

curl localhost:8080/actuator/refresh -d {} -H "Content-Type: application/json" 

The screenshot below shows the /refresh endpoint being called with cURL. In the application log you can see the call to the Config Service to retrieve the latest properties for the uat profile. 

property refresh via  actuator endpoint

Automating Service Configuration Updates

Manually invoking the /refresh endpoint is fine in this simple example, but in a production environment we'd have multiple instances of each service. After pushing config changes to GitHub it would be tedious if we had to grab the IP of each service and call its /refresh endpoint manually. 
We could easily create a script to poll the properties repository for changes and then use some kind of service discovery mechanism to get all registered service instances and call their /refresh endpoint. 
Some simple enough scripting used in conjunction with a centralised Config Service should allow you to push configuration changes out across your micro services with minimal effort.  

Wrapping Up

In this post we looked at Spring Cloud Config and how it can be used to create a centralised configuration service that leverages GitHub as the property repository.  We also saw how properties can be specified for different environments using profiles and how these properties can be retrieved by a simple micro service. Finally we looked at how property changes can be applied to running services. The full source code for this post is available on GitHub so feel free to pull it down and experiment. If you have any questions or suggestions please leave a comment below. 

Comments

Post a Comment

Popular posts from this blog

Spring Web Services Tutorial

Image
Update
I've recently written a more up to date version of this post which describes how to build a contract first web service using Apache CXF. Check it out here.

Introduction
Modern enterprise applications are rarely stand alone and often rely on data and services provided by external systems. In order for different types of systems to communicate there must be a  communication protocol of some sort, a standard way of sending and receiving messages in a format that is recognised and supported by all major platforms. SOAP (Simple Object Application Protocol) is such a protocol, and allows applications to communicate by exchanging messages in a standard XML format.
SOAP Web Services provide a platform agnostic integration mechanism that allows disparate systems to exchange data regardless of the platform they are running on. For example, SOAP web services are commonly used to integrate .NET applications with applications running on the Java platform. Almost all modern platforms and …
Read more

REST Endpoint Testing With MockMvc

In this post I'm going to show you how to test a Spring MVC Rest endpoint without deploying your application to a server. In the past, full integration tests were the only meaningful way to test a Spring REST endpoint. This involved spinning up a test server like Tomcat or Jetty, deploying the application, calling the test endpoint, running some assertions and then terminating the server. While this is an effective way to test an endpoint, it isn't particularly fast. We're forced to wait while the entire application is stood up, just to test a single endpoint.
An alternative approach is to unit test by manually instantiating the Controller and mocking out any required dependencies.  While such a test will run much faster than a full integration test, its of limited value. The problem is that by manually instantiating the Controller we're bypassing Springs Dispatcher Servlet, and as a result missing out on core features like
request URL mapping request deserialization re…
Read more

An Introduction to Wiremock

This post provides a brief introduction to Wiremock, showing how it can be used to to quickly and easily mock remote API calls. We'll use Wiremock to write some integration tests for a simple Dropwizard app and show you how it can be put to use in a real world scenario.

Why would I need to mock external API calls? There are a number of scenarios where it makes sense to mock an external API rather than call a live service.
The external API may still be in development and not yet available for integration. In this instance as long as a data contract has been defined (e.g. Swagger spec, WSDL), the remote API can be stubbed based on the data contract. Stubbed endpoints allows a team to continue development even when external APIs isn't fully implemented.You may have little or no control over the external API uptime in development or test.  As a result you cannot guarantee it will be available to call when running integration tests. In this instance it makes sense to use mocked respo…
Read more

Spring Batch Tutorial

Image
Introduction Spring batch is a lightweight framework that provides a solid foundation on which to build robust and scalable batch applications. It provides developers with a set of tried and tested patterns that solve common batch problems and allows developers to focus more on the business requirement and less on complex batch infrastructure. Spring batch contains a variety of out of the box configurable components that can be used to satisfy many of the most common batch use cases. Extensive XML configuration and an extensible programming model mean that these components can be customised and used as building blocks to quickly deliver common batch functionality.
This tutorial will show you how to build a very simple batch application to read fixed length data from a flat file and write it to a database table. This is a common batch use case and should be sufficient to demonstrate some of the fundamental concepts of Spring batch and provide you with a foundation on which to build mor…
Read more

Externalising Spring Configuration

Enterprise applications typically contain a number of  properties and  runtime resources that must be configured before the application is deployed. Property and resource values can either be baked into the artefact at build time using something like Maven Profiles or derived at runtime using Spring Profiles. My preferred approach is to externalise all configuration so that the artefact taken from the build server is environment agnostic and can be deployed onto any environment . For example, we could take a WAR file from our build server and  deploy it to our development, UAT or production environments without changing any configuration in the WAR file itself.

Spring Property Configuration Spring applications use the PropertyPlaceholderConfigurer to load application properties. Property files are typically loaded from the class path but the example below uses a slightly different approach, loading the properties file from a directory outside of the application itself. This allows the…
Read more

Docker & Spring Boot

Docker allows you to package an application with its dependencies, into a light weight, portable container that can run on almost any environment.  You can think of a Docker container as a run time, a mini virtual machine that encapsulates your application and its dependencies.
In order to run a container you need a Docker image. An image is like a template that defines everything that will exist within the container. You can almost think of an container as a run time instance of the image it was created from. In this post we'll define and build 3 slightly different Docker images that run a simple Java app.

Installing Docker If you haven't already done so you'll need to install Docker. The official documentation is pretty good so following it step by step should see you up and running in about 15 minutes.
I've installed Docker on Windows and Ubuntu but to be honest I prefer running it on Ubuntu and have found it a bit more reliable than with Docker Toolbox on Windows.��…
Read more

jQuery Tables

Introduction
This post will show you how to create rich data grids using the DataTables plugin for jQuery. DataTables is a flexible framework that allows you to create grids using a variety of data sources. This tutorial will show you how DataTables can be used with DOM and JavaScript data sources.

DOM Data Source
The simplest way to use DataTables is with a DOM data source. This approach simply 'decorates' an existing HTML table and is ideal for enhancing plain HTML tables created using template languages like JSP or Velocity.

MakeModelBody ShapeEngine Size (Cubic Capacity)Horse PowerBMW320dSaloon2000181BMW330dCoupe3000231BMW335dEstate3000286BMW335iSaloon3000306BMWM3Coupe4000415BMW520dSaloon2000181BMW530dSaloon3000231BMW535dEstate3000286BMW535iSaloon4000306BMWM5Saloon5000508
To turn this rather dull looking HTML table into a rich data grid we simply decorate the table on document load.
1: <script type="text/javascript"> 2: $(document).ready(func…
Read more

Health Checks, Metrics & More with Spring Boot Actuator

Image
I've been working with Spring Boot for a few years now and I'm a big fan. There are lots of things to like about Spring Boot, but one thing that really stands out for me is the emphasis on production readiness.
Production Readiness Production readiness is about looking beyond functional requirements and ensuring your application can be properly managed and monitored in production. Some key things to consider when thinking about production readiness are...
Health checksViewing application configuration - application properties, Environment variables etc. Viewing and altering log configurationViewing application metrics - JVM, classloader, threading and garbage collection. Audibility of key application events Spring provides all of this functionality out of the box via Spring Boot Actuator, a sub project of Spring Boot. A range of RESTful management and monitoring endpoints are provided so you don't have to implement these features each time you build an application. To enable…
Read more

Axis2 Web Service Client Tutorial

Image
Introduction
Axis2 is a Java framework that provides comprehensive support for exposing and consuming web services. This short post will look at its SOAP client support and how it can be used to get a simple web service client up and running.
For convenience I'm going to be calling a web service that I recently built as part of another blog post.  If you don't already have a web service to call you can grab the full source code for my sample service from github. Simply run a Maven build and deploy the WAR to your Servlet container.

What is a Web Service Client
This post doesn't attempt to explain the detailed inner workings of a web service client, but its still pretty useful to have an idea of what's going on under the hood. Most web service clients provide the following
A client side proxy for the remote service we want to call, that allows our application to invoke a SOAP service using a simple method call. The proxy insulates our application from the intricacies of s…
Read more