Configuration Management
When working with large-scale software systems, configuration management becomes crucial. The supporting non-uniform environments get greatly simplified if you decouple code from the configuration. While building complex software/products such as Flickr, we had to come up with a simple yet powerful way to manage configuration. Our new solution combines both the extremely popular GitHub and cfg4j library which will work with applications of any size.
Overview of configuration management
Configuration repository and editor
- Easy to read and write
- It should add a new configuration set
- Able to review changes if your team is bigger than one person
- Must see a history of changes
- Multi-tenancy support
Push cache
The main role of push cache is to decrease load put on the GitHub server and improve configuration fetch time. Since speed is the only concern here, we decided to keep the push cache simple - it’s just a key-value store. Consul was our choice: the nice thing is that it’s fully distributed.
You can install Consul clients on the edge nodes and they will keep being synchronized across the fleet. This greatly improves both reliability and performance of the system. If performance is not a concern, any key-value store will do. You can skip using push cache altogether and connect directly to Github, which comes in handy during development
CD Pipeline
When the configuration repository is updated, a CD pipeline kicks in. This fetches configuration, converts it into a more optimized format and pushes it to the cache. Additionally, the CD pipeline validates the configuration (once at the pull-request stage and again after being merged to master) and controls multi-phase deployment by deploying config change to only 20% of production hosts at one time.
Bootstrap configuration
Before we can connect to the push cache to fetch configuration we need to know where it is. That’s where bootstrap configuration comes into play - it’s very simple. The config contains the hostname, port to connect to, and the name of the environment to use. You need to put this config with your code or as part of the CD pipeline. This simple yaml file binding Spring profiles to different Consul hosts suffices for our needs.
Configuration library
The configuration library takes care of fetching the configuration from push cache and exposing it to your business logic. We use the library called cfg4j (“configuration for java”). This library re-loads configurations from the push cache every few seconds and injects them into configuration objects that our code uses. It also takes care of local caching, merging properties from different repositories, and falling back to user-provided defaults when necessary.
Configurable code
The most important piece is business logic. To best make use of a configuration service, the business logic has to be able to re-configure itself in runtime. Here are a few rules of thumb and code samples:
- Use dependency injection for injecting configuration.
- Use configuration objects to inject configuration instead of providing configuration.
