Cloud Configuration Ecosystem at Intuit
Configuration management at Intuit has been reshaped over the last 18 months since the adoption of Spring Cloud Config Server. This work represents a breakthrough in configuration management practices that are changing how Intuit implements configuration management since the company’s inception over 20+ years ago. In essence, any application ranging from desktop and service monoliths started their migration to the cloud without breaking their own DNA: configuration was still part of the binary built on Continuous Integration to be deployed in different data centers. As a consequence, we were still facing the same old challenges: what happens when a new configuration change is required for the entire fleet on multiple private data centers and the cloud? The new answer lies in the adoption of the Spring Cloud Config Server as our One Intuit Configuration Service using the SaaS model, which represents a new shift from manual Operational changes to the simple Pull Requests on related Github Enterprise repositories.
Needless to say, ranging from small internal services to the giants of TurboTax and Quickbooks that are used by millions of users worldwide, there are amazing results with the adoption of this Configuration practice and service such as the decreased time to change configuration from hours to minutes without involving Operations team while getting consistent configuration across a fleet of services. On the other hand, the strong adoption rate brought up a set of new challenges for us to support this new approach in the Enterprise: how to properly architect Spring Cloud Config to be deployed as a SaaS application in the Enterprise? how can we guarantee that users are pushing valid configuration properties to their repo? How can we help them debug their properties consistently, but without relying solely on Github Pull Requests? Finally, what if we need to replicate this solution for Mobile clients? Do we need to deploy hundreds of Configuration servers in the Cloud, and consequently, take the bite on cost?
Overall, the solutions to the questions above are comprised of SaaS deployment of the Spring Cloud Config with some enterprise tweaks for security and performance. Then, we have created a Github Pre-receive hook called Spring Cloud Config Validator to validate user’s config repositories and a web application called Spring Cloud Config Inspector that helps users debug their config keys as associated values, secrets, etc. Lastly, our Spring Cloud Config Publisher solution allows users to use their applications to console a subset of their config properties from an Amazon S3 bucket that the publisher will be publishing to at every new valid commit.
First, and foremost, deploying “Spring Cloud Config as a SaaS” application required us to implement ACL mechanisms that can guarantee isolation and access control to a given application name and GitHub repo name, using our own definition of service authorization keys. Next, we implemented a couple of enhancements in our fork of Spring Cloud Config Client with not only the ACL enhancements, but also a mechanism to store the client’s currently loaded state into an Amazon S3 bucket and, thus, providing a fault tolerant capability for customers in the Cloud. Finally, we have contributed the lazy loading of configurations based on the HTTP Cache protocol namely e-tags, which results in less payload served from GitHub when the state of a given client is known.
Among the different solutions to solve problems around users pushing invalid configuration properties to the Github repo, we implemented a Dockerized Github Enterprise’s Pre-Receive hook that can be used in any Github Enterprise appliance. In fact, Github Pre-Receive hooks are executed prior to commits being merged in the GitHub repo they are being pushed to. As a consequence, we got not only the simplest way to guarantee that the new set of configurations is syntactically valid but also the most effective one that does not require users to change their local GitHub repo hooks structure. Our “Spring Cloud Config Validator” validates each of the supported file format based on the extension, namely such .yml, .json and .properties.
With proper configuration properties submitted to Github, our users that are not necessarily Spring Framework users needed a way to understand how each property value is evaluated. That is, they reported that they were spending a significant amount of time debugging the actual configuration state deployed on their applications. For instance, what key was loaded before or after one of the overrides such as environment variables, bootstrap files, and GitHub config repositories. Based on a feature request from one team within TurboTax, our team worked on a web application that takes the config URL, application name, profiles, and label as input and loads, interprets and helps users debug their application configuration. In short, our web application solution called “Spring Cloud Config Inspector” integrates with both Spring Cloud Config and Github Enterprise APIs to show why a given key has a specific value in terms of the precedence order in which config files are loaded by the Config server, providing navigation features such as filtering keys and secrets values, as well as allowing users to diff between different labels and profiles. With this solution, we have helped our users significantly improve their productivity when it comes to the hardest times of all: debug configuration changes and view exactly what has been deployed to servers.
Last, but not least, one of the most expensive requirements, but the one that saved the most amount of money, was to provide the same solution to teams that are running mobile applications at global scale. In order to do provide Global access without spending money to deploy a farm of Config servers, we have implemented the “Spring Cloud Config Publisher”, which pushes a subset of a user-defined set of config application properties to a user-provided Amazon S3 bucket. In fact, our Mobile applications can implement fetching the files from either a regular Spring Cloud Config server or from the Amazon S3 bucket since we recreated the file objects in the S3 bucket following the same structure as the Spring Cloud Config API. As a result, we mitigated the risk of overwhelming our Github Enterprise appliance from millions of connections from mobile applications at global scale. In the end, we provide customers the choice of selecting which CDN they want to use, ranging from Amazon CloudFront to Akamai, which can front an Amazon S3 bucket.
Although some of the SaaS solutions were primarily implemented for Internal use at Intuit, we have decided to open source the Config Validator, Inspector, and Publisher solutions to the broader community. We would like to work with Pivotal and the Spring Cloud community to help shape those and other integrations with the overall Spring Cloud ecosystem, helping others along the way.
Marcello de Sales
Staff Software Engineer