After working through the release process a few times, and messing it up a time or two, it was clear that our release process at NativeX needed some automation love. Automation is one of the main tenants of Continuous Delivery, a discipline in which each step of the deployment process is automated, allowing code to be deployed to production as quickly and efficiently as possible while maintaining quality. Essentially what this means is that an engineer checks in code and the code gets to production with no manual steps. This doesn’t mean the code is deployed blindly or recklessly, the goal of implementing continuous delivery is to do all of the stuff you normally do during a deployment that ensures quality, but automate each step so it just happens. This requires change in process and culture to work, but it’s worth it for the benefits.
|Tip: Don’t Try To Boil the Ocean. A good approach to getting going with continuous delivery is to start small and not try to boil the ocean by automating everything out of the gate. Take the one or two steps from the deployment process that are the most painful or the most repetitive and start there. Even if you automate one step or reduce risk in any way you are better off than you were before and one step closer to continuous delivery.|
The first step we took was to look at the current deployment process and review each step. The NativeX process had 40 manual steps that included sending notifications, backing up code, moving code, merging code, deploying database schema changes, etc. Out of the 40 manual steps we decided we wanted to be able to automate the following: Build code, run unit tests, run automated tests (Soap UI), backup code, take servers off-line, move code, merge code, and send notifications (release notes). There were other processes that we identified as “automatable” but decided to leave these less trivial items (such as database deployments) for a later phase so that we could realize the benefit of automation sooner than later.
In order to achieve automation we either needed to roll our own system or purchase a tool. We use TFS in-house already but TFS is very Microsoft specific and can be cumbersome to configure (modifying workflow templates + C# code). Plus, NativeX is not solely a Microsoft shop so using something other than TFS as our delivery automation tool just made sense. We did a proof-of-concept on a few different tools such as Thoughtworks Go, Hudson, and Red Gate’s Deployment Manager. After much trial and tribulation we decided that using Atlassian’s Bamboo was the most cost effective approach. The tool didn’t do everything we needed out of the box but it had the best interface and functionality of all the tools plus it was very extensible so we could add on functionality if we needed to. For example, the team was able to build a Bamboo plug-in that would talk to TFS in order to automate merges from one branch to another. Being able to overcome the challenge of making a Java app talk to a Microsoft app like TFS gave us the confidence that we could get Bamboo to do pretty much whatever we needed.
After deciding on the tool and completing the proof-of concept, it was time to implement. We purchased a license, completed installation and started working on a pilot to automate the deployment of our Mobile API. The process was implemented as follows:
- An engineer commits code; Bamboo recognizes this and starts the delivery process.
- Bamboo pulls all source code to a repository and runs MS build.
- If successful, Bamboo then executes all MS unit tests within each project included.
- If the unit tests pass, Bamboo then deploys to the development server and executes a series of automated tests against the newly deployed API.
- If the automated tests all pass, then Bamboo prepares the build for production, sends a notification alerting the engineer that the build is ready for production and then pauses for manual intervention. The point of this pause is to allow the engineer time to test, notify, etc. before triggering the deployment. Note: At some point we intend to remove the manual gate in step #4 to allow things to flow to production automatically if all stages of the delivery process pass with a green light.
Since we proved out what we wanted to automate during the proof-of-concept, implementing the above process was fairly trivial. Nevertheless, we did run into a small challenge in getting our Soap UI tests to run as part of the build / delivery process. We ended up having to refactor the tests so that they could be managed by the engineering teams as well as structuring them in a way that they could be run in a parallel fashion (in order to stay as lean as possible). Finally, we had to create a mechanism for pulling the latest set of tests to Bamboo, running them, and then making a decision based on the results of the tests. The solution consists of using Ant to pull the SoapUI dependencies from a Maven repository, using Ant to execute the SoapUI tests, and then using TestNG to render the results.
As noted above, implementing continuous delivery did require some changes in process and the way we think about deployments. A very typical SDLC consists of an engineer checking in some code and then having a QA engineer test it before deploying it to production. With continuous delivery we moved the testing work in front of the check-in. Leaving the QA process in the middle of the delivery pipeline would slow down the process while testing occurred which goes against what continuous delivery is about. To facilitate upfront testing we create a shelveset or a feature branch to test on and write automated tests against. Once the team has tested everything, the code and the automated tests are checked in together and the deployment process is triggered. Since the automated tests verify that the old code and the new code are working as expected, we can have confidence in clicking the button that deploys the code to all the servers.
Another change in process is that we now wrap new code in feature flags. A feature flag is a toggle that is implemented in code that allows a feature to be turned on or off via a user interface. This allows half-baked code to be wrapped in a feature flag, checked in, and deployed to production without really affecting anything in production. And, the automated tests that are part of the deployment process will prove it!
Implementing continuous delivery is an important step to becoming a truly agile software shop. It allows teams to move faster with less risk. Engineers can now check in code at any time knowing that a feature flag or automated tests will ensure their changes have no negative impact on production. Anyone can deploy to production at any time with the click of a button so there are no more hot fixes. Just implement the fix on the main line, check it in, and let it rip to production. Testing happens earlier in the process so there is a tendency to pay more attention to it and work as a team to make sure quality is high.
In the end continuous delivery saved engineers 2-4 hours per deployment by reducing our 40 manual tasks per deployment down to 14. And, it reduced risk by illuminating manual tasks that could be missed or done incorrectly by a human. Implementing continuous delivery for our Mobile API was definitely a huge win at NativeX and we plan on spreading continuous delivery to other projects and areas of the business in the very near future!