Automated Deployment – Another way we increase reliability

Posted by:

I’ve talked about the automated testing process that we use before.  But I’ve never talked about the automated deployment process that we use.

Deployments in software are just what they sound like. They are when you take the code changes you’ve been working on, and deploy them somewhere– normally in a production environment.

Deploying to a production environment is a risk. Anytime you change a production environment, you risk causing problems for your users, maybe affecting your availability (link) or any number of other things that can happen.

Deploying is Risky, but it’s essential to a solid and evolving product

But deploying is also how you get fixes & new features to your users.

So, the more often you can deploy, the better – as long as you can be very confident that your deployments are done in a safe manner and won’t cause any degradation of service.

Like most other things we do we use as much automation as possible. The less hands involved in the process, the less chance that human error gets involved. I’m pretty much the walking example of human error!

A day in the life of our deployment process

We don’t want to break things–we like to test them first. 

It all starts when a developer makes some changes and commits those changes to our source code repository.  When that happens our deployment engine is automatically notified.

Our deployment engine then wakes up and asks the source code repository for all of the changes. Afterwards, it takes those changes and build out a fresh environment to test against.

In the fresh environment it goes through a whole slew of tests! First, we run our automated test library that we talked about before- security analysis, bug analysis. We also ensure our foreign language translations are current & complete, and that our database and other components are all in proper order. We don’t just analyze things in a static-state but also exercise those interactions. If all of those tests pass the next step is to deploy to our staging environment.

On to staging environment

Our staging environment is an exact copy of our production environment. We use it for testing of major changes and new features to be sure we don’t break anything (our dev environments may not be identical to the exactness that staging is.) We have a pile of deployment scripts that automatically prepare the staging environment and deploy the changes to that environment.

After the changes are deployed, more scripts ensure the site is still up and accessible.

And now, for the production environment!

If we’re doing a production deployment, the next step after a successful staging deployment is a production one.  We don’t deploy directly to our running production environment – that would cause downtime or affect our users.  Instead the deployment scripts create a new identical copy of the virtual machines we run on that matches our current production environment. Those scripts then deploy the code changes to those servers.  Then they run a health check on that new version of production.

If all is well we switch the two versions of production seamlessly so the next time our users click on a button or a link in the application they get the updated version.

We again run a health check on the now public facing production to ensure all is well and our extensive monitoring is always watching.  A short 15 minutes later the idle second copy of our production environment is automatically spun down so we’re ready to release more changes.

This is a complex process with a lot of moving parts. Thanks to automation the entire process that would have otherwise taken days in the past now happens in about 15-20 minutes.

0

Add a Comment