CI is a Reality and not Just for Google and Facebook

I’ve worked for many recognizable names and they all have time and energy invested in a software process that only shows results once a quarter. Hundreds of thousands or millions of dollars and you can only see the results once ever 3 months… maybe. Each time the releases have been painful, long, and fraught with errors and frustration. To alleviate the pain of deployment, Continuous Integration and Continuous Delivery are the answer. When ever the idea of a faster release schedule is introduced, usually by a business actor, most IT organizations push back. Typically the reasons are “What we did worked last time so we don’t want to change.”,  “The tools are too expensive”, and “We are not google or facebook”. As I will demonstrate the second argument is no longer valid. Using a set of free open source tools Continuous Integration and Continuous Delivery are easily achievable with some time and effort.

Example

The linked presentation and video example are a demonstration open source CI.

MS PowerPoint (DevOps_Opensource_CI_and_the_Cloud.pptx)

LibreOffice Impress (DevOps_Opensource_CI_and_the_Cloud.odp)

Youtube (https://www.youtube.com/watch?v=gIxCcJAl86M)

MP4 (https://s3.amazonaws.com/aws_wordpress/Coveros+Puppet+and+CI-voiceover.mp4)

The Scenario

In order to backup my statement I needed to prove it out with a non-trivial example. For the purposes of this demonstration I chose to show changes in a drupal site. The work flow could be characterized as follows:

1. Developer creates tests
2. Developer creates code
3. Developer commits code and tests
4. CI Framework detects changes
5. CI Framework stands up empty VMs and databases
6. CI Framework pushes down data, code, and configuration to the blank VMs
7. CI Framework runs automated tests on the site and returns results

This process is completely hands off from the developer commit to the test results. No ops group needs to be contacted, no changes need to be filed, no e-mails need to be sent to the QA group alerting them to changes. The drupal instance is a multi-box, separate DB and Webserver, the example is not just a out of the box apache web server with the “it works” page.

The Tool Set

The software you use will vary from implementation to implementation, the software listed in this section is just what was needed for the demo. One size does not fit all, however, with the enormous amount of open source projects available there are basic tools for almost any need. Jenkins and Puppet are really the center pieces in this particular demonstration. Jenkins is the control point and executes the jobs and coordinates everything. Puppet is the CM database holding the necessary information to configure a VM and each application on each VM. PERL and sed are used for text manipulation, taking the output from the EC2 API or the puppet REST interface and turning the output into a usable format for Jenkins. EC2 itself is the infrastructure and elastic capacity for testing.

Path to CI and DevOps

Two words, “small steps”. Depending on the development organization you are working in or with trying to do this all at once it risky. It is easier to lay out a road map of implementing each individual technology and then integrating them. Start with OS packaging. Even if your deployment process is manual, developers and ops people can usually latch on to this idea. The two most challenging pieces are typically automated testing and hands off deployments. One requires a change in the way the testing organization works and the other may require a change to policy and personnel of the operations group. However, as more and more CI pieces get implemented, organizationally these changes will make more sense. In the end the inertia built up by increased efficiency and better releases may be the best leverage.

Conclusion

The three arguments against CI are typically cost, what we did worked last time, and while it works for other organizations it can’t work for us. From the demo you can see that CI can work and can be implemented with opens source tools negating the cost argument. While you have to invest in some (not alot) engineering effort, the cost of the tools themselves should be close to zero. Secondly, if you release cycle is more than 2 weeks – 3 weeks from written idea to implementation your process does not work. While there may be various IT heroes that expend monumental effort to make the product function, your development process does not work in a healthy sense negating the “what we do works” argument. Lastly is the argument that since “we” don’t have the resources of google or facebook we can’t possibly adopt a technology this complicated. The demo above was done by one person in about a weeks worth of effort. The tools are available and they use standard technologies. All the pieces exist and are well tested so you don’t need an engineering staff the size of google to implement CI, refuting some of the most common arguments against CI and DevOps.