DevOps, Opensource, CI, and the Cloud

CI is a Reality and not Just for Google and Facebook

I’ve worked for many recognizable names and they all have time and energy invested in a software process that only shows results once a quarter. Hundreds of thousands or millions of dollars and you can only see the results once ever 3 months… maybe. Each time the releases have been painful, long, and fraught with errors and frustration. To alleviate the pain of deployment, Continuous Integration and Continuous Delivery are the answer. When ever the idea of a faster release schedule is introduced, usually by a business actor, most IT organizations push back. Typically the reasons are “What we did worked last time so we don’t want to change.”,  “The tools are too expensive”, and “We are not google or facebook”. As I will demonstrate the second argument is no longer valid. Using a set of free open source tools Continuous Integration and Continuous Delivery are easily achievable with some time and effort.

Example

The linked presentation and video example are a demonstration open source CI.

MS PowerPoint (DevOps_Opensource_CI_and_the_Cloud.pptx)

LibreOffice Impress (DevOps_Opensource_CI_and_the_Cloud.odp)

Youtube (https://www.youtube.com/watch?v=gIxCcJAl86M)

MP4 (https://s3.amazonaws.com/aws_wordpress/Coveros+Puppet+and+CI-voiceover.mp4)

The Scenario

In order to backup my statement I needed to prove it out with a non-trivial example. For the purposes of this demonstration I chose to show changes in a drupal site. The work flow could be characterized as follows:

  1. Developer creates tests
  2. Developer creates code
  3. Developer commits code and tests
  4. CI Framework detects changes
  5. CI Framework stands up empty VMs and databases
  6. CI Framework pushes down data, code, and configuration to the blank VMs
  7. CI Framework runs automated tests on the site and returns results

This process is completely hands off from the developer commit to the test results. No ops group needs to be contacted, no changes need to be filed, no e-mails need to be sent to the QA group alerting them to changes. The drupal instance is a multi-box, separate DB and Webserver, the example is not just a out of the box apache web server with the “it works” page.

The Tool Set

The software you use will vary from implementation to implementation, the software listed in this section is just what was needed for the demo. One size does not fit all, however, with the enormous amount of open source projects available there are basic tools for almost any need. Jenkins and Puppet are really the center pieces in this particular demonstration. Jenkins is the control point and executes the jobs and coordinates everything. Puppet is the CM database holding the necessary information to configure a VM and each application on each VM. PERL and sed are used for text manipulation, taking the output from the EC2 API or the puppet REST interface and turning the output into a usable format for Jenkins. EC2 itself is the infrastructure and elastic capacity for testing.

Path to CI and DevOps

Two words, “small steps”. Depending on the development organization you are working in or with trying to do this all at once it risky. It is easier to lay out a road map of implementing each individual technology and then integrating them. Start with OS packaging. Even if your deployment process is manual, developers and ops people can usually latch on to this idea. The two most challenging pieces are typically automated testing and hands off deployments. One requires a change in the way the testing organization works and the other may require a change to policy and personnel of the operations group. However, as more and more CI pieces get implemented, organizationally these changes will make more sense. In the end the inertia built up by increased efficiency and better releases may be the best leverage.

Conclusion

The three arguments against CI are typically cost, what we did worked last time, and while it works for other organizations it can’t work for us. From the demo you can see that CI can work and can be implemented with opens source tools negating the cost argument. While you have to invest in some (not alot) engineering effort, the cost of the tools themselves should be close to zero. Secondly, if you release cycle is more than 2 weeks – 3 weeks from written idea to implementation your process does not work. While there may be various IT heroes that expend monumental effort to make the product function, your development process does not work in a healthy sense negating the “what we do works” argument. Lastly is the argument that since “we” don’t have the resources of google or facebook we can’t possibly adopt a technology this complicated. The demo above was done by one person in about a weeks worth of effort. The tools are available and they use standard technologies. All the pieces exist and are well tested so you don’t need an engineering staff the size of google to implement CI, refuting some of the most common arguments against CI and DevOps.

Short intro to Puppet

Puppet can be daunting at first. Here is a quick explanation of the most important elements I found useful when first being introduced to puppet.

Configuration Language

The configuration language reference can be currently found at http://docs.puppetlabs.com/puppet/2.7/reference/ and http://docs.puppetlabs.com/puppet/3/reference/index.html . There are several parts to the puppet language. What follows is a quick description of the items most useful to me: facter, certain directives, augeas, and stages.

Facter

Facter is a simple name value pair of configuration items. For example

swapfree => 0.00 kB
swapsize => 0.00 kB
timezone => Local time zone must be set--see zic manual page
uniqueid => d40a1a41
uptime => 37 days
uptime_days => 37
uptime_hours => 910
uptime_seconds => 3276155

There are a set of default facts you get when you install facter which are generally useful, cpu type, number of cpus, hypervisor info, OS type, etc. This catalogue of facts is used to identify the machine in your configuration management database. Additionally, these facts are all accessable as variables inside your puppet code.

Lastly regarding facter you can create custom facts. Custom facts are authored in ruby. You create a small ruby program that returns a string representing the answer to a complicated calculation. Every time puppet is run on this machine, the fact will be calculated.

Puppet Language Highlights

Puppet allows you to describe an intended system state. It does this by creating resources that represent the major pieces of a running OS. Packages, Services, Files, Users, Groups, Host, and Mounts are just some examples of resource types.

File

Typically you are looking at files to control a machines configuration. That said puppet gives you two main ways to deal with these configuration file, copy them wholesale from a repository or generate them from a template based on facts.

file {
"/etc/sshd/sshd_config" :
content => template("openssh/sshd_config.erb"),
require => Package["openssh"]
}

This statement will create an sshd_config file in /etc/sshd based on the template sshd_config.erb. The template files are plain text except for the variables. For example:

ListenAddress <%= ipaddress %>

This allows the puppet class to create a correct config file based on the current environment or facts pre-populated. Dynamically generating the configuration allows the system to be right for where ever it is.

Services

Services are system level programs that run in the background. Generally these are services in the microsoft world or daemons in the UNIX world. The Service resource is very similar to the file resource in structure.

service {
"syslog":
enable => "true",
ensure => "running",
hasstatus => "true",
require => File["syslog.conf"],
subscribe => File["syslog.conf"]
}
This makes syslog start at boot, run if its not currently running, is able to return status, has its config file, and will automatically restart if the config file changes. This class can be applied to a running system and have the changes reflected immediately and correctly. If this resource is constructed correctly, a service will know of all its dependencies and react accordingly.

Packages

Packages are how an OS manages an installable feature. In UNIX land typically this refers to the RPM manager, yum, yast etc. In Gentoo its the emerge system and ebuilds, in the microsoft world this is the package installers.

 

Above is an example of puppet installing a package.

Augeas

While Augeas is technically a resource type, its a quite complicated way of manipulating configuration files. Basically, augeas hold a grammar for each config file you may be interested in, httpd.conf, sshd_config, etc. Augeas loads the parse tree into memory and allows you to manipulate the parse tree to add syntactically correct  statements. For example, you can insert changes in one fell swoop in a php.ini file without any complicated text manipulation.


augeas {
"php.ini":
notify => Service[httpd],
require => Package[php],
context => "/files/etc/php.ini/PHP",
changes => [
"set post_max_size 10M",
"set upload_max_filesize 10M",
];
}

This changes just those two values in the file without any sed or perl magic. This can be applied to more complicated objects like httpd.conf virtual hosts and other directives. Augeas is the most reliable way to change a file you don’t want to templateize.