One of the things that I really liked about the version of CCNet that I was using was the ability to publish artifacts for download on the build report web page.  Here is a snapshot of the featured artifacts section of the summary report:

Beyond the nice GUI you could count on the url for the artifacts to conform to a pattern.  For example the weblogic release installer exe can be found at this address:

http://cir51-build03/Chapter33-5.2/Artifacts/20120314114423/Chapter33-Weblogic-Release-5.2.1.40468.exe

This enabled me to write a script, Chapter33.Deploy.Release.For.Tester.bat, that would download the http content, database installer, and weblogic installer and install the Chapter33 application in a tester’s private(local) workspace.  Well I am getting a little ahead of myself, the script also read from a CCNet REST interface.  It would gather up the build numbers for the last 6 successful Release Builds and present a GUI to the tester asking them which version of the application they would like to install.

Jenkins offers the analogous features, both artifacts for download and a REST interface, that will enable an analogous script to be written.  First lets look at the REST interface.  You can quickly see the xml from a REST request in you browser by adding /api/xml to any Jenkins page you are on.  So if you were at the following URL(a project dashboard):

Just adding /api/xml to the URL will call the REST interface:

If you want some more info on options for calling the REST API just /api to the URL…  Now the Jenkins xml REST interface is pretty cool in that you can specify the depth of information as well as tune it with an xpath query.  We will need a deeper depth of information as the current depth does not show the result, success or failure, for each build, nor does it give us the build display name.  By adding the query param depth=1 we get:

Finally we just need the build number for the last 6 successful build.  Note on my projects where we set the build display name to the SVN revision number we select the fullDisplayName as opposed to the build number here.  By adding the this xpath query xpath=(/*/build[result='SUCCESS'])[position()<=6]/number we get the following results: 

Only 5 results are returned here because there are only 5 successful builds retained on this job.  With this REST query we have enough to get started on a Groovy script to download the artifacts.

def protocol = ‘http://’
def serverName = ‘ci.jruby.org’
def port = ”
def jobName = ‘jruby-dist-release’
def resultSetSize = 6

def restEndPointBuildList = "${protocol}${serverName}${port}/job/${jobName}/api/xml?depth=1&xpath=(/*/build[result='SUCCESS'])[position()<=${resultSetSize}]/number&wrapper=builds"
def slurper = new XmlSlurper()
def buildList = slurper.parse(restEndPointBuildList)

def goodBuilds = new ArrayList<String>()

buildList.number.each {
    goodBuilds.add(it.text())
}

This code will create an array of the build numbers returned from the REST query.  It uses the XmlSlurper to accomplish this, both downloading the REST result and parsing it so that we can loop over it creating the array of good build numbers.  Next we need to present the user with a UI so that they may choose a build that they wish to download and install.  I did not care about making the UI pretty so I opted to use a basic antforms UI.

def goodBuildsFlat = ”

goodBuilds.sort().each {
    if (goodBuildsFlat == ”){
        goodBuildsFlat = it
    }else{
        goodBuildsFlat = it + "," + goodBuildsFlat
    }
}

def ant = new AntBuilder()
ant.taskdef(
    name:"antform",
    classname: "com.sardak.antform.AntForm",
    classpath: System.getenv(‘ANT_HOME’) + "/lib/antform.jar"
);

ant.antform(title: "Choose Which Build to Deploy"){
   ant.label("Choose a build")
    ant.radioSelectionProperty(label: "Builds: ", property: "buildChosen", values: "${goodBuildsFlat}")
}
def buildNumber = ant.project.properties.buildChosen;

The antforms task will accept a comma separated value string of options to make radio buttons out of.  I wanted them to be sorted, that is first bit of code here.  Next we need to load the antforms task.  You can see here that the jar is loaded from the ant lib dir.  You could load it from anywhere…  After the ant task has been loaded we call to display the form passing in several options like a title and labels.  When the user clicks the OK button the ant property buildChosen will contain the value they chose.  This is placed in the local variable buildNumber.

def restEndPointBuild = "${protocol}${serverName}${port}/job/${jobName}/api/xml?depth=1&xpath=/*/build[number='${buildNumber}'][1]"

def build = slurper.parse(restEndPointBuild)

build.artifact.each {
    def artifactFileName = it.fileName.text()
    def artifactRelativePath = it.relativePath.text()
    def artifactUrl = "${protocol}${serverName}${port}/job/${jobName}/${buildNumber}/artifact/${artifactRelativePath}"
    download(artifactUrl, "C:\\Temp\\${artifactFileName}")
}

Next the script makes a REST query getting the build details for the build chosen.  It then loops over the artifacts published in that build downloading each.  This is the part of the script that you will need to start tailoring to your needs.  Where do you want to download to?  Do you want to download all the artifacts or just a specified few?  What are you going to do after downloading the artifacts?  On the project that I am working on at the moment we publish full on headless installers.  After downloading these installers we execute them.  This provides a push button script for the testers or anyone on the project to deploy a personal instance of the application any time they wish.  They even get to choose the version they want to install.

Ooh before I forget here is the last bit of the script, the download method.  I used Curl so that if the download fails it will pick back up where it left off we you retry.  This assumes that Curl is in your PATH.

def download(url, destination){
    String command = "-0 -m 86400 -S -C – -o \"${destination}\" -k -L –retry 5 ${url}"

    def ant = new AntBuilder()
    ant.exec(executable: "curl"){
        arg(line: "${command}")
    }
}

One of the things that I really liked about the version of CCNet that I was using was the custom build labeller we were using.  It would append to a base string the SVN revision number.  Here is a snapshot of the Recent Builds widget on one of our Release Build reports:

This build server was working on the 5.2.1 branch and the most recent build was of SVN revision 40468.  This allows developer to easily communicate what build their changes or fixes are in.  So for example a tester can easily understand which build they want to download, install, and test…  That is very convenient, yes?

Jenkins does not have this feature.  In all fairness neither did CCNet, we had to write a plugin.  In this case I found a Groovy plugin for Jenkins that I used to script up a solution.  The documentation for the plugin can be found here.

I made the first build step a system Groovy step and added the following code:

Executing as a system Groovy step means this will be executed in the same JVM as Jenkins, allowing access to all the Jenkins objects…so we can alter the build display name.  The first two lines are importing the Jenkins(Hudson) packages.  The javadocs for what is available are located here.

import hudson.model.*
import hudson.util.*

The next line is a neat little trick that will give you a reference to the current build.

def build = Thread.currentThread().executable

We will first use this build object to get the workspace folder path.  This is where the build is executing out of.  We need this piece of information so that we can execute the subversion command “info” in the root of the workspace.

def workspace = build.getWorkspace()

In Groovy if you want to specify the directory from which to execute a shell command and you want to capture the command’s output the easiest way to use the Ant task exec like so:

def ant = new AntBuilder()
ant.exec(executable: "svn", outputproperty: "output", dir: workspace){
    arg(line: "info")
}

svnInfo = ant.project.getProperty("output")

That captured the out of the svn “info” command into the variable svnInfo.  Now we can use a regular expression to extract the revision we are currently on.  Here is some example output from an svn “info” command:

C:\Projects\Chapter33\Trunk\Build>svn info
Path: .
URL: https://va33-repo01/svn/Chapter33/Trunk/Build
Repository Root: https://va33-repo01/svn/Chapter33
Repository UUID: f1ce2e10-74e2-f14b-9613-4d7166fa18d4
Revision: 40498
Node Kind: directory
Schedule: normal
Last Changed Author: bassettt
Last Changed Rev: 40484
Last Changed Date: 2012-03-14 16:24:37 -0400 (Wed, 14 Mar 2012)

We want to extract from all that the Last Changed Rev value, and we can do that with this code:

def pattern = /Last\s+Changed\s+Rev:\s+(\d+)/
def matcher = (svnInfo =~ pattern)

def buildLabel = ‘Dev-’ + matcher[0][1]

We take the extracted value, in this example 40484, and set a variable named buildLabel to “Dev-40484”.  Lastly we set the Jenkins build display name.

println ‘setting build label for this build’

build.setDisplayName(buildLabel)

This results in a Build History widget that looks like this:

If you are familiar with Jenkins you might ask why not just use the Build Name Setter Plugin?  I would have but the svn env var Jenkins sets is often incorrect as documented here(I too see this bug).  So I wrote my own solution…  I also use variations on this to show versions of the application as the move through the build pipeline.  Instead of grabbing the build name from subversion in downstream builds I grab it from the triggering upstream build.  There are lots of interesting uses for this.

At the beginning of most projects that I have been on the default starting position has been that dev and test will each get their own environment to share.  Developers share the dev env and testers share the test env.  I think we need to change this.  Shared environments are good for only a select set of scenarios and development and testing are not among them.

If they have been the norm what has changed to allow us to go a different route?  Two things: first increased performance of workstations and laptops and second improved automation in the dev/test workspace.  Increased performance in hardware has allowed us to run more servers in a local workspace.  On my laptop I can easily run several Weblogic servers and an Oracle Database with resources left for an IDE and other dev/test tools. Not so long ago this was impossible.  Managing all my servers, the applications running on them, and one or more databases would leave little time to do anything else in a fast changing development project.  Automation alleviates this time sink.  Providing push button headless automation to setup, deploy, and manage these servers is key.  This too was thought of as impossible not to long ago, yet fully automated deployments are more and more common these days.

Okay, so maybe it is possible you say.  Why would I want to do this?  Haven’t shared environments been working…

No, I don’t think they have been working.  The whole purpose behind this change is to enable developers and testers to test the application more easily, spend less time identifying and recovering from collisions, and for developers specifically to spend less time chasing the version of the shared env.  When several people share an environment they can easily collide with each other.  There are many types of collision, they depend on the architecture of the application under test as well as it’s dependencies.  Many of these types of collision revolve around data.  There are schemes to minimize data collisions, yet no scheme is foolproof.  If every developer and tester has a private environment to work in several things become possible:

• testers and developers can easily roll back to any version of the application to replicate a bug, or return to last working version
• testers and developers can execute automated tests at will, or any kind of test for that matter, no more collisions due to data, execution resources, etc…
• no one is forced to update to a new version of the application, in shared envs updates are normally done nightly, for example a developer could work uninterrupted on a bug through one day and into the next, i.e. no more chasing the shared env version…

All of these things will dramatically increase the productivity of a team!

At this point you may think that I am claiming that a whole environment should be hosted locally on a developer/tester workspace.  I am not.  I only think that all read/write servers should be local.  Readonly servers could, most of the time should, be shared.  Remember that what counts is how you interact with the server.  If your application only ever reads from it then you should treat it as a readonly server.

If you search around the internet on this subject you will mostly find that this subject has been written on from the database point of view.  Here are a few good examples:

• The unnecessary evil of the shared development database
• Working on a Shared Development Server is Bad
• Top 4 Reasons Why a Shared Development Database is Evil.

Most of the issues that have been documented around shared database servers effect shared webservices, EJBs, .Net Remoting, REST, etc…

Just in case you are not yet convince let’s try some systems thinking.  This situation of shared servers or resources indicates that we should take a look at the archetype “Tragedy of the Commons”.  This is taken from the site http://www.systems-thinking.org:

Tragedy of the Commons

The Tragedy of the Commons structure represents a situation where, to produce growth, two or more Reinforcing Loops are dependent on the availability of some common limited resource.

A’s activity interacts with the resources available adding to A’s results. A’s results simply encourage more of A’s activity. The same sequence plays out for B’s activity. And, the more resources used the greater the results. This simply encourages A and B to use more resources.

A’s activity and B’s activity combine to produce some total activity. This total activity subtracts from the resources available. The extent of the resources available being defined by the resource limit.

Total activity continues until it completely depletes the resources available. When this happens A’s results and B’s results stop growing as there are no more resources to use. What makes this structure even worse is that whoever figures out the structure first, A or B, wins because they use all the resources before the other has a chance to. This structure is often referred to as "All for one and none for all."

Managing the Structure

This structure repeatedly appears in organizational contexts where a service organization supports the success of multiple departments who fail to support the service organization in return. There are two strategies for dealing with this structure, one more effective than the other.

• The most effective strategy for dealing with this structure is to wire in feedback paths from A’s results and B’s results to the resource limit so as A and B use resources their results promotes the availability of additional resources.
• The alternate, and less effective, strategy for dealing with this structure is to add an additional resource to control the use of resources by A and B. This strategy limits the overall potential results of the structure to the predefine resource limit. It also adds additional resource to the equation, and probably results in endless disputes as to the fairness associate with the allocation of resources. While not really the most appropriate strategy this is the one most often used — out of ignorance I would suspect.

Examples

• If It’s Free I Want All I Can Get
• When Fair is Unfair

In our case the resource can be replenished in a couple of ways depending on how it was or is being depleted.  A scorch and rebuild of the data will replenish a data depletion.  If the server resources, CPU, memory, IO, etc… where depleted then simply curtailing the over usage will replenish the server.  I find this site’s offered solutions lacking.  It’s presentation is as if it is a syllogism.  An easy third solution is to dedicate a resource per user.  This removes some of the limiting factor from the system and all of the shared aspect leaving a set of independent reinforcing loops, one per user.  I this case both A and B get their own resource.