See how happy people are when they have a safety net?

Getting Started

I’ve made a simple example app that shows code coverage in action. Check out the source code from the code_coverage branch of my spex repository:

git clone -b code_coverage --single-branch https://github.com/eyefodder/spex.git

Now go to the ops directory and run vagrant up To get the virtual machine running. Next, let’s hop into the virtual machine and run the test suite:

vagrant ssh
...some output...
vagrant@spex:~$ cd /app
vagrant@spex:~$ rspec

Now, check out the reports folder. You’ll see that there is a coverage/rcov folder. Open the index file in the browser and you see an easy to digest code coverage report:

Pretty nifty huh? You can click on the rows in the table to see each class in more detail, and find out exactly which lines aren’t being executed:

Let’s take a look at how this was all set up…

Code Coverage Gems

First up, we need to add a couple of gems to the Gemfile:

  # code_coverage
  gem 'simplecov', :require => false
  gem 'simplecov-rcov', :require => false

Once we’ve run a bundle install, our next step is to configure our test suite to generate coverage reports.

Configuring Code Coverage

This, again is a pretty simple affair. We need to launch SimpleCov before any application code has run, so we have this code at the top of the spec/spec_helper.rb

if ENV['GENERATE_COVERAGE_REPORTS'] == 'true'
  require 'simplecov'
  require 'simplecov-rcov'
  SimpleCov.start 'rails' do
    coverage_dir ENV['CI_COVERAGE_REPORTS']
  end
  SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
end

There’s a few things happening here. We have a couple of environment variables that tell us if we should create reports: GENERATE_COVERAGE_REPORTS and if we do, where we should put them: CI_COVERAGE_REPORTS. If you’ve followed my earlier post on getting Guard to send Growl notifications, you will know to find these in ops/dotfiles/guest_bash_profile which is a profile automatically generated when we launch the virtual machine with vagrant up. If not, well, now you do!
The next thing you’ll notice is the SimpleCov.start 'rails' call on line 4. This configures SimpleCov to have a profile that is good for most Rails applications. For example, the spec and config folders are excluded from coverage stats. You can read more about profiles here.
Finally, we tell SimpleCov that we want to format our results with the SimpleCov::Formatter::RcovFormatter. When we get to running our build as part of a continuous integration process with Jenkins, we can use this format to parse results to be viewed in the dashboard.

Viewing Code Coverage Reports generated on a Guest VM

The last thing we have to deal with is the fact that the reports are generated on the guest virtual machine. In our existing setup, we use rsync to push code changes from the host to the virtual machine. But this only works one way, and if we add content within the virtual machine you won’t see them on the host. We solve this with these lines in the Vagrantfile

  config.vm.synced_folder '../reports', '/reports'
  config.vm.synced_folder "../", "/app", type: "rsync", rsync__exclude: [".git/", "ops/*", "reports/", "tmp/", "log/", ".#*"]

What this does is exclude the reports from the main rsync and instead setup a new (regular) shared folder that will map reports to /reports on the virtual machine (note this is a root level folder, not in the /app folder on the guest. This is why we have used an environment variable to tell SimpleCov where to output reports.

Beware the emperor’s new code coverage

One thing to bear in mind is that code coverage really is a very crude metric. There are different types of coverage metrics, and SimpleCov only provides ‘C0′ coverage: lines of code that executed. Other types include branch and path coverage, but as far as I know, there aren’t any tools for these in Ruby. Let me show you an example of where this falls down:

If you look at this report, we can see that the some_method_with_conditionals gets called, but only the say_yes path (lines 12 and 13) executes, and we never confirm that ‘no’ gets sent if we pass false to the method. So far, so good, until we look at some_method_with_ternary. This is basically the same method refactored to be more compact, and with the same tests run against it. Yet we are told it is totally covered. So is the metric even still useful?

I still think code coverage is a valuable metric, if only to show you where there are holes in your test suite. If you go in with this knowledge and understanding the limitations, then you will be better equipped to maintain the quality of your app over time.

Code Coverage is a temporal metric

The last thing I want to mention about code coverage is that it’s useful to understand how your coverage changes over time. Particularly if you are managing a team of developers, it provides a quick warning if developers are slipping on their test writing. If you have a Continuous Integration machine, you can track these sort of metrics over time, which can really help you get a sense of where things are headed.
In my next post I’ll show how to set up your very own CI machine with just a few clicks…

The post Code Coverage — a simple Rails example appeared first on Eyefodder.

In order to try and do this I decided to use as much of the code as possible that the FlexCover guys had written, and only change what I needed to get it working.Once I had it up and running I could worry about making it faster / leaner / more neatly coded…

The approach

So here’s the basic plan:

1. Swap out the default localConnection reporting mechanism for one that will pick and choose depending on environment the app is running in
2. Create a headless agent that will pull the code coverage data collection and reporting side of the coverageViewer into the app
3. Use commandline options to swap between the headless and localConnnection agent
4. Default to using the localConnnection

How it works

When your app compiles using the instrumented SDK created for using FlexCover, your application makes use of a class called CoverageManager. This manager is a patch that allows you to plug in a custom code coverage agent for use in your app. By default it uses a class called LocalConnectionCoverageAgent which broadcasts code coverage metadata to the coverage viewer application. What my patch does is get in and allow you to use a different, headless agent. To do this simply call during the preinitialize event of your main application:

private function injectAgent():void{
CoverageManager.agent = new CoverageAgentSwitch();
}

As a quick test, run your app as you had previously and confirm it still works by sending data over a localConnection to the CoverageViewer. This is expected as by default, the switch will create a LocalConnectionCoverageAgent. In order to use the headless agent you need to set a few commandline properties. First you need to tell the switch that you want to go headless. Then the headless agent needs to know where it’s getting its metadata from, where it should output the code coverage report. The commandline options are:

-coverage-agent-type=headless <em>Headless is only option, anything else will default to localConnectionAgent</em>
-coverage-metadata='/full/path/to/coverage/metadata.cvm'
-coverage-output='/full/path/to/coverage/reportInOriginalFormat.cvr'
-emma-report='/full/path/to/coverage/reportInEmmaFormat.xml'

Note that if you don’t specify the metadata path and at least one of the report formats, the headless agent will log errors but otherwise fail silently.
In terms of the edits I made to get it working, that is pretty much it. The headlessCoverageAgent swc is probably larger than it needs to be and is definitely grossly inefficient. I will update this soon to improve this, but right now I only have time to get this post up.
Obviously you will have to change your build script to pass these new commandline parameters in to the test harness when you run it. If there is any interest, I’ll post my updated build script and test harness modifications that dispense with the log parser altogether and make for a more repeatable build script.

The Result

My build process is now much quicker because I don’t have to wait to be sure that the coverage viewer has initialized. It’s also completely stable and the code coverage trend has been a great motivator for the team.

Code Coverage trend without crazy variance found using the external viewer
Download headless library

Update

A few of you have asked for a copy of the source code so you can play around with it and undoubtedly make it all work better for your needs. Attached here—with no warranty or support—is the headless agent project. In order to get it to work, you will need to have the CoverageAgent and CoverageUtilities projects in your workspace, as well as the modified CoverageViewer project from previous posts. Good luck with it – It’s code I haven’t looked at for a while so don’t hate on me for it :/

The post Code Coverage with Flex – a headless agent for CI builds appeared first on Eyefodder.

So by now I had my code coverage viewer happily spitting out EMMA formatted code coverage reports that my build machine was just dying to consume. All I had to do was get the running of the code coverage test and the outputting of the report to be an integral part of my build. Here is the sequence of events for my build process:

1. clean my output folders (delete and recreate them)
2. Compile the main application using the instrumented SDK. This will generate the .cvr metadatafile that the coverageViewer needs
3. Compile the test harness using the instrumented SDK. This is so that when it runs, the hooks injected by the SDK report coverage data.
4. Launch the code coverage viewer Application (specifying EMMA output file)
5. Run the test harness. When it is finished, call CoverageManager.exit() which will close the viewer
6. Wait for the test results and code coverage report to be available. By implication this means that the test harness and code coverage viewer have quit.
7. Compile the main app using the standard SDK
8. (AIR Apps only) remove test harness and any test data from the output folder
9. (AIR Apps only) package up the application

In theory this is all well and good. In practice there are a few problems:

• The viewer has to launch and parse the code coverage metadata file. There isn’t a simple way to feed this back to the ant script, so you have to get the script to wait. I set mine to 30 seconds, which should be enough. Nevertheless, this sort of thing frankly just irks me – either at some point it will fail, and up until that point, the build is taking longer han it needs to.
• There are two applications being launched by the build process. In order for this to work, you need to launch the coverageViewer with spawn='true' If something goes wrong with the build, the script no longer has control of this process (I might be wrong on this one – correct me if so..)
• The code coverage agent (that gets injected into our test harness by the instrumented SDK) and the viewer communicate via LocalConnection. This allows two flash applications on the same machine to talk to one another. Unfortunately, LocalConnections can be kind of flaky and you have to build a separate local connection for each direction of communication.

The first two I could live with, but when I put this together, I found that there was quite a wide variance of about 10% from build to build of code coverage values:

Code Coverage trend showing wide variance of reporting results

I was left with one of two conclusions:

1. There was a bug/ fragility in the localConnection and it couldn’t be relied on
2. There was a bug in the coverageAgent.exit() code sequence

The way CoverageAgent.exit() works is that you call it from within your application and it:

1. Waits until the application has sent all remianing code coverage data
2. Sends a message to the code coverage viewer asking it to prepare for exit
3. The viewer writes out any report files
4. Exits, and sends a message back to the main application telling it to exit

Somewhere in this back and forth code coverage data was getting lost, and occasionally the build was failing because the report file was not getting written. I could have attempted to debug the code, but in my experience, debugging LocalConnections is a royal ballache. In the end I decided to bring the reporting side of the viewer ‘in-house’ and avoid the need for a second application altogether. My next post will show this and give you a download to the swc so that you can do it yourself…
Just in case you are interested, I have attached the buildfile I used for running the coverageViewer application as part of the build. If you can wait till I post the next update to this, I’d suggest you do
Download build file archive

The post Code Coverage with Flex – ANT build for running the viewer appeared first on Eyefodder.

So as I was starting to look at this, I could see one of three paths:

1. Create a Hudson plugin to consume flexCover’s code coverage report format
2. Add some sort of XSLT transform to my build process
3. Modify FlexCover to be able to output a report format that Hudson understands

Creating a plugin for Hudson certainly seemed like a possibility, but the plugins for EMMA and Cobertura were already there and stable, so it seemed like it would be much simpler to try and create a code coverage report in a format one of these plugins would understand. Creating an XSLT would work for this, but I’m not an XML guroid, so I figured I’d go down the route of the simplest thing that would work for me and hack extend flexcover to be able to output EMMA formatted code coverage reports.
As it turns out this was a pretty simple job, and after a few hours work I managed to create a patch that allows you to output .cvr or EMMA reports with an additional commandline argument. Alex, Joe and I have talked and although my way of doing it works, it is not designed to be scalable or modular to support more formats. They are incorporating the patch, but will probably rework it with these things in mind. They are super busy guys and it wont happen quickly; if you want to in the meantime you can grab the patch from here, but bear in mind it is offered with absolutely no warranties or support

Installation

First thing you need to do is checkout the FlexCover code from Google code (http://flexcover.googlecode.com/svn/trunk/CoverageViewer) and patch it with the file attatched. The patch is created from the project level down. To do this, you can right click the file from within Eclipse and select ‘Team>Apply Patch…’ Follow the instructions and you should be good to go…

Usage

In order to write out the code coverage report file with the viewer, you need to supply the commandline argument:

-output /full/path/to/my/flexcoverreport.xml

This functionality is unchanged, but now you can also specify an EMMA formatted file using the following:

-emma-report /full/path/to/my/EmmaReportName.xml

And that’s about it! My next post covers the challenges I had integrating this into my build process, and how I overcame them…

Download patch file

Update

Sorry this has taken so long to get together, and work out what was going on. I got swamped under a tonne of work, and neglected to update this

The link above is for an updated patch file (and note – I did this patch from the root level of the project), and for your pleasure below is the infamous missing class. I had to hunt through my old email as my old dev machine died a way back and appalingly I didn’t have the file under source control (I think it’s because I was patching someone elses repository). Anyhoo – enough excuses; here is the missing class: now renamed EMMAReportAdapter

The post Code Coverage with Flex – creating EMMA formatted reports appeared first on Eyefodder.

So, grab the jar from here and unzip it to your externals/lib directory. Then add the following properties to your build:

<property name="logParser" value="${lib}/FlashLogParser.jar"/>
<property name="flashStatus.location" value="${logs}/status.txt"/>
<property name="flashOutput.location" value="${logs}/TEST-testOutput.xml"/>

Then, change your parseFlashLog target to look like this:

<target name="parseFlashLog" description="parses flash log" depends="clean" >
<java jar="${logParser}" failonerror="true" fork="true">
<arg line="'${flashlog.location}'"/>
<arg line="'${flashStatus.location}'"/>
<arg line="'${flashOutput.location}'"/>
</java>
</target>

Now you should be good to go…
In my next post I am going to do a full dissection of my latest build file – I have spent some time refining it over the last year, and I think its probably helpful to spend some time looking at it bit by bit…

The post Continuous Integration with Flex – a better log parser appeared first on Eyefodder.

So, when I ran my ant build, all went well until the runTest target executed (or at least tried to execute). I got the following build error:

Execute failed: java.io.IOException: debugPlayerMac.app cannot execute

You see the problem is that on a mac, the standalone player (like other applications on the Mac) is actually a folder containing all sorts of cleverness inside. Ant doesn’t know how to execute a folder, so I was a bit stuck. Until, after lots of digging, I found the answer in an old osflash mailing list archive. Basically I have to use the ‘open’ command to launch the app – something like this:

<exec executable="open">
<arg line="${pathToFlashPlayer}"/>
<arg line="${pathToSWFToPlay}"/>
</exec>

I tried this, and it worked nicely. The problem is that my ant build needs to wait until the test harness runs and closes the player before reading the log and parsing the results for cruisecontrol to consume. In the ant script above, the open command only stalls the ant script until it has finished doing its opening magic, so ant gets upset as the test results haven’t been written yet. I slept on this, then couldn’t think of an answer so I sent an email to the ant users mailing list. Mere minutes later I got a reply that helped me work out how to crack this.
Basically, ant has the ability to run two threads, and move on when they are both complete. So in one thread we launch the player, and in another thread we first wait for the file to be available, then we do a check to see if it contains what we want (I am testing against the flag -----------------TESTRUNNEROUTPUTENDS---------------- which I used in my result printer). It worked like a charm:

<target name="runTest" description="runs the test harness" depends="compileTest">
<parallel>
<exec executable="open" spawn="no">
<arg line="${debugPlayer}"	/>
<arg line="'${testHarness.swf}'"/>
</exec>
<sequential>
<waitfor>
<available file="${flashlog.location}"/>
</waitfor>
<waitfor>
<isfileselected file="${flashlog.location}">
<contains text="-----------------TESTRUNNEROUTPUTENDS----------------"/>
</isfileselected>
</waitfor>
</sequential>
</parallel>
</target>

The last step to making this properly cross platform for cruisecontrol was to have the right <exec> command called depending on the OS. Fortunately there is an os attribute you can specify on <exec> and the task will only run if the os matches the platform you are running the task on. A few minutes later, I had my amended runTest target:

<target name="runTest" description="runs the test harness" depends="compileTest">
<parallel>
<exec executable="${debugPlayerWin}" spawn="no" os="Windows XP">
<arg line="'${testHarness.swf}'"/>
</exec>
<exec executable="open" spawn="no" os="Mac OS X">
<arg line="${debugPlayerMac}"	/>
<arg line="'${testHarness.swf}'"/>
</exec>
<sequential>
<waitfor>
<available file="${flashlog.location}"/>
</waitfor>
<waitfor>
<isfileselected file="${flashlog.location}">
<contains text="-----------------TESTRUNNEROUTPUTENDS----------------"/>
</isfileselected>
</waitfor>
</sequential>
</parallel>
</target>

I’m planning to post my completed project build script and step through it in more detail than I did previously. I’ve updated it quite a lot in the last year, and so far, I think its a lot neater. Before I do that, I’m going to post about my removing the python dependency from my continuous integration process.

The post CruiseControl on the Mac – modifying the build script to work x-platform appeared first on Eyefodder.

You see, on windows, I just press ctrl+c and the process terminates so you can then restart the sever. Well, on the mac, that doesn’t happen; if you ctrl+c, the server instance still runs. So, thanks to some friendly assistance, I found out how to kill the server. What you need to do first is type ps -e. This will list all the processes running. The one you are looking for will look something like this:

3267   p2  S  0.15.38   /System/Library/Frameworks/JavaVM.framework/Home/bin/java -Djavax.management.build

.
When you have that, note the PID, and then type kill -9 XXXX where XXXX is the pid you found in the previous step.

The post Restarting cruisecontrol on Mac OSX appeared first on Eyefodder.

First step, download the latest source from the downloads page here. Simply expand that to where you want your build instance to live. Next open up a terminal window, and type the following:
cruisecontrol.sh
More than likely, you will get the following error message:
-bash: cruisecontrol.sh: command not found
More hunting on the mighty intergoogle and a quick look at ‘bash guide for beginners’ told me the answer – I need to tell bash which shell to use to execute the cruisecontrol.sh script. I found either of the following lines worked:

sh cruisecontrol.sh
bash -x cruisecontrol.sh

I’m not a Unix geek, so I don’t pretend to know what the difference is, or which is preferable – if anyone can tell me I’d love to know..

The post CruiseControl on Mac OSX appeared first on Eyefodder.

I based my work here on how the flexunit.textui.TestRunner and flexunit.textui.ResultPrinter class work. I created two classes: CruiseControlTestRunner and CruiseControlResultPrinter.

CruiseControlTestRunner

This class is incredibly similar to TestRunner. In fact the only difference is that our private variable printer is set up to use our other new class – CruiseControlResultPrinter. If this variable had been declared protected, then I would have subclassed TestRunner and simply changed the setup of the printer to use our new class… You can download the file here

CruiseControlTestPrinter

The printer is a little more involved, but still not too tricky. what I do here is create an XML Object and append results as they come through. TestListeners (the interface that the printer implements) have functions that get triggered when a test starts and ends, and also if a test fails, or if there is an error. These can be used to construct our XML as the suite runs. Lets look at the class bit by bit: (or you can just download the class and get started

Constructor

Dead simple, we just initialize our XML with a top-level node:

public function CruiseControlResultPrinter()
{
	__resultXML = new XML("");
}

startTest

When a test starts, we want to append a testcase node to our XML. Each testcase node actually needs to sit in a testsuite node, so I created a helper function to automatically create a testsiute node for the testcase if one doesnt exist (ie if its the first test run in a particular Test Class):

 public function startTest( test:Test ):void
{
	var suiteNode:XML = getSuiteNode(test)
	suiteNode.appendChild(new XML(""));
	__testTimer = getTimer();
}
//------------------------------------------------------------------------------
private function getSuiteNode (test:Test):XML{
	var outNode:XML = __resultXML.testsuite.(@name==test.className)[0];
	if(outNode==null){
	outNode = new XML("");
	__resultXML.appendChild(outNode);
}
return outNode;
}

addError / addFailure

When a test fails or errors, we need to append failure details to the testcase node. The child node is named either failure or error depending on the type of problem (these nodes get displayed differently in CruiseControl). A child node populated with the stack trace is appended to the testcase node:

public function addError( test:Test, error:Error ):void{
	onFailOrError(test,error,"error");
}
//------------------------------------------------------------------------------
public function addFailure( test:Test, error:AssertionFailedError ):void{
	onFailOrError(test,error,"failure");
}
private function onFailOrError(test:Test,error:Error, failOrError:String):void{
	__suiteSuccess = false;
	var testNode:XML = getTestNode(test);
	var childNode:XML = new XML("&lt;"+failOrError+"&gt;"+error.getStackTrace()+"<!--"+failOrError+"-->");
	testNode.appendChild(childNode);
}
private function getTestNode(test:Test):XML{
	return __resultXML.testsuite.testcase.(@name==TestCase(test).methodName)[0];
}

endTest

If you look at the startTest code, you will see that we set a __testTimer variable. This comes into play in the endTest callback, where we set the execution time of the test Case:

public function endTest( test:Test ):void
{
	var testNode:XML = getTestNode(test);
	testNode.@time = (getTimer() - __testTimer)/1000;
}

print

When the test suite is complete, the TestRunner calls the print function on our class. This is actually now a much simpler affair than the one I previously wrote for ASUnit (although I will be revisiting this for ASUnit for AS3 over the next few days…). It simply traces out our generated XML, and also includes the line at the bottom specifying test Suite success (this is read by our ANT build):

public function print( result:TestResult, runTime:Number ):void
{
	printHeader(runTime);
	printMain();
	printFooter(result);
}
private function printHeader( runTime:Number ):void
{
	trace("-----------------TESTRUNNEROUTPUTBEGINS----------------");
}
private function printMain():void{
	trace(__resultXML);
}
private function printFooter( result:TestResult ):void
{
	trace("Test Suite success: "+(result.errorCount()+result.failureCount()==0)+"n");
	trace("-----------------TESTRUNNEROUTPUTENDS----------------");
}

And that’s basically it.. My next post is going to be about how I actually went about implementing these classes and got the test siute to display results differently depending on whether it was being run by the developer or as part of an ANT build…

The post FlexUnit for Cruise Control – XML output appeared first on Eyefodder.

• Simple syncronous tests
• asynchronous tests
• asynchronous setup (e.g. if you ned to load in some data before performing the tests)
• printing the test result out to a log file so that Cruise Control can interpret the results
• broadcast some sort of event when the test suite is complete so we can close the test harness and continue with the build
• works equally well outside of the flex framework

I’ll be looking at both FlexUnit and ASUnit to see which one will work for our needs best. Ideally I would like to offer up solutions to using both so that you can fit CI into any testing framework you are currently using. I’ll post my findings as I find them, so stay posted…

The post Unit test frameworks for AS3 and Continuous Integration appeared first on Eyefodder.