Continuous Integration with Flex – Part 6

• Tweet

• Tweet

In this, the final entry on CI with Flex, I look at an extra change we made to the ASUnit sources, take a look at what CI has done for us, and finally let you in on what we are looking at next…

Execution Time

When we got CI up and running I started looking more closely at the information that JUnit gives out versus the information that ASUnit gives. The only omission I found was that in JUnit you get feedback on the execution time of a particular assert. I looked at how we might implement this in ASUnit, and made a slight chage to the sourcecode that gives us an execution time. The problem here is that execution time is displayed per assert (in Cruise Control), but calculated per function, so its not completely accurate. What it will do however is help to nail down code that executes slowly.. in order to get this to work, you will need to find the runMethod() function in TestCase, change it and add the method appendExecutionTime like this:

private function runMethod(method:String)
{
	setUp();
	setCurrentMethod(method);
	var before:Number = getTimer();
	this[method]();
	var elapsed:Number = getTimer() - before;
	appendExecutionTime(elapsed/1000);
	tearDown();
}
private function appendExecutionTime(n:Number):Void{
	var list:TestRunner = getTestRunner();
	var max:Number = list.length;
	var cName = getCurrentClassName();
	var cMeth = getCurrentMethod();
	var item:Object;
	for (var i=0;i<max;++i){
		item =list[i];
		if(item.methodName==cMeth && item.className==cName ){
			item.executionTime=n;
		}
	}
}

How the non-technical team responded to CI

As we had expected, CI increased the visibility of the development process. We expected this to be seen as a good thing, and in many cases it was. We also had a few unexpected results too. Members of the business team would panic when the most recent build was either broken, or had seemed to take a step back. We as a development team had figured that everyone else would realise that as a development machine, sometimes we will break stuff (of course, we fix it as soon as possible…) and that this was no big deal. But the problem was that they hadn’t been used to this level of visibility, and it was just a scary thing. A little bit of education down the line and they were more relaxed about it though…

On the plus side, having CI as part of our process did reduce the number of nasty, hard to find bugs in our app. Unfortunately I dont have any hard evidence for this, just empirical, but I think I’m right…

Next Steps

This is only the beginning of our automated testing experiment. We would like to take this approach further by looking into automating acceptance(functional) tests into our framework. Adobe are working on a functional testing tool to integrate into Mercury QTP. We would like to look at incorporating this into our automation process so that there is an even higher degree of confidence that what the development team offer out to the larger project team works harder better faster and stronger…

Another thing I am interested in looking at is how to shorten the feedback loop even further. I would like to have our Cruise Control build results page have a simple feedback form in it which will tie into our bug tracking system. This will mean that it is even easier for people to give us feedback on development. We will also be tagging feedback to build numbers, so we can keep track of when a bug was spotted.

There are two things I have been thinking of as weaknesses in our process that could do with being addressed, but I dont really know the answer to yet. The first is unit testing GUI’s which I have never seen a satisfactory solution for. We have mitigated this by removed everything but layout from MXML components, so at least we can test business logic, but Im sure there is a better solution out there.

The second thing that has been nagging me is how we go about writing better (read complete) unit tests. The thing is that when doing test-driven-development, its all too easy to think you are too busy to write a test just now, or that you will ‘go back and write the tests later’. This is a bit of an arse about tit mentality – you will never get around to writing tests if you dont do it when you write the code. Some gains can be made with self-discipline, but I have been mulling the idea of a code coverage tool for AS. I’m still not sure if this will be useful (or used) but its one of many side-projects spinning round in my head at the moment…