Code Coverage with Flex – ANT build for running the viewer

• Tweet

• Tweet

In my last post, I gave you my elegant extension hack for generating EMMA style code coverage reports from FlexCover. This post covers the first route I took to incorporating this in my build process. It does work, but it’s not very consistent in its reporting and I’ll explain why at the end…

So by now I had my code coverage viewer happily spitting out EMMA formatted code coverage reports that my build machine was just dying to consume. All I had to do was get the running of the code coverage test and the outputting of the report to be an integral part of my build. Here is the sequence of events for my build process:

1. clean my output folders (delete and recreate them)
2. Compile the main application using the instrumented SDK. This will generate the .cvr metadatafile that the coverageViewer needs
3. Compile the test harness using the instrumented SDK. This is so that when it runs, the hooks injected by the SDK report coverage data.
4. Launch the code coverage viewer Application (specifying EMMA output file)
5. Run the test harness. When it is finished, call CoverageManager.exit() which will close the viewer
6. Wait for the test results and code coverage report to be available. By implication this means that the test harness and code coverage viewer have quit.
7. Compile the main app using the standard SDK
8. (AIR Apps only) remove test harness and any test data from the output folder
9. (AIR Apps only) package up the application

In theory this is all well and good. In practice there are a few problems:

• The viewer has to launch and parse the code coverage metadata file. There isn’t a simple way to feed this back to the ant script, so you have to get the script to wait. I set mine to 30 seconds, which should be enough. Nevertheless, this sort of thing frankly just irks me – either at some point it will fail, and up until that point, the build is taking longer han it needs to.
• There are two applications being launched by the build process. In order for this to work, you need to launch the coverageViewer with spawn='true' If something goes wrong with the build, the script no longer has control of this process (I might be wrong on this one – correct me if so..)
• The code coverage agent (that gets injected into our test harness by the instrumented SDK) and the viewer communicate via LocalConnection. This allows two flash applications on the same machine to talk to one another. Unfortunately, LocalConnections can be kind of flaky and you have to build a separate local connection for each direction of communication.

The first two I could live with, but when I put this together, I found that there was quite a wide variance of about 10% from build to build of code coverage values:

Code Coverage trend showing wide variance of reporting results

I was left with one of two conclusions:

1. There was a bug/ fragility in the localConnection and it couldn’t be relied on
2. There was a bug in the coverageAgent.exit() code sequence

The way CoverageAgent.exit() works is that you call it from within your application and it:

1. Waits until the application has sent all remianing code coverage data
2. Sends a message to the code coverage viewer asking it to prepare for exit
3. The viewer writes out any report files
4. Exits, and sends a message back to the main application telling it to exit

Somewhere in this back and forth code coverage data was getting lost, and occasionally the build was failing because the report file was not getting written. I could have attempted to debug the code, but in my experience, debugging LocalConnections is a royal ballache. In the end I decided to bring the reporting side of the viewer ‘in-house’ and avoid the need for a second application altogether. My next post will show this and give you a download to the swc so that you can do it yourself…
Just in case you are interested, I have attached the buildfile I used for running the coverageViewer application as part of the build. If you can wait till I post the next update to this, I’d suggest you do
Download build file archive