See how happy people are when they have a safety net?

Getting Started

I’ve made a simple example app that shows code coverage in action. Check out the source code from the code_coverage branch of my spex repository:

git clone -b code_coverage --single-branch https://github.com/eyefodder/spex.git

Now go to the ops directory and run vagrant up To get the virtual machine running. Next, let’s hop into the virtual machine and run the test suite:

vagrant ssh
...some output...
vagrant@spex:~$ cd /app
vagrant@spex:~$ rspec

Now, check out the reports folder. You’ll see that there is a coverage/rcov folder. Open the index file in the browser and you see an easy to digest code coverage report:

Pretty nifty huh? You can click on the rows in the table to see each class in more detail, and find out exactly which lines aren’t being executed:

Let’s take a look at how this was all set up…

Code Coverage Gems

First up, we need to add a couple of gems to the Gemfile:

  # code_coverage
  gem 'simplecov', :require => false
  gem 'simplecov-rcov', :require => false

Once we’ve run a bundle install, our next step is to configure our test suite to generate coverage reports.

Configuring Code Coverage

This, again is a pretty simple affair. We need to launch SimpleCov before any application code has run, so we have this code at the top of the spec/spec_helper.rb

if ENV['GENERATE_COVERAGE_REPORTS'] == 'true'
  require 'simplecov'
  require 'simplecov-rcov'
  SimpleCov.start 'rails' do
    coverage_dir ENV['CI_COVERAGE_REPORTS']
  end
  SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
end

There’s a few things happening here. We have a couple of environment variables that tell us if we should create reports: GENERATE_COVERAGE_REPORTS and if we do, where we should put them: CI_COVERAGE_REPORTS. If you’ve followed my earlier post on getting Guard to send Growl notifications, you will know to find these in ops/dotfiles/guest_bash_profile which is a profile automatically generated when we launch the virtual machine with vagrant up. If not, well, now you do!
The next thing you’ll notice is the SimpleCov.start 'rails' call on line 4. This configures SimpleCov to have a profile that is good for most Rails applications. For example, the spec and config folders are excluded from coverage stats. You can read more about profiles here.
Finally, we tell SimpleCov that we want to format our results with the SimpleCov::Formatter::RcovFormatter. When we get to running our build as part of a continuous integration process with Jenkins, we can use this format to parse results to be viewed in the dashboard.

Viewing Code Coverage Reports generated on a Guest VM

The last thing we have to deal with is the fact that the reports are generated on the guest virtual machine. In our existing setup, we use rsync to push code changes from the host to the virtual machine. But this only works one way, and if we add content within the virtual machine you won’t see them on the host. We solve this with these lines in the Vagrantfile

  config.vm.synced_folder '../reports', '/reports'
  config.vm.synced_folder "../", "/app", type: "rsync", rsync__exclude: [".git/", "ops/*", "reports/", "tmp/", "log/", ".#*"]

What this does is exclude the reports from the main rsync and instead setup a new (regular) shared folder that will map reports to /reports on the virtual machine (note this is a root level folder, not in the /app folder on the guest. This is why we have used an environment variable to tell SimpleCov where to output reports.

Beware the emperor’s new code coverage

One thing to bear in mind is that code coverage really is a very crude metric. There are different types of coverage metrics, and SimpleCov only provides ‘C0′ coverage: lines of code that executed. Other types include branch and path coverage, but as far as I know, there aren’t any tools for these in Ruby. Let me show you an example of where this falls down:

If you look at this report, we can see that the some_method_with_conditionals gets called, but only the say_yes path (lines 12 and 13) executes, and we never confirm that ‘no’ gets sent if we pass false to the method. So far, so good, until we look at some_method_with_ternary. This is basically the same method refactored to be more compact, and with the same tests run against it. Yet we are told it is totally covered. So is the metric even still useful?

I still think code coverage is a valuable metric, if only to show you where there are holes in your test suite. If you go in with this knowledge and understanding the limitations, then you will be better equipped to maintain the quality of your app over time.

Code Coverage is a temporal metric

The last thing I want to mention about code coverage is that it’s useful to understand how your coverage changes over time. Particularly if you are managing a team of developers, it provides a quick warning if developers are slipping on their test writing. If you have a Continuous Integration machine, you can track these sort of metrics over time, which can really help you get a sense of where things are headed.
In my next post I’ll show how to set up your very own CI machine with just a few clicks…

The post Code Coverage — a simple Rails example appeared first on Eyefodder.

Growl isn’t as hairy as ‘old Ephraim’ but is probably more useful

Getting Started

To get started and see Growl notifications, we need to do a little more than normal. Let’s start by checking out the source code:

git clone -b guard_growl --single-branch https://github.com/eyefodder/spex.git

Prepping Growl

Download Growl if you don’t already have it. Once it’s installed, open up the preferences pane and click on the ‘network’ panel. Check the box marked ‘Listen for incoming notifications’ and enter a password:

Prepping Vagrant

The next thing we need to do is install a plugin for Vagrant that will allow us to execute a script on the host machine when we boot our virtual machine. I’ll explain what that is below the fold, but for now, make sure that you have Vagrant 1.6.4 or above (hint, type vagrant -v to find out) and update if necessary. Next, install the Vagrant Triggers plugin by entering this on the command line:

$ vagrant plugin install vagrant-triggers

Bringing the virtual machine up

Next, run vagrant up as usual. When it runs, it creates a file ops/dotfiles/guest_bash_profile:

# Edit the following values with your own values

# Growl password set in Growl > Preferences > Network
export GROWL_PASSWORD=enter_growl_password

# The following entries are automatically generated
# Do not edit unless you know what you are doing!
# They are regenerated each time the virtual machine is rebooted
export HOST_IP=10.0.1.28

Go ahead and enter your Growl password. Now we’re good to go. run vagrant rsync-auto to keep things in sync and in another window, then vagrant ssh into the machine.

Hey presto! Growl notifications

So let’s get up and running! When you are ssh’d into the host machine, fire up Guard with cd /app && bundle exec guard -p. Make a change to your code and when the tests run you should see a notification:

You’re good to go now, but read on if you’d like to understand how all this stuff works…

How it works

There are a lot of moving parts to get this thing going. I’m going to work from the Guardfile back, and then the Vagrantfile forwards. I guess we’ll meet somewhere in the middle so bear with me.

Guardfile changes

If you look in the Guardfile you will see a few new lines of code at the top:

host_ip = ENV['HOST_IP']
growl_pass = ENV['GROWL_PASSWORD']
PLACEHOLDER_GROWL_PASS='enter_growl_password'

if host_ip.blank? || growl_pass.blank? || (growl_pass==PLACEHOLDER_GROWL_PASS)
  puts 'host notifcations off: you must set HOST_IP and GROWL_PASSWORD in ops/dotfiles/guest_bash_profile'
  notification :off
else
  notification :gntp, :sticky => false, :host => ENV['HOST_IP'], :password => ENV['GROWL_PASSWORD']
end

Line 23 tells Guard to use GNTP for notifications (assuming HOST_IP and GROWL_PASSWORD have been set). It’s basically a means for sending Growl notifications over a network. You’ll need to add the GNTP gem, so we’ve added this to our Gemfile:

  # guard_growl
  gem 'ruby_gntp'

We can see that this system relies on a couple of environment variables being set. We got a clue that these get set in ops/dotfiles/guest_bash_profile, let’s see how that file gets created, and how we get that linked into the guest Virtual Machine.

Creating the guest bash profile

For this setup to work, we need two environment variables set: GROWL_PASSWORD and HOST_IP. The script that creates the file is ops/setup_guest_bash_profile. This script does a few things, so let’s step through it:

Create a profile file if it doesn’t exist

The first thing we want to do is create a profile file if one doesn’t exist already:

# make a profile file if it doesn't exist
bash_file=dotfiles/guest_bash_profile
touch $bash_file

Next, we want to add some comments to the file. As we will run this script every time the machine boots up we only really want to add these comments if they don’t already exist. The add_comments_once function does this by checking for a match, and only adding the comment if it isn’t already in the file:

add_comment_once(){
  comment=$1
  grep -q "$comment" $bash_file
  if [ $? -ne 0 ]
  then
    echo "# $comment" >> $bash_file
  fi
}

Adding a bash entry for users to change

Next thing we want to do is add environment variables for users to change. But as this code is run multiple times, we actually have three scenarios:

• An entry doesn’t yet exist in the file: create it
• An entry exists but hasn’t been changed from the default: tell user to change it
• A user has entered their own value: don’t do a thing!

The add_bash_entry function achieves this:

add_bash_entry(){
  token=$1
  default=$2
  change_message="You need to enter your $1 in $bash_file"
  entry_comment=$3

  grep -q $token= $bash_file
  if [ $? -eq 0  ]
  then
    grep -q $default $bash_file
    if [ $? -eq 0 ]
    then
      # entry exists, but it's set to default value from this script
      echo $change_message
    fi
  else
    # variable entry doesn't exist
    echo $change_message
    echo $entry_comment
    echo "" >> $bash_file
    echo "# $entry_comment" >> $bash_file
    echo "export $token=$default" >> $bash_file
    echo " " >> $bash_file
  fi
}

The function gets three arguments: the token GROWL_PASSWORD, the default value enter_growl_password, and a message to explain to the user what to do. If the token is found (lines 22 & 23) then it looks for the default value (lines 25 & 26) and prints a change message if it’s there. If the token isn’t found (lines 32-38) then we write that into the file.

Adding the HOST_IP as a system generated value

The next thing we want to do is add the host system’s IP address to the file so that Growl knows where to send notifications:

sed -i '' '/HOST_IP/d' $bash_file
# grab current host IP
ipaddr=`ifconfig | grep "inet " | grep -v 127.0.0.1 | cut -d ' ' -f2`
# and write to file
echo "export HOST_IP=$ipaddr" >> $bash_file

This script deletes any existing lines with HOST_IP in them (line 65), then uses a little bash trickery to find the current host ip. I found out how to do it from this post, although I needed to change the delimiter from \ to ' ' (that’s from an escaped space character to a space in quotes. Finally we write this out to our guest_bash_profile file. The next step is getting this script to run when we want it.

Running the profile setup script when Vagrant starts

Thanks to the Vagrant Triggers plugin, this is a really simple affair. We just add the following to our Vagrantfile:

  config.trigger.before [:up, :reload], :stdout => true do
    run "sh ./setup_guest_bash_profile.sh"
  end

This will run our script every time we call vagrant up or vagrant reload, ensuring that our host ip address is always up to date in the file. The last piece of the puzzle is to make sure we use this file to actually set environment variables on the guest machine.

Linking the guest bash profile to the guest virtual machine

This is a relatively simple two-part process. First thing we do is share the ops/dotfiles directory on the virtual machine:

  config.vm.synced_folder 'dotfiles', '/dotfiles'

Secondly, we want that file symlinked in the guest machine to ~/.bash_profile. I created a new Puppet class to achieve this. Check out ops/puppet/modules/spex/dotfile_symlink.pp:

class spex::dotfile_symlink{
  file { '/home/vagrant/.bash_profile':
    ensure => link,
    target => '/dotfiles/guest_bash_profile'
  }
}

Super simple here. We tell puppet to ensure that /home/vagrant/.bash_profile is a symlink to /dotfiles/guest_bash_profile. In ops/puppet/manifests/default.pp we simply include the class with the others:

include spex::base_packages
include spex::postgres_setup
include spex::ruby_setup
include spex::dotfile_symlink

Now we have everything wired up and ready to go!

Conclusion

This wraps up my example for getting Growl notifications from Guard into the host machine. Although there are a bunch of steps to jump through, once it’s working I hope you’ll find it a pretty robust solution.

The goal in all of this is to shorten feedback loops when you develop. This process should give you some confidence that when your code changes the right tests run. The power of this is greatest when you are confident that your tests cover enough of your application such that you will know when you break things. Our next step is to look into the breadth of your tests and setting up code coverage metrics for your app…

The post Getting Growl notifications from your Virtual Machine appeared first on Eyefodder.

Things are zippier on Spring(s)

Getting the sample code for Spring with RSpec

As with all of the examples, I’ve setup the simplest possible example to show setting up Spring with RSpec. Go ahead and checkout the code here:

git clone -b rspec_guard_spring --single-branch https://github.com/eyefodder/spex.git

Once you have checked out the code, jump into the ops directory and vagrant upto bring the VM up. When it’s up and running, get synching running by entering vagrant rsync-auto and then in a new terminal window enter the following:

vagrant ssh
cd /app
bundle exec guard -p

And that is basically it! Now when you run your tests, you will be taking advantage of Spring! Read on to see how to get this working…

How we got it working

So to get this working we did a few things. First off, we added the spring commands for rspec with the spring-commands-rspec gem:

  # rspec_guard_spring
  gem 'spring-commands-rspec'

Next, we let spring create the binstub for us (in the example we already did this, so you will see the outputted file at bin/rspec:

pbh$ spring binstub --all
* bin/rake: spring already present
* bin/rspec: generated with spring
* bin/rails: spring already present

Lastly, we need to update the command that Guard uses to run RSpec in the Guardfile:

guard :rspec, cmd: 'bin/rspec' do
  ...
end

The ‘gem pristine…’ warning

When your specs run you will probably see an error that talks about running gem pristine --all to speed up loading:

Warning: Running `gem pristine --all` to regenerate your installed gemspecs (and deleting then reinstalling your bundle if you use bundle --path) will improve the startup performance of Spring.

I did a little digging on this and found the gems that are causing the issue using the code posted in this answer:

vagrant@spex:/app$ ruby -e 'p Gem::Specification.stubs.reject(&:stubbed?).reject(&:default_gem?).map(&:name)'
["bigdecimal", "io-console", "json", "minitest", "psych", "rake", "rdoc", "test-unit"]

A little more digging told me that these gems are probably system Ruby packages that appear as gems thanks to the rubygems-integration. Long story short, you have to live with with this warning message for now. Although startup performance might be affected, overall performance each time you run RSpec should still be great! Now you have Spring with RSpec running, our next post will show you how to set up notifications with Growl so you don’t have to keep your terminal window visible while you code.

The post Using Spring with RSpec and Guard to speed up testing appeared first on Eyefodder.

Guard and RSpec : rather fancy

Guard is a command line tool that responds to filesystem change events. You can use it to do all sorts of stuff using one of the many plugins that have been built for it. In our simple example we are going to use it to trigger rspec tests when we change code in our app.

Getting Started

For this we are going to grab an example from the Spex repository where I keep all my code examples. Clone the branch from github:

git clone -b rspec_guard --single-branch https://github.com/eyefodder/spex.git

Confirming the tests run

When you have the code cloned, jump into the ops directory and execute vagrant up to bring up the virtual machine that will run our example. As instructed, run vagrant rsync-auto; this will ensure that if you make changes to the code, they will get synced in the virtual machine (which is where Guard will be running). Now, open a new Terminal window, vagrant ssh into the machine and type the following:

cd /app
rspec

By doing this, we should see our tests run and get the following output:

vagrant@spex:~$ cd /app
vagrant@spex:/app$ rspec
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
....

Finished in 0.29975 seconds (files took 1.65 seconds to load)
4 examples, 0 failures

Top 4 slowest examples (0.2972 seconds, 99.1% of total time):
  Static Pages root page has the expected title
    0.27993 seconds ./spec/integration/static_pages_spec.rb:9
  ApplicationHelper #some_method_to_test returns 'result'
    0.00696 seconds ./spec/helpers.lication_helper_spec.rb:4
  Static Pages root page has a link to my blog
    0.00691 seconds ./spec/integration/static_pages_spec.rb:13
  Static Routing root path routes to the static#home path
    0.0034 seconds ./spec/routing/static_routing_spec.rb:5

Top 3 slowest example groups:
  Static Pages
    0.14342 seconds average (0.28684 seconds / 2 examples) ./spec/integration/static_pages_spec.rb:2
  ApplicationHelper
    0.00696 seconds average (0.00696 seconds / 1 example) ./spec/helpers/application_helper_spec.rb:2
  Static Routing
    0.0034 seconds average (0.0034 seconds / 1 example) ./spec/routing/static_routing_spec.rb:2

Randomized with seed 13496

So what happens if we break something? Go ahead and change the code in app/helpers/application_helper.rb:

  def some_method_to_test
    'resultd'
  end

Now when we run rspec again we see that four tests have run and one failed as expected. But really, given we just changed one file, did we have to run all the tests? And did we have to manually re-run the tests? The answers to these is no; Guard and Rspec together are pretty awesome…

Guard and Rspec together

To get things up and running, type in the following:

bundle exec guard -p

You should see something like this:

vagrant@spex:/app$ bundle exec guard -p
21:18:32 - INFO - Guard is using NotifySend to send notifications.
21:18:32 - INFO - Guard is using TerminalTitle to send notifications.
21:18:32 - INFO - Guard::RSpec is running
21:18:32 - INFO - Guard is now watching at '/app'
[1] guard(main)> 

If you hit the Enter key, you’ll see all your tests run again. This is pretty awesome right? Now you can save yourself having to type rspec. But wait, that’s not all. Go ahead and change your application_helper.rb file back the way it was. Now you should see something like this:

21:22:13 - INFO - Running: spec/helpers/application_helper_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
.

Finished in 0.0103 seconds (files took 1.99 seconds to load)
1 example, 0 failures

Top 1 slowest examples (0.00878 seconds, 85.2% of total time):
  ApplicationHelper #some_method_to_test returns 'result'
    0.00878 seconds ./spec/helpers.lication_helper_spec.rb:4

Randomized with seed 15725


[1] guard(main)> 

Pretty sweet huh? Basically, anytime a file changes Guard figures out what test or tests we need to run. This helps to keep my test suite running lean and fast; aiding rather than slowing down development. Let’s take a look at what we have added in our example to make things work.

How this was setup

First off we added two gems to the Gemfile:

group :development do
  # rspec_guard
  gem 'guard-rspec', require: false
end

group :development, :test do
  # rspec_guard
  gem 'rspec-rails', '~> 3.0.0'
end

Next, we have written some specs which you will find in the spec folder. I’m not going to go into detail on the tests I’ve written, but they should be pretty legible and cover just enough to demonstrate what we need to about Guard. Check out spec/helpers/application_helper.rb, spec/integration/static_pages_spec.rb, and spec/routing/static_routing_spec.rb. The spec/rails_helper.rb and spec/spec_helper.rb files are the ones generated by the rspec-rails gem when we run rails generate rspec:install.

The Guardfile

The real magic of getting Guard and RSpec working together comes from the Guardfile which is used to tell Guard what to watch for, and then what to do with it. I’ve modified the file generated by guard init rspec to demonstrate some common usage patterns based on the way I test. Watch statements follow simple regular expressions to determine when to fire. As you build up your app, you will want to keep this file updated as you may need to introduce new patterns. Let’s take a look through some examples in the file:

Run any spec that changes

Any file that is in the spec folder and ends with ‘_spec.rb’ will get run if it changes:

  # run a spec file if it changes
  watch(%r{^spec/.+_spec\.rb$})

Re-run the spec if the helper files change

spec_helper.rb and rails_helper.rb are files that RSpec uses to setup your specs. So if they change for any reason, it would affect all the tests.

  # re-run if spec helper or rails helper changes
  watch('spec/spec_helper.rb')  { "spec" }
  watch('spec/rails_helper.rb')  { "spec" }

Run the matching spec if something changes in the app folder

As a good catch all, if a file changes in the app folder we want to run the corresponding spec:

  # run a file matching same path with _spec at the end
  # eg /app/models/foo.rb will run /spec/models/foo_spec.rb
  watch(%r{^app/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }

Re-run the suite if support files change

I put RSpec shared examples and custom matchers into the support folder, so if anything in here changes I’ll need to re-run the full suite:

  watch(%r{^spec/support/(.+)\.rb$})  { "spec" }

Integration tests for views and controllers

I have found it simpler to test my views and controllers by creating integration tests that mimic user behaviour (although the tests you see aren’t great examples of that, we’ll be building out better ones in a future example on using Capybara). For this to work, I have a naming structure such that specs related to the foo controller (or views) live in spec/integration/foo_pages_spec.rb. If a controller is pluralized, that is handled well too, so that app/controllers/things_controller.rb will trigger spec/integration/thing_pages_spec.rb:

  watch(/^app\/controllers\/(.+)_controller.rb/) do |m|
    ["spec/integration/#{m[1].singularize}_pages_spec.rb",
     "spec/routing/#{m[1]}_routing_spec.rb"]
  end

And if a view file changes, we run the corresponding integration test using this:

  # if something within a view folder changes, run that spec
  # eg app/views/static/anyfile runs /spec/integration/static_pages_spec.rb
  watch(%r{^app/views/(.+)/}) do |m|
      "spec/integration/#{m[1].singularize}_pages_spec.rb"
  end

If any of the overall layouts change, it could affect any of our integration tests, so we should re-run them all:

  watch(%r{^app/views/layouts/(.*)}) { 'spec/integration' }

Abstract classes

Lastly, you will inevitably end up having abstract classes in your app. If these change you probably want to run all the specs for classes that inherit from it. I only included an example for the application_controller.rb changing but you can extrapolate from here:

  watch('app/controllers/application_controller.rb')  { "spec/integration" }

A word on Guard with polling

The more astute amongst you will notice that Guard is started up with a -p option. This option forces Guard to poll for changes rather than listen for filesystem change notifications. The reason is that with our setup using rsync to synchronize changes in the virtual machine causes Guard to not always respond to changes. Basically an earlier issue with Guard-Rspec was solved by having rspec only run when it sees a file as modified rather than added. However, with rsync it seems that the listen gem sometimes think the file has been added rather than modified. I’m trying to figure out a resolution for this, but in the meantime, polling it is!

But wait, there’s more!

This super-simple example should help you get up and running with Guard and RSpec. With the setup here you would find that every time the specs run the Rails application has to load up. This makes every test run take at least a few seconds. In earlier versions of Rails I used to use Spork to pre-load the framework, but now Rails comes with Spring which is much easier to setup. In my next post I’ll walk you through it; until then happy testing!

The post Getting Started with Guard and RSpec appeared first on Eyefodder.

We think we are getting this...

...but often end up stuck with this

As Mateo mentioned, we are pretty much trained to reuse code wherever we can (and that includes our own—more on that later). The assumption is that just because it’s on google code or GitHub it’s good to go. If you’re not careful you will spend more time working through someone else’s library than you would coding from scratch. In practice the decision to reuse or not is a very delicate balancing act that you can easily get wrong. Here are some things to consider when you are faced with the choice:

Good times to consider reuse

It’s generally accepted to be best in class

No-one is going to regret choosing to use JQuery as a base framework for their javascript app. If you know the library is a de-facto standard for the problem you’re trying to solve you should be in good shape.

You don’t know much about the problem / algorithm

I’m not a statistics guy, and when I decided to code up an application that uses some simple estimates to get a more realistic prediction of development time I needed to calculate the shape of a beta distribution chart. Rather than do it myself, I found a library that saved me having to learn how to code a bunch of statistical formula.

There is an a active community around the code

If there are O’reilly books on the subject, if there are questions (and answers) about the code on stack overflow, then you can probably feel a little more comfortable that when you get stuck, your questions will get answered.

It’s not the main problem you’re trying to solve

Again, in the example of the software estimation app I built, the main problem I was after solving was showing people how you can quickly get a sense of the length of your project, not hardcore statistical analysis on data. I knew I didn’t need to do too much with the statistical analysis, and if it all went wrong, I could back out and use simpler, back of the napkin estimates

When to be wary

The code isn’t being maintained

If the code is marked at version 0.1 and was last updated three years ago, just be aware that the owner has probably moved on to other things and if you find problems, you might end up having to fix someone else’s issues.

You know a lot about the problem domain and could put something together quickly

If I know a problem domain really really well, then I can probably write my own stuff in the time it takes me to learn how someone else has approached the problem. Essentially I can do it quicker than ‘most any learning curve.

The code doesn’t have supporting tests

At the risk of sounding like an agile hitler, if code doesn’t have a decent, usable test harness written against it, I have very little faith in it’s utility. It will mean that I have to proceed more slowly and cautiously and likely not understand half the intent of the code.

It’s your own stuff

So you wrote some stuff on your last project that applies to what you’re doing now? Good on you; should save a tonne of time, right? Think again—just because you wrote it doesn’t mean it’s any better than someone else’s work. This one deserves a little more explanation below

It’s going to be really difficult to back out of later

If the library you’re going to use is going to form the backbone of your application or the way its structured, then you need to be careful about the decision you’re making. Things like an architectural framework deserve careful though because the cost of changing your mind can get prohibitively high.

On re-using your own code

There are a couple of reasons why we need to be wary of our own work. First off, we tend to take a revisionist view on things; we assume that the work we did in the past is completely applicable to the problem we have at hand today. And I don’t know about you but I’ve become a better coder over time, so my old code is not as good or clean as the code I write today. Any time you reuse work, you need to weigh your own code with the same rigour as you would someone else’s. Don’t get me wrong, I’m really not advocating re-writing your own code for every project; what I’m suggesting is that if you are going to reuse your own code, make sure it’s fit for purpose, well-tested and as good as you can make it.

So, what can De Niro teach me about reuse?

The decision about whether to reuse or not is a tough one. You need to invest some time evaluating the library, and at some point make a judgement call on how steep the learning curve is versus just going it alone. The smart thing to do when you decide to reuse code is to make it easy to back out of the decision. As De Niro says in Heat:

Have no attachments; allow nothing to be in your life that you cannot walk out on in thirty seconds flat if you feel the heat around the corner

What I mean by this is that if you are going to use a library or someone else’s code, make sure you have it well isolated in your application. Try to only interact with it in one place, and make sure you have solidly tested that integration point. That way, if you later decide to back out of your decision, you can cleanly, surgically remove the library and replace it with your own work (or vice versa). Lowering the cost of changing your mind is the thing that can make the difference between a happy marriage and a bitter, bitter divorce that bankrupts you in the process.

The post Have you been hurt by code reuse? Learn from De Niro appeared first on Eyefodder.

• Its summer
• Work has been insanely busy
• I’ve been beavering away on a new plugin for eclipse

The plugin is designed with one key goal in mind: shortening the develop-test feedback loop. It integrates eclipse and the flexunit framework to make our lives as developers easier.
The plan is to release the plugin on labs as soon as possible (I’m about to go on holiday for a couple of weeks so it’ll be October at the earliest…) but I thought I’d give you an early heads-up on what I’ve been doing when its been too hot to walk the New York streets…

So, here’s what I’ve done so far:

• Modded the flexunit visual layout to work better in an IDE window
• Setup the plugin to switch test runner when you switch between projects
• Modded the flexunit swf to make it easier to re-run the test by clciking on the bar
• Re-run the test when the test harness is recompiled
• Set it up so that the developer can run a subset of tests
• Set it up so that the test harness mxml is templated and can be distinct per project

Plans for the future:

• integrate more tightly with CI build processes
• autogenerate ANT build script
• ability to choose tests on a per-method rather than per-test class basis
• Incorporate other very cool stuff that flexunit has coming down the line

So, for now we are planning to have an internal release; the plugin is definitely alpha and I’d like to keep it to a limited group to iron out bugs, tighten up my bad Java and find out what a cross-section of our developers need. The plan is to release internally to Adobe Consulting, but if you are a forgiving and patient person who could really benefit from this, drop me a note and we can chat…

The post FlexUnit eclipse plugin alpha appeared first on Eyefodder.

So, when I ran my ant build, all went well until the runTest target executed (or at least tried to execute). I got the following build error:

Execute failed: java.io.IOException: debugPlayerMac.app cannot execute

You see the problem is that on a mac, the standalone player (like other applications on the Mac) is actually a folder containing all sorts of cleverness inside. Ant doesn’t know how to execute a folder, so I was a bit stuck. Until, after lots of digging, I found the answer in an old osflash mailing list archive. Basically I have to use the ‘open’ command to launch the app – something like this:

<exec executable="open">
<arg line="${pathToFlashPlayer}"/>
<arg line="${pathToSWFToPlay}"/>
</exec>

I tried this, and it worked nicely. The problem is that my ant build needs to wait until the test harness runs and closes the player before reading the log and parsing the results for cruisecontrol to consume. In the ant script above, the open command only stalls the ant script until it has finished doing its opening magic, so ant gets upset as the test results haven’t been written yet. I slept on this, then couldn’t think of an answer so I sent an email to the ant users mailing list. Mere minutes later I got a reply that helped me work out how to crack this.
Basically, ant has the ability to run two threads, and move on when they are both complete. So in one thread we launch the player, and in another thread we first wait for the file to be available, then we do a check to see if it contains what we want (I am testing against the flag -----------------TESTRUNNEROUTPUTENDS---------------- which I used in my result printer). It worked like a charm:

<target name="runTest" description="runs the test harness" depends="compileTest">
<parallel>
<exec executable="open" spawn="no">
<arg line="${debugPlayer}"	/>
<arg line="'${testHarness.swf}'"/>
</exec>
<sequential>
<waitfor>
<available file="${flashlog.location}"/>
</waitfor>
<waitfor>
<isfileselected file="${flashlog.location}">
<contains text="-----------------TESTRUNNEROUTPUTENDS----------------"/>
</isfileselected>
</waitfor>
</sequential>
</parallel>
</target>

The last step to making this properly cross platform for cruisecontrol was to have the right <exec> command called depending on the OS. Fortunately there is an os attribute you can specify on <exec> and the task will only run if the os matches the platform you are running the task on. A few minutes later, I had my amended runTest target:

<target name="runTest" description="runs the test harness" depends="compileTest">
<parallel>
<exec executable="${debugPlayerWin}" spawn="no" os="Windows XP">
<arg line="'${testHarness.swf}'"/>
</exec>
<exec executable="open" spawn="no" os="Mac OS X">
<arg line="${debugPlayerMac}"	/>
<arg line="'${testHarness.swf}'"/>
</exec>
<sequential>
<waitfor>
<available file="${flashlog.location}"/>
</waitfor>
<waitfor>
<isfileselected file="${flashlog.location}">
<contains text="-----------------TESTRUNNEROUTPUTENDS----------------"/>
</isfileselected>
</waitfor>
</sequential>
</parallel>
</target>

I’m planning to post my completed project build script and step through it in more detail than I did previously. I’ve updated it quite a lot in the last year, and so far, I think its a lot neater. Before I do that, I’m going to post about my removing the python dependency from my continuous integration process.

The post CruiseControl on the Mac – modifying the build script to work x-platform appeared first on Eyefodder.

Overall

Both frameworks seem to have followed the junit framework, and attempted a port from java to AS3. The good thing about this is that there are very close similarities between usage of the two frameworks, and so switching from one to the other isn’t too much of a ballache. Both setups have the notion of a testRunner and resultPrinter, and I actually found that it was a very simple affair to adapt the code I wrote to use FlexUnit with Cruise Control to work with ASUnit instead.

I did however find a problem when using the ASUnit testListener. When you run a test, the following methods get called on any testListeners you have (in my case its the CruiseControlResultPrinter):

1. startTest
2. If an error / failure occurs:
   • addError
   • addFailure
3. endTest

The problem is that when startTest gets clled, the getCurrentMethod() method of the testCase returns null instead of the testMethod that is about to be executed. If you have had a nosy at my result printer code, you will see that I need this method to return the correct value when startTest gets called so I can append a testCase node with the method name as its ‘name’ attribute. I managed to work around this by allowing a node to be added with a ‘null’ value for its name; then when I try and access the testcase node, it first searches for a node with a matching name attribute, then for a node with a value of null for the name. I didn’t feel very comfortable with this, as it injects a weak link in my code. However, it wasn’t enough for me to strongly favour one framework over the other yet…

The next problem I had with ASUnit was unfortunately for me a show stopper. One of my criteria is that I am able to execute an asynchronous test (as well as asynchronous test setup). Unfortunately I didn’t manage to find a clean solution for doing this in ASUnit. There was a discussion on the mailing list about how one might go about doing this, but unfortunately I couldnt find a solution that worked within my requirements. Luke made a very good point about the difference between asynchronous and event-driven, but nevertheless I still couldnt get an async test to work and correctly report back before the alltestscomplete event gets triggered with the test suite.

One of my other criteria was also that the framework didnt rely on Flex. For me right now, this is secondary as my current work is using Flex. However, it is still important to me to be able to have the ability to build a test suite for a flash based project. As far as I can see, the only real reliance Flexunit has on flex is the testRunner / result printer that ships with the framework. This would mean that if I needed to, I could build a flash only test runner. I think that the name flexunit doesn’t help, and if it were me, I’d be tempted to find a different name.

I also really like the ability to automatically create test suites that ASUnit provides. Johannes pulled out the functionality for this to enable us to do this as a python script, so I guess with different templates, you could get this working for flexunit.

In conclusion

If you have got this far, I guess you will have worked out that I will be going with flexunit for the time being. I’m kind of sad about this – I suppose I’m quite attached to ASUnit, as Ive been using it for a year and a half, and I really like the community spirit behind it. Ultimately though, it doesnt quite do what I need, and I really didnt want to get deeply under the hood of the framework code. I’m going to keep a very open mind on this decision, and see how asunit progresses. If anyone has anything to add to sway my decision one way or the other, I’d love to hear it too…

The post FlexUnit / ASUnit deathmatch results appeared first on Eyefodder.

To acheive this, my Flexunit test harness uses a factory method pattern to decide how torunthe test suite. Here’s how…

I first started by taking the skeleton flexunit testsuite implementation outlined by Darron Schall here. Next, I removed the testRunner MXML node, and changed my creationComplete function to look like this:

private function onCreationComplete():void{
	var testRunner:IFlexUnitUI = FlexUnitUIFactory.createUI();
	var visualTR:UIComponent = testRunner as UIComponent;
	addChild(visualTR)
	visualTR.percentWidth = 100;
	visualTR.percentHeight = 100;
	testRunner.setTest(createSuite());
	testRunner.startTest();
}

OK, so far this doesnt really tell you guys much, so lets delve a little deeper into whats going on… my test runner satisfies an interface that simply allows me to set a test and start a test. This decouples the way we run our tests (and display our results) from the main testHarness. The flexUnitUIFactory
is a very simple interface:

package com.fmr.flexunit
{
	import flexunit.framework.TestSuite;
	public interface IFlexUnitUI
	{
		function setTest(t:TestSuite):void;
		function startTest():void;
	}
}

Now, lets look at the factory itself:

package com.fmr.flexunit
{
	import flash.system.Capabilities;
	//creates an IFlexUnit UI for use by the test harness.
	// the idea is that one can have either visual feedback, or a lightweight text feedback, depending on requirements
	public class FlexUnitUIFactory
	{
		public static function createUI():IFlexUnitUI{
		var environment:String = Capabilities.playerType;
		var outObj:IFlexUnitUI;
		switch(environment){
			case "StandAlone":
				outObj = new TextOutputUI();
				break;
			case "PlugIn":
			default:
				outObj = new FlexUnitUI();
				break;
			}
			return outObj;
		}
	}
}

The factory is actually very simple – it chooses a TextOutputUI if its running in a standalone player (and thus being run as an ANT build), or uses a pretty GUI if in the browser (thus being run by the developer). Each of these UI’s must satisfy our interface, and so I needed to subclass the flexunit GUI to get it to satisfy the interface:

package com.fmr.flexunit
{
	import flexunit.flexui.TestRunnerBase;
	import flash.events.Event;
	import flexunit.framework.TestSuite;
	public class FlexUnitUI extends TestRunnerBase implements IFlexUnitUI
	{
		public static var testsCompleteEvent:String = "testsComplete";
		override public function onAllTestsEnd() : void {
			super.onAllTestsEnd();
			dispatchEvent(new Event(testsCompleteEvent));
		}
		public function setTest(t:TestSuite):void{
			test = t;
		}
	}
}

The TextOutputUI uses our CruiseControl friendly outputting classes, and is simply a VBox (as we need our elememnt to be a IUObject that can be atached to our main app):

<mx:VBox xmlns:mx="http://www.adobe.com/2006/mxml" width="100%" height="100%" implements="com.fmr.flexunit.IFlexUnitUI">
<mx:Script>
<![CDATA[
	import flexunit.framework.TestSuite;
	import flexunit.textui.TestRunner;
	public var test:TestSuite;
	public function setTest(t:TestSuite):void{
		test = t
	}
	public function startTest():void{
		CruiseControlTestRunner.run( test, onTestsComplete );
	}
	private function onTestsComplete():void{
		fscommand("quit");
	}
]]>
</mx:Script>
</mx:VBox>

Notice also that in our textOutputUI, when the tests are complete, the app is told to quit – this is a requirement of our ANT build process as once the unit tests are run, the main app should be built.

And that’s basically it. Using this method, our test harness can run differently depending on its environment. I havent had a chance to test this in the field yet, but I’ll let you know how it all works out…

The post FlexUnit UI factory appeared first on Eyefodder.

I based my work here on how the flexunit.textui.TestRunner and flexunit.textui.ResultPrinter class work. I created two classes: CruiseControlTestRunner and CruiseControlResultPrinter.

CruiseControlTestRunner

This class is incredibly similar to TestRunner. In fact the only difference is that our private variable printer is set up to use our other new class – CruiseControlResultPrinter. If this variable had been declared protected, then I would have subclassed TestRunner and simply changed the setup of the printer to use our new class… You can download the file here

CruiseControlTestPrinter

The printer is a little more involved, but still not too tricky. what I do here is create an XML Object and append results as they come through. TestListeners (the interface that the printer implements) have functions that get triggered when a test starts and ends, and also if a test fails, or if there is an error. These can be used to construct our XML as the suite runs. Lets look at the class bit by bit: (or you can just download the class and get started

Constructor

Dead simple, we just initialize our XML with a top-level node:

public function CruiseControlResultPrinter()
{
	__resultXML = new XML("");
}

startTest

When a test starts, we want to append a testcase node to our XML. Each testcase node actually needs to sit in a testsuite node, so I created a helper function to automatically create a testsiute node for the testcase if one doesnt exist (ie if its the first test run in a particular Test Class):

 public function startTest( test:Test ):void
{
	var suiteNode:XML = getSuiteNode(test)
	suiteNode.appendChild(new XML(""));
	__testTimer = getTimer();
}
//------------------------------------------------------------------------------
private function getSuiteNode (test:Test):XML{
	var outNode:XML = __resultXML.testsuite.(@name==test.className)[0];
	if(outNode==null){
	outNode = new XML("");
	__resultXML.appendChild(outNode);
}
return outNode;
}

addError / addFailure

When a test fails or errors, we need to append failure details to the testcase node. The child node is named either failure or error depending on the type of problem (these nodes get displayed differently in CruiseControl). A child node populated with the stack trace is appended to the testcase node:

public function addError( test:Test, error:Error ):void{
	onFailOrError(test,error,"error");
}
//------------------------------------------------------------------------------
public function addFailure( test:Test, error:AssertionFailedError ):void{
	onFailOrError(test,error,"failure");
}
private function onFailOrError(test:Test,error:Error, failOrError:String):void{
	__suiteSuccess = false;
	var testNode:XML = getTestNode(test);
	var childNode:XML = new XML("&lt;"+failOrError+"&gt;"+error.getStackTrace()+"<!--"+failOrError+"-->");
	testNode.appendChild(childNode);
}
private function getTestNode(test:Test):XML{
	return __resultXML.testsuite.testcase.(@name==TestCase(test).methodName)[0];
}

endTest

If you look at the startTest code, you will see that we set a __testTimer variable. This comes into play in the endTest callback, where we set the execution time of the test Case:

public function endTest( test:Test ):void
{
	var testNode:XML = getTestNode(test);
	testNode.@time = (getTimer() - __testTimer)/1000;
}

print

When the test suite is complete, the TestRunner calls the print function on our class. This is actually now a much simpler affair than the one I previously wrote for ASUnit (although I will be revisiting this for ASUnit for AS3 over the next few days…). It simply traces out our generated XML, and also includes the line at the bottom specifying test Suite success (this is read by our ANT build):

public function print( result:TestResult, runTime:Number ):void
{
	printHeader(runTime);
	printMain();
	printFooter(result);
}
private function printHeader( runTime:Number ):void
{
	trace("-----------------TESTRUNNEROUTPUTBEGINS----------------");
}
private function printMain():void{
	trace(__resultXML);
}
private function printFooter( result:TestResult ):void
{
	trace("Test Suite success: "+(result.errorCount()+result.failureCount()==0)+"n");
	trace("-----------------TESTRUNNEROUTPUTENDS----------------");
}

And that’s basically it.. My next post is going to be about how I actually went about implementing these classes and got the test siute to display results differently depending on whether it was being run by the developer or as part of an ANT build…

The post FlexUnit for Cruise Control – XML output appeared first on Eyefodder.