See how happy people are when they have a safety net?

Getting Started

I’ve made a simple example app that shows code coverage in action. Check out the source code from the code_coverage branch of my spex repository:

git clone -b code_coverage --single-branch https://github.com/eyefodder/spex.git

Now go to the ops directory and run vagrant up To get the virtual machine running. Next, let’s hop into the virtual machine and run the test suite:

vagrant ssh
...some output...
vagrant@spex:~$ cd /app
vagrant@spex:~$ rspec

Now, check out the reports folder. You’ll see that there is a coverage/rcov folder. Open the index file in the browser and you see an easy to digest code coverage report:

Pretty nifty huh? You can click on the rows in the table to see each class in more detail, and find out exactly which lines aren’t being executed:

Let’s take a look at how this was all set up…

Code Coverage Gems

First up, we need to add a couple of gems to the Gemfile:

  # code_coverage
  gem 'simplecov', :require => false
  gem 'simplecov-rcov', :require => false

Once we’ve run a bundle install, our next step is to configure our test suite to generate coverage reports.

Configuring Code Coverage

This, again is a pretty simple affair. We need to launch SimpleCov before any application code has run, so we have this code at the top of the spec/spec_helper.rb

if ENV['GENERATE_COVERAGE_REPORTS'] == 'true'
  require 'simplecov'
  require 'simplecov-rcov'
  SimpleCov.start 'rails' do
    coverage_dir ENV['CI_COVERAGE_REPORTS']
  end
  SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
end

There’s a few things happening here. We have a couple of environment variables that tell us if we should create reports: GENERATE_COVERAGE_REPORTS and if we do, where we should put them: CI_COVERAGE_REPORTS. If you’ve followed my earlier post on getting Guard to send Growl notifications, you will know to find these in ops/dotfiles/guest_bash_profile which is a profile automatically generated when we launch the virtual machine with vagrant up. If not, well, now you do!
The next thing you’ll notice is the SimpleCov.start 'rails' call on line 4. This configures SimpleCov to have a profile that is good for most Rails applications. For example, the spec and config folders are excluded from coverage stats. You can read more about profiles here.
Finally, we tell SimpleCov that we want to format our results with the SimpleCov::Formatter::RcovFormatter. When we get to running our build as part of a continuous integration process with Jenkins, we can use this format to parse results to be viewed in the dashboard.

Viewing Code Coverage Reports generated on a Guest VM

The last thing we have to deal with is the fact that the reports are generated on the guest virtual machine. In our existing setup, we use rsync to push code changes from the host to the virtual machine. But this only works one way, and if we add content within the virtual machine you won’t see them on the host. We solve this with these lines in the Vagrantfile

  config.vm.synced_folder '../reports', '/reports'
  config.vm.synced_folder "../", "/app", type: "rsync", rsync__exclude: [".git/", "ops/*", "reports/", "tmp/", "log/", ".#*"]

What this does is exclude the reports from the main rsync and instead setup a new (regular) shared folder that will map reports to /reports on the virtual machine (note this is a root level folder, not in the /app folder on the guest. This is why we have used an environment variable to tell SimpleCov where to output reports.

Beware the emperor’s new code coverage

One thing to bear in mind is that code coverage really is a very crude metric. There are different types of coverage metrics, and SimpleCov only provides ‘C0′ coverage: lines of code that executed. Other types include branch and path coverage, but as far as I know, there aren’t any tools for these in Ruby. Let me show you an example of where this falls down:

If you look at this report, we can see that the some_method_with_conditionals gets called, but only the say_yes path (lines 12 and 13) executes, and we never confirm that ‘no’ gets sent if we pass false to the method. So far, so good, until we look at some_method_with_ternary. This is basically the same method refactored to be more compact, and with the same tests run against it. Yet we are told it is totally covered. So is the metric even still useful?

I still think code coverage is a valuable metric, if only to show you where there are holes in your test suite. If you go in with this knowledge and understanding the limitations, then you will be better equipped to maintain the quality of your app over time.

Code Coverage is a temporal metric

The last thing I want to mention about code coverage is that it’s useful to understand how your coverage changes over time. Particularly if you are managing a team of developers, it provides a quick warning if developers are slipping on their test writing. If you have a Continuous Integration machine, you can track these sort of metrics over time, which can really help you get a sense of where things are headed.
In my next post I’ll show how to set up your very own CI machine with just a few clicks…

The post Code Coverage — a simple Rails example appeared first on Eyefodder.

Growl isn’t as hairy as ‘old Ephraim’ but is probably more useful

Getting Started

To get started and see Growl notifications, we need to do a little more than normal. Let’s start by checking out the source code:

git clone -b guard_growl --single-branch https://github.com/eyefodder/spex.git

Prepping Growl

Download Growl if you don’t already have it. Once it’s installed, open up the preferences pane and click on the ‘network’ panel. Check the box marked ‘Listen for incoming notifications’ and enter a password:

Prepping Vagrant

The next thing we need to do is install a plugin for Vagrant that will allow us to execute a script on the host machine when we boot our virtual machine. I’ll explain what that is below the fold, but for now, make sure that you have Vagrant 1.6.4 or above (hint, type vagrant -v to find out) and update if necessary. Next, install the Vagrant Triggers plugin by entering this on the command line:

$ vagrant plugin install vagrant-triggers

Bringing the virtual machine up

Next, run vagrant up as usual. When it runs, it creates a file ops/dotfiles/guest_bash_profile:

# Edit the following values with your own values

# Growl password set in Growl > Preferences > Network
export GROWL_PASSWORD=enter_growl_password

# The following entries are automatically generated
# Do not edit unless you know what you are doing!
# They are regenerated each time the virtual machine is rebooted
export HOST_IP=10.0.1.28

Go ahead and enter your Growl password. Now we’re good to go. run vagrant rsync-auto to keep things in sync and in another window, then vagrant ssh into the machine.

Hey presto! Growl notifications

So let’s get up and running! When you are ssh’d into the host machine, fire up Guard with cd /app && bundle exec guard -p. Make a change to your code and when the tests run you should see a notification:

You’re good to go now, but read on if you’d like to understand how all this stuff works…

How it works

There are a lot of moving parts to get this thing going. I’m going to work from the Guardfile back, and then the Vagrantfile forwards. I guess we’ll meet somewhere in the middle so bear with me.

Guardfile changes

If you look in the Guardfile you will see a few new lines of code at the top:

host_ip = ENV['HOST_IP']
growl_pass = ENV['GROWL_PASSWORD']
PLACEHOLDER_GROWL_PASS='enter_growl_password'

if host_ip.blank? || growl_pass.blank? || (growl_pass==PLACEHOLDER_GROWL_PASS)
  puts 'host notifcations off: you must set HOST_IP and GROWL_PASSWORD in ops/dotfiles/guest_bash_profile'
  notification :off
else
  notification :gntp, :sticky => false, :host => ENV['HOST_IP'], :password => ENV['GROWL_PASSWORD']
end

Line 23 tells Guard to use GNTP for notifications (assuming HOST_IP and GROWL_PASSWORD have been set). It’s basically a means for sending Growl notifications over a network. You’ll need to add the GNTP gem, so we’ve added this to our Gemfile:

  # guard_growl
  gem 'ruby_gntp'

We can see that this system relies on a couple of environment variables being set. We got a clue that these get set in ops/dotfiles/guest_bash_profile, let’s see how that file gets created, and how we get that linked into the guest Virtual Machine.

Creating the guest bash profile

For this setup to work, we need two environment variables set: GROWL_PASSWORD and HOST_IP. The script that creates the file is ops/setup_guest_bash_profile. This script does a few things, so let’s step through it:

Create a profile file if it doesn’t exist

The first thing we want to do is create a profile file if one doesn’t exist already:

# make a profile file if it doesn't exist
bash_file=dotfiles/guest_bash_profile
touch $bash_file

Next, we want to add some comments to the file. As we will run this script every time the machine boots up we only really want to add these comments if they don’t already exist. The add_comments_once function does this by checking for a match, and only adding the comment if it isn’t already in the file:

add_comment_once(){
  comment=$1
  grep -q "$comment" $bash_file
  if [ $? -ne 0 ]
  then
    echo "# $comment" >> $bash_file
  fi
}

Adding a bash entry for users to change

Next thing we want to do is add environment variables for users to change. But as this code is run multiple times, we actually have three scenarios:

• An entry doesn’t yet exist in the file: create it
• An entry exists but hasn’t been changed from the default: tell user to change it
• A user has entered their own value: don’t do a thing!

The add_bash_entry function achieves this:

add_bash_entry(){
  token=$1
  default=$2
  change_message="You need to enter your $1 in $bash_file"
  entry_comment=$3

  grep -q $token= $bash_file
  if [ $? -eq 0  ]
  then
    grep -q $default $bash_file
    if [ $? -eq 0 ]
    then
      # entry exists, but it's set to default value from this script
      echo $change_message
    fi
  else
    # variable entry doesn't exist
    echo $change_message
    echo $entry_comment
    echo "" >> $bash_file
    echo "# $entry_comment" >> $bash_file
    echo "export $token=$default" >> $bash_file
    echo " " >> $bash_file
  fi
}

The function gets three arguments: the token GROWL_PASSWORD, the default value enter_growl_password, and a message to explain to the user what to do. If the token is found (lines 22 & 23) then it looks for the default value (lines 25 & 26) and prints a change message if it’s there. If the token isn’t found (lines 32-38) then we write that into the file.

Adding the HOST_IP as a system generated value

The next thing we want to do is add the host system’s IP address to the file so that Growl knows where to send notifications:

sed -i '' '/HOST_IP/d' $bash_file
# grab current host IP
ipaddr=`ifconfig | grep "inet " | grep -v 127.0.0.1 | cut -d ' ' -f2`
# and write to file
echo "export HOST_IP=$ipaddr" >> $bash_file

This script deletes any existing lines with HOST_IP in them (line 65), then uses a little bash trickery to find the current host ip. I found out how to do it from this post, although I needed to change the delimiter from \ to ' ' (that’s from an escaped space character to a space in quotes. Finally we write this out to our guest_bash_profile file. The next step is getting this script to run when we want it.

Running the profile setup script when Vagrant starts

Thanks to the Vagrant Triggers plugin, this is a really simple affair. We just add the following to our Vagrantfile:

  config.trigger.before [:up, :reload], :stdout => true do
    run "sh ./setup_guest_bash_profile.sh"
  end

This will run our script every time we call vagrant up or vagrant reload, ensuring that our host ip address is always up to date in the file. The last piece of the puzzle is to make sure we use this file to actually set environment variables on the guest machine.

Linking the guest bash profile to the guest virtual machine

This is a relatively simple two-part process. First thing we do is share the ops/dotfiles directory on the virtual machine:

  config.vm.synced_folder 'dotfiles', '/dotfiles'

Secondly, we want that file symlinked in the guest machine to ~/.bash_profile. I created a new Puppet class to achieve this. Check out ops/puppet/modules/spex/dotfile_symlink.pp:

class spex::dotfile_symlink{
  file { '/home/vagrant/.bash_profile':
    ensure => link,
    target => '/dotfiles/guest_bash_profile'
  }
}

Super simple here. We tell puppet to ensure that /home/vagrant/.bash_profile is a symlink to /dotfiles/guest_bash_profile. In ops/puppet/manifests/default.pp we simply include the class with the others:

include spex::base_packages
include spex::postgres_setup
include spex::ruby_setup
include spex::dotfile_symlink

Now we have everything wired up and ready to go!

Conclusion

This wraps up my example for getting Growl notifications from Guard into the host machine. Although there are a bunch of steps to jump through, once it’s working I hope you’ll find it a pretty robust solution.

The goal in all of this is to shorten feedback loops when you develop. This process should give you some confidence that when your code changes the right tests run. The power of this is greatest when you are confident that your tests cover enough of your application such that you will know when you break things. Our next step is to look into the breadth of your tests and setting up code coverage metrics for your app…

The post Getting Growl notifications from your Virtual Machine appeared first on Eyefodder.

Guard and RSpec : rather fancy

Guard is a command line tool that responds to filesystem change events. You can use it to do all sorts of stuff using one of the many plugins that have been built for it. In our simple example we are going to use it to trigger rspec tests when we change code in our app.

Getting Started

For this we are going to grab an example from the Spex repository where I keep all my code examples. Clone the branch from github:

git clone -b rspec_guard --single-branch https://github.com/eyefodder/spex.git

Confirming the tests run

When you have the code cloned, jump into the ops directory and execute vagrant up to bring up the virtual machine that will run our example. As instructed, run vagrant rsync-auto; this will ensure that if you make changes to the code, they will get synced in the virtual machine (which is where Guard will be running). Now, open a new Terminal window, vagrant ssh into the machine and type the following:

cd /app
rspec

By doing this, we should see our tests run and get the following output:

vagrant@spex:~$ cd /app
vagrant@spex:/app$ rspec
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
....

Finished in 0.29975 seconds (files took 1.65 seconds to load)
4 examples, 0 failures

Top 4 slowest examples (0.2972 seconds, 99.1% of total time):
  Static Pages root page has the expected title
    0.27993 seconds ./spec/integration/static_pages_spec.rb:9
  ApplicationHelper #some_method_to_test returns 'result'
    0.00696 seconds ./spec/helpers.lication_helper_spec.rb:4
  Static Pages root page has a link to my blog
    0.00691 seconds ./spec/integration/static_pages_spec.rb:13
  Static Routing root path routes to the static#home path
    0.0034 seconds ./spec/routing/static_routing_spec.rb:5

Top 3 slowest example groups:
  Static Pages
    0.14342 seconds average (0.28684 seconds / 2 examples) ./spec/integration/static_pages_spec.rb:2
  ApplicationHelper
    0.00696 seconds average (0.00696 seconds / 1 example) ./spec/helpers/application_helper_spec.rb:2
  Static Routing
    0.0034 seconds average (0.0034 seconds / 1 example) ./spec/routing/static_routing_spec.rb:2

Randomized with seed 13496

So what happens if we break something? Go ahead and change the code in app/helpers/application_helper.rb:

  def some_method_to_test
    'resultd'
  end

Now when we run rspec again we see that four tests have run and one failed as expected. But really, given we just changed one file, did we have to run all the tests? And did we have to manually re-run the tests? The answers to these is no; Guard and Rspec together are pretty awesome…

Guard and Rspec together

To get things up and running, type in the following:

bundle exec guard -p

You should see something like this:

vagrant@spex:/app$ bundle exec guard -p
21:18:32 - INFO - Guard is using NotifySend to send notifications.
21:18:32 - INFO - Guard is using TerminalTitle to send notifications.
21:18:32 - INFO - Guard::RSpec is running
21:18:32 - INFO - Guard is now watching at '/app'
[1] guard(main)> 

If you hit the Enter key, you’ll see all your tests run again. This is pretty awesome right? Now you can save yourself having to type rspec. But wait, that’s not all. Go ahead and change your application_helper.rb file back the way it was. Now you should see something like this:

21:22:13 - INFO - Running: spec/helpers/application_helper_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
.

Finished in 0.0103 seconds (files took 1.99 seconds to load)
1 example, 0 failures

Top 1 slowest examples (0.00878 seconds, 85.2% of total time):
  ApplicationHelper #some_method_to_test returns 'result'
    0.00878 seconds ./spec/helpers.lication_helper_spec.rb:4

Randomized with seed 15725


[1] guard(main)> 

Pretty sweet huh? Basically, anytime a file changes Guard figures out what test or tests we need to run. This helps to keep my test suite running lean and fast; aiding rather than slowing down development. Let’s take a look at what we have added in our example to make things work.

How this was setup

First off we added two gems to the Gemfile:

group :development do
  # rspec_guard
  gem 'guard-rspec', require: false
end

group :development, :test do
  # rspec_guard
  gem 'rspec-rails', '~> 3.0.0'
end

Next, we have written some specs which you will find in the spec folder. I’m not going to go into detail on the tests I’ve written, but they should be pretty legible and cover just enough to demonstrate what we need to about Guard. Check out spec/helpers/application_helper.rb, spec/integration/static_pages_spec.rb, and spec/routing/static_routing_spec.rb. The spec/rails_helper.rb and spec/spec_helper.rb files are the ones generated by the rspec-rails gem when we run rails generate rspec:install.

The Guardfile

The real magic of getting Guard and RSpec working together comes from the Guardfile which is used to tell Guard what to watch for, and then what to do with it. I’ve modified the file generated by guard init rspec to demonstrate some common usage patterns based on the way I test. Watch statements follow simple regular expressions to determine when to fire. As you build up your app, you will want to keep this file updated as you may need to introduce new patterns. Let’s take a look through some examples in the file:

Run any spec that changes

Any file that is in the spec folder and ends with ‘_spec.rb’ will get run if it changes:

  # run a spec file if it changes
  watch(%r{^spec/.+_spec\.rb$})

Re-run the spec if the helper files change

spec_helper.rb and rails_helper.rb are files that RSpec uses to setup your specs. So if they change for any reason, it would affect all the tests.

  # re-run if spec helper or rails helper changes
  watch('spec/spec_helper.rb')  { "spec" }
  watch('spec/rails_helper.rb')  { "spec" }

Run the matching spec if something changes in the app folder

As a good catch all, if a file changes in the app folder we want to run the corresponding spec:

  # run a file matching same path with _spec at the end
  # eg /app/models/foo.rb will run /spec/models/foo_spec.rb
  watch(%r{^app/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }

Re-run the suite if support files change

I put RSpec shared examples and custom matchers into the support folder, so if anything in here changes I’ll need to re-run the full suite:

  watch(%r{^spec/support/(.+)\.rb$})  { "spec" }

Integration tests for views and controllers

I have found it simpler to test my views and controllers by creating integration tests that mimic user behaviour (although the tests you see aren’t great examples of that, we’ll be building out better ones in a future example on using Capybara). For this to work, I have a naming structure such that specs related to the foo controller (or views) live in spec/integration/foo_pages_spec.rb. If a controller is pluralized, that is handled well too, so that app/controllers/things_controller.rb will trigger spec/integration/thing_pages_spec.rb:

  watch(/^app\/controllers\/(.+)_controller.rb/) do |m|
    ["spec/integration/#{m[1].singularize}_pages_spec.rb",
     "spec/routing/#{m[1]}_routing_spec.rb"]
  end

And if a view file changes, we run the corresponding integration test using this:

  # if something within a view folder changes, run that spec
  # eg app/views/static/anyfile runs /spec/integration/static_pages_spec.rb
  watch(%r{^app/views/(.+)/}) do |m|
      "spec/integration/#{m[1].singularize}_pages_spec.rb"
  end

If any of the overall layouts change, it could affect any of our integration tests, so we should re-run them all:

  watch(%r{^app/views/layouts/(.*)}) { 'spec/integration' }

Abstract classes

Lastly, you will inevitably end up having abstract classes in your app. If these change you probably want to run all the specs for classes that inherit from it. I only included an example for the application_controller.rb changing but you can extrapolate from here:

  watch('app/controllers/application_controller.rb')  { "spec/integration" }

A word on Guard with polling

The more astute amongst you will notice that Guard is started up with a -p option. This option forces Guard to poll for changes rather than listen for filesystem change notifications. The reason is that with our setup using rsync to synchronize changes in the virtual machine causes Guard to not always respond to changes. Basically an earlier issue with Guard-Rspec was solved by having rspec only run when it sees a file as modified rather than added. However, with rsync it seems that the listen gem sometimes think the file has been added rather than modified. I’m trying to figure out a resolution for this, but in the meantime, polling it is!

But wait, there’s more!

This super-simple example should help you get up and running with Guard and RSpec. With the setup here you would find that every time the specs run the Rails application has to load up. This makes every test run take at least a few seconds. In earlier versions of Rails I used to use Spork to pre-load the framework, but now Rails comes with Spring which is much easier to setup. In my next post I’ll walk you through it; until then happy testing!

The post Getting Started with Guard and RSpec appeared first on Eyefodder.

Don't overpromise and foreclose on your project

Performance as money

I first was introduced to the idea of performance as money through watching an MIT lecture series on algorithms. In the opening lecture, Charles Leiserson talks about performance as currency:

A good way I think about performance, and the reason it’s on the bottom of the heap, is sort of like performance is like money, it’s like currency. You say what good does a stack of hundred dollar bills do for you? Would you rather have food or water or shelter or whatever? And you’re willing to pay those hundred dollar bills, if you have hundred dollar bills, for that commodity.

when he talks about performance being on the bottom of the heap, what he’s talking about is how other things are more important than performance; features, security, ease of use and so on.

Performance constraints as the debt limit

I’ve posted before on non functional requirements every application should have, and first and foremost amongst those constraints is performance. The deal you need to make with yourself and your users is how much of that performance stack of bills are you willing or able to spend to get there? all of the decisions you make, from language choice to functionality to visual design are informed by this decision. Setting your performance constraint is just like setting your household budget (or the government’s debt ceiling).

And the credit rating?

To extend the metaphor a little bit further than is natural, the credit rating is akin to the likelihood that your project will fail overall. We don’t have a good measure for that in software development, but I do think drawing the parallel is interesting. If you read the S&P downgrade overview, you will see that the decision was in part due to the rising debt burden (technical debt?), but more significantly it is due of the infighting and lack of a clear plan forward:

More broadly, the downgrade reflects our view that the effectiveness, stability, and predictability of American policymaking and political institutions have weakened…

This sounds so much like entrenched behavior of stakeholders on a poorly managed software projects (and its effect) that I couldn’t pass on the opportunity to use it. Simply put, if you have a lack of alignment and clear plan about what is acceptable, what is important and how you are going to spend your resources (or generate income) things aren’t going to be plain sailing.

Performance—Avoid foreclosure

O.K. I really am stretching it here, but I wanted to finish up with how I think you best avoid getting mired in software rot or simply just building something that no-one wants to use. And it’s really similar to being financially prudent:

• Understand that almost everything you do bears some cost to performance
• Decide what is an acceptable level of performance…
• …and spend within your means
• Don’t go into debt(let performance drop below acceptable level)
• But sometimes it’s going to happen. If it does, get a plan in place quickly to address it.

The post Performance – spend it wisely and never raise the debt ceiling appeared first on Eyefodder.

We think we are getting this...

...but often end up stuck with this

As Mateo mentioned, we are pretty much trained to reuse code wherever we can (and that includes our own—more on that later). The assumption is that just because it’s on google code or GitHub it’s good to go. If you’re not careful you will spend more time working through someone else’s library than you would coding from scratch. In practice the decision to reuse or not is a very delicate balancing act that you can easily get wrong. Here are some things to consider when you are faced with the choice:

Good times to consider reuse

It’s generally accepted to be best in class

No-one is going to regret choosing to use JQuery as a base framework for their javascript app. If you know the library is a de-facto standard for the problem you’re trying to solve you should be in good shape.

You don’t know much about the problem / algorithm

I’m not a statistics guy, and when I decided to code up an application that uses some simple estimates to get a more realistic prediction of development time I needed to calculate the shape of a beta distribution chart. Rather than do it myself, I found a library that saved me having to learn how to code a bunch of statistical formula.

There is an a active community around the code

If there are O’reilly books on the subject, if there are questions (and answers) about the code on stack overflow, then you can probably feel a little more comfortable that when you get stuck, your questions will get answered.

It’s not the main problem you’re trying to solve

Again, in the example of the software estimation app I built, the main problem I was after solving was showing people how you can quickly get a sense of the length of your project, not hardcore statistical analysis on data. I knew I didn’t need to do too much with the statistical analysis, and if it all went wrong, I could back out and use simpler, back of the napkin estimates

When to be wary

The code isn’t being maintained

If the code is marked at version 0.1 and was last updated three years ago, just be aware that the owner has probably moved on to other things and if you find problems, you might end up having to fix someone else’s issues.

You know a lot about the problem domain and could put something together quickly

If I know a problem domain really really well, then I can probably write my own stuff in the time it takes me to learn how someone else has approached the problem. Essentially I can do it quicker than ‘most any learning curve.

The code doesn’t have supporting tests

At the risk of sounding like an agile hitler, if code doesn’t have a decent, usable test harness written against it, I have very little faith in it’s utility. It will mean that I have to proceed more slowly and cautiously and likely not understand half the intent of the code.

It’s your own stuff

So you wrote some stuff on your last project that applies to what you’re doing now? Good on you; should save a tonne of time, right? Think again—just because you wrote it doesn’t mean it’s any better than someone else’s work. This one deserves a little more explanation below

It’s going to be really difficult to back out of later

If the library you’re going to use is going to form the backbone of your application or the way its structured, then you need to be careful about the decision you’re making. Things like an architectural framework deserve careful though because the cost of changing your mind can get prohibitively high.

On re-using your own code

There are a couple of reasons why we need to be wary of our own work. First off, we tend to take a revisionist view on things; we assume that the work we did in the past is completely applicable to the problem we have at hand today. And I don’t know about you but I’ve become a better coder over time, so my old code is not as good or clean as the code I write today. Any time you reuse work, you need to weigh your own code with the same rigour as you would someone else’s. Don’t get me wrong, I’m really not advocating re-writing your own code for every project; what I’m suggesting is that if you are going to reuse your own code, make sure it’s fit for purpose, well-tested and as good as you can make it.

So, what can De Niro teach me about reuse?

The decision about whether to reuse or not is a tough one. You need to invest some time evaluating the library, and at some point make a judgement call on how steep the learning curve is versus just going it alone. The smart thing to do when you decide to reuse code is to make it easy to back out of the decision. As De Niro says in Heat:

Have no attachments; allow nothing to be in your life that you cannot walk out on in thirty seconds flat if you feel the heat around the corner

What I mean by this is that if you are going to use a library or someone else’s code, make sure you have it well isolated in your application. Try to only interact with it in one place, and make sure you have solidly tested that integration point. That way, if you later decide to back out of your decision, you can cleanly, surgically remove the library and replace it with your own work (or vice versa). Lowering the cost of changing your mind is the thing that can make the difference between a happy marriage and a bitter, bitter divorce that bankrupts you in the process.

The post Have you been hurt by code reuse? Learn from De Niro appeared first on Eyefodder.

The estimation app in action

The application

The application is really simple. You just break down your project into a few tasks and enter three estimates for each—Optimistic, Nominal and Pessimistic (To get what I mean about these terms, check out this post). What it then does is calculate the expected duration of the tasks as well as a high and low end of a range that you can have 95% confidence in. On top of that, it consolidates all of your estimates into a rolled up view that tells you how long you can expect your full project to take. See, there can be some science behind software estimation after all (if you want the details behind how the estimates are worked out, they’re covered in a previous post).

The sparklines

One of the things I wanted to do over and above a spreadsheet was to make something that was clean and simple to read yet packed a lot of information into each row. What originally was six data points has been reduced to two thanks to using sparklines. I could have possibly also done away with the low and high end estimates and had them as rollovers (like in the main result chart) but I just found it to be too overloaded and hard to read. I’ve found that there has been some really nice outcomes from using the sparklines:

• I can see which tasks have the most uncertainty in them (those with the broadest range)
• I can see how different tasks stack up against one another by comparing the sparklines vertically (they’re all on the same x axis scale). I can do this much quicker than if it were a numeric table
• I can present a hierarchy of information. The ranges are most important, then the expected time (which you can see on rollover), followed by the variance and the shape of the estimate which you can infer from the shape of the chart

The results script

One of my big frustrations is that when people come up with software estimates, they give you values that imply a far greater precision than is reasonable. I’ve seen software estimates for a project in terms like “3042.5 hours” What this does is imply that the estimate is between 3042.45 and 3042.55 hours. Far more likely is that you have added up a number of estimates and at best you might be precise to within ten, twenty or even fifty hours (tough to say without knowing more about the estimate, but you get the drift). With the results script, I’ve tried to represent this by using natural language to lessen the implied precision. A few examples:

• Anything less than 1 is represented as “A few hours” “less than a day”
• Up to a week you have “A couple of days” or “A few days”
• Up to a month you have “About a week, a couple of weeks, three weeks
• More than that, time is measured in months

While looking into this, I also found this article talking about the psychology of the zero. In essence it says that people are more likely to ‘accept’ a number with implied precision than one without. If you want to try this out and use the calculated numbers, rolling over the dots on the results plot will give you them. I also left in the calculated numbers for the individual tasks so you can see what’s going in to the final estimate.

What’s next?

Well, I’m hoping you’ll tell me. Right now, all estimates are in days and part days, but I thought I could add an option so you could put estimates in hours for smaller tasks. Maybe some export options or something. If you use this app and get some value from it, let me know if there’s anything else you’d like to see it do. So go ahead and try it out and let me know what you think!

The post Try this, my mini-application for software estimation appeared first on Eyefodder.

How long would it take you to build “the free world’s most formidable weapon”?

The real reason why your project will take longer than you think

What the Navy realized is that when you give a baseline estimate, the number you tend to give is how long you think it will take in ordinary circumstances; the outcome that will happen most of the time. This isn’t however the average amount of time the task will take. How so? Consider this extract from Bob Martin’s excellent book The Clean Coder where he describes a typical software estimation conversation:

Mike: “How likely is it that you’ll be done in three days?

Peter: “Pretty likely.”

Mike: “Can you put a number on it?”

Peter: “Fifty or sixty percent.”

Mike: “So there’s a good chance that it’ll take you four days.”

Peter: “Yes, in fact it might even take me five or six, though I doubt it.”

Mike: “How much do you doubt it?”

Peter: “Oh, I don’t know … I’m ninety-five percent certain I’ll be done before six days have passed.”

Mike: “You mean it might be seven days?”

Peter: “Well, only if everything goes wrong. Heck, if everything goes wrong, it could take me ten or even eleven days. But it’s not very likely that so much will go wrong.”

As Uncle Bob goes on to explain, what’s going on here is that Mike is describing a probability distribution, something like this:

What is happening is that when you describe how long you think it will take, you are describing the mode – what will happen most often. However because there is more chance of running over than under, you need to take that into account. What the Navy did was find a really quick simple way of finding the mean by looking at three simple estimates for the duration of a task…

Three little estimates

They figured out that if you can get people to estimate these three possible outcomes, you can calculate the expected duration for a task:

Nominal (ψ)

This is the standard issue estimate, what people are used to giving—the amount of time you would expect a task to take under normal circumstances. In the example above it would probably be 3.

Nukem (ω)

This is the longest it could take if pretty much everything went wrong. This is the one in a thousand worst case scenario, the Nukem (for more on this read my last post). Mike’s Nukem is 11.

Mu-nicorn (μ)

I took extreme license with naming this one, but bear with me. This is the your estimate of how long this would take if the stars aligned, unicorns pair programmed with you and generally everything went so well you have to check yourself.

Super Simple Math

Now you have these estimates, you can calculate the expected duration of the task using this formula:

T_e = (μ + 4ψ + ω)/6

You can also get the standard deviation of the estimate, which gives a measure of how uncertain the task is:

σ = (ω – μ )/6

Now, you can tell people that you are pretty confident that your task will take between T_e-2σ and T_e+2σ. This statement is not strictly correct, and I will explain at the end why this is so, but to keep it simple, I would use this.

Putting it to practice

In order to put this into on your project I would recommend breaking your app up into somewhat smaller, more manageable pieces for estimating. In theory, this helps because the law of large numbers implies that your errors cancel out. In practice, it’s also just plain easier to understand your app in small pieces. When you have your estimates for a number of tasks, you can simply add the expected times together to get the expected duration for the project. To get the standard deviation, you have to calculate the square root of the sum of the squares:
σ _project = √ ∑ σ ²

Conclusion

There is nothing like real development data and experience to understand how long things are going to take. Unfortunately you will often have a situation when you don’t have that information available and you are still called upon to give a estimate. This technique can help you do that and have solid reasoning to back you up. I hope you find this useful, and if you want to look into estimation more, I highly recommend both Bob Martins book The Clean Coder mentioned above and Mike Cohn’s book on Agile Estimating and Planning

Post-script – there (probably) be dragons down here

This gets a little involved down here, so if you want to keep it simple, don’t worry about this stuff. As you are usually at the beginning of a project when you do this and are wildly imprecise anyway it shouldn’t make much difference.

The more astute amongst you will have seen an issue with how we worked out our range by simply adding / subtracting two standard deviations from the estimated duration of the project. That assumes that its equally likely we run over as it is that we run under—that is, it’s a normal distribution like this:

Trouble is, as we saw in the example, there is a tail off to the right. This is what’s known as a beta distribution (or at least an example of one), something more like this:

If you look at these overlaid onto one another, you see that the range estimate when using normal distribution actually underestimates the duration a bit. If you want to keep things simple you are probably ok, but it’s worth knowing:

The shape of the distribution is called a beta distribution, and it makes the math for working out the range quite a bit more complicated. You need to calculate a couple of variables that describe the shape of the curve: α and β. These are a bit complicated to work out, but for the sake of completeness, I hunted them down for you here. They are:

α = ((mean-min)/(max-min)) * ((((mean-min)*(max-mean))/σ²)-1)

β = (max-mean)/(mean-min)*α

Once you have these, you can use the Excel BETAINV function to get your minimum and maximum ranges. For example, if you want to calculate an duration as a range you can have 95% confidence in:

low end of range = BETAINV(0.025,α,β,min,max)

high end of range = BETAINV(0.975,α,β,min,max)

The post Software Estimation — A good simple way courtesy of the Navy and the cold war appeared first on Eyefodder.

Duke Nukem Forever Launched after 14 years development; what can this big chested antihero teach you about your project?

Why it overran

There are a number of reasons why the Duke took so long to get his politically incorrect ass back out in the wild. As an adoring public were salivating at the thought of a new Nukem, the development team and its management were stuck in a pattern of events that pretty much ensured a death march of a project:

Scope Creep

Anyone who has ever been on a development project will be familiar with this problem. After the success of Duke Nukem 3D, George Broussard—the game’s creator—kept seeing opportunities for ever more awesome features to put in the experience:

Broussard simply couldn’t tolerate the idea of Duke Nukem Forever coming out with anything other than the latest and greatest technology and awe-inspiring gameplay. He didn’t just want it to be good. It had to surpass every other game that had ever existed, the same way the original Duke Nukem 3D had.

Snow scenes? You got it. Online Multiplayer? Of course! These sorts of things may seem price-of-admission now, but bear in mind that these features were being added over ten years ago. And – I’m not saying that these were bad ideas or unnecessary ideas, just that the goalposts kept moving, which makes it kind of tough to finish.

Technology Death Spiral

When Duke Nukem 3D came out, it was built on a game engine named “build” by a child prodigy called Ken Silverman. In order to keep up with ever increasing needs of the project (see above), Broussard swapped out for a new game engine not once, but twice. This, again, may have been necessary, but it meant that the development team were always playing catch up.

Too much cash, not enough business

Here’s the thing: Duke Nukem 3D was a runaway success. So much so, that unlike before, Broussard could basically fund development themselves. Creative freedom is great, but without the business reminding you that what your building is for people to use and enjoy you run the risk of getting stuck into some sort of artwank vortex.

Fear of Success—gilding the lily

I’m not a console game developer, but when I heard it takes up to a year from “locking a game down” to launch I really felt sorry for those guys. The projects I work on have some level of locking down code to close out the last remaining bugs, but if it took a year, I can see how it would be an almost paralysing decision to make.

You’re no different from Nukem

To some extent or another, your project will have some of these issues. Or the server will blow up. If you don’t have complete freedom and never know when to finish the business will be on your ass. Or the IT department. Or someone else’s sucky APIs will kill you slowly as you realise you’re the first person to use them. Or you will get a fundamental set of assumptions so wrong that you need to reset. Or you hire someone that turns out to be worse than useless (and yeah – I know you will fire them quickly, but it will still suck energy and time from the team).

Once you realize that some or all of these things will happen to your project, you can start to use this information to your advantage…

Duke Nukem Forever is at the limit of how long things will take

See here’s the thing: Duke Nukem is pretty much at the outer limit of how long you can get away with a project overrunning. Any longer and it would have been be laid to rest. If it weren’t for the cult-like following DNF’s only achievement would have been 3 Vaporware of the year awards (and a lifetime award to boot). Gearbox Software, the company that picked up development after the original developer went bankrupt probably realized that they were better off launching a mediocre game and starting from scratch than suffering yet another rewrite. This gives them the opportunity to reset the ‘development time’ clock.

Enter the Nukem as a unit of development time

With all this in mind, I propose that we consider using the lesson of Duke Nukem Forever to give us a new unit of software estimation:

A Nukem: The amount of time a project or part thereof will take when pretty much anything that can go wrong does.

So – how long will your next project take?

I can’t be precise, but I can be accurate. This is how long your next project will take longer than you expect it to to and less than a Nukem. Or, in mathematical terms:

ψ < x < ω

Where:

ψ = the amount of time you expect it to take

x = the amount of time it will actually take

ω = a Nukem

What’s interesting here is that many of the issues that Nukem had weren’t really down to code or developers, yet developers are the ones who get asked to estimate project duration and get held to it. Oh well, tough shit, that’s what you get for being a creator. Broussard’s stock response of “screw you publisher, it’ll be ready when it’s ready” actually ended up killing his baby. Of course if you go to someone who is paying the bills with this range, you’ll probably not get vey far. But there is hope. In my next post I’ll show you how you can use this information to arrive at a range that is more palatable to them, has some science behind it, and allows the business to make some decisions based on your assumptions.

The post Your project will take longer than you think and less than a Nukem appeared first on Eyefodder.

The trouble with typical non functional requirements

“it’s not fast enough…” That was the response I always had with my client / UX lead when I was developing when I asked them what they thought of the application. The trouble was, the client was used to seeing a vision demo which had canned data and used smoke and mirrors to mimic the application without having to worry about anything like network latency or the size of data. During development, the software you are building will go through various degrees of speed and performance as you strike a balance between functionality, flexibility and speed. In order for you not to get the shaft as a much harried developer, you need to arrive at an understanding with the client about what performance is acceptable and in what conditions.

Make your requirements SMART

First off, the creation and agreement of how the application needs to behave has to be a conversation between UX, the business and the technical team; you all fail if they are crappy, and you all need to sign up to them. What you need to do is come away with a clearly defined set of goals for ‘quality’ of the application. Like any other requirement, they should be clear, simple and understandable. A great way to check this is to use the SMART principle first written up by Mike Mannion and Barry Keepence in a 1995 paper on this. Basically they suggest that requirements should be:

Specific - Clear, consistent and simple
Measurable – If you can’t measure it, you won’t know when you’ve acheived it
Attainable – The requirement must be something that the team thinks is acheivable given what they know
Realizable – Can it actually be done given the constraints you know about the way the project is being executed (is there enough money / time to acheive this goal for example)
Traceable – It is important to understand why a requirement exists so that it can be justified (and to question if it is still needed if the original driver or assumption changes)

A common structure for defining the requirements

I did quite a bit of hunting looking for a way that I could describe requirements as diverse as conformance to coding standards and ability to cope with year-on-year growth needs. Fortunately I found this post by Ryan Shriver that talks about how he uses a method by Tom Gilb

[requirements] can 1) be specified numerically and 2) systems can be engineered to meet specific levels of performance. Qualities can be specified using a minimum of six attributes:

Name: A unique name for the quality
Scale: “What” you’ll measure (aka the units of measure, such as seconds)
Meter: “How” you’ll measure (aka the device you’ll use to obtain measurements)
Target: The level of performance you’re aiming to achieve (how good it can be)
Constraint: The level of performance you’re trying to avoid (how bad it can be)
Benchmark: Your current level of performance.

Some sample requirements and reports

I think it’s best to take a look at a few examples from my last post on essential non functional requirements for building quality software. What I’m showing here is how I would specify test and report the constraint:

Processing times

First off a relatively simple one. Measuring processing time is a matter of allowing the system to log out how long it took to do stuff.

Name: Parsing of a large search result
Scale: Elapsed milliseconds from receiving an XML response in the client containing 1000 results
Meter: Log messages with a defined signature that output the parsing time in milliseconds
Target: <1200ms (i.e. 1200 ms or less is a target time)
Constraint: >5000 ms (i.e. more than 5000 ms is an epic fail)
Benchmark: Build 155; test environment 3/21/11: 4500ms

Reporting this over time could give a chart something like this:

Architectural standards

This one seems a bit more tricky at first glance. How can you measure how compliant a codebase is against architectural standards? Fortunately there are some great static analysis tools out there that allow you to inspect your code for potential problems. We have used PMD and FlexPMD in our projects. You can write your own rules or draw from common rulesets. Using this we are able to reduce things like architectural conformance to something quantifiable and thus testable:

Name: Conformance to standard Flex architectural best practices
Scale: # of PMD high warnings from the ‘cairngorm’ and ‘architecture’ PMD rulesets
Meter: Flex PMD warnings reported by the nightly Hudson Jenkins build divided by the number of classes
Target: <1200ms (i.e. 1200 ms or less is a target time)
Constraint: >5000 ms (i.e. more than 5000 ms is an epic fail)
Benchmark: Build 155; test environment 3/21/11: 4500ms

You can see here how architectural standards started to lapse in sprint 3. The good news was that it was caught and addressed. This sort of conversation is far easier to have with your team when you have data to support the conversation.

Year on year growth requirements

Last one to demonstrate some breadth. If you want to test growth requirements, you are basically going to have to think about what your likely point of failure is going to be in your other requirements as your user base grows. It could be storage needs or response times, or more likely it’s peak throughput. In the example below, we increase the standard load on the system until the throughput test fails. We use that to predict how much runway the current system has before breaking down.

Name: 60% year-on-year growth
Scale: Number of months for system to hit peak throughput failure
Meter: Standard Load Scenario increased by 5% per month and measured in each scenario until failure
Target: 8 months
Constraint: 3 months
Benchmark: Current application; 3/21/11; Production environment; 2 months

Simplicity and Consistency

This pretty much wraps it up for non functional requirements (and FWIW, I agree with Mike Cohn that we should be calling these things constraints). Non functional requirements can be a gnarly area as they attempt to describe areas of the software most prone to gradual decay and entropy, but I think these are the areas that are important to test if you are serious about building decent quality software. What I like about the approach is that the output reports are nice, clean and simple. They allow you to report on really varied requirements in a consistent manner and allow you to have a data driven conversation if things start to slide.

The post Making Quality Software: how to test non-functional requirements appeared first on Eyefodder.

How I arrived at this list

I started to look at this a little while back when I was putting together our playbook for how we create software for our customers by bringing together agile and user centred design together. We articulate our requirements as User Stories, but outside of the functional aspects of the system, I realized that there are a common set of non-functional requirements that we saw the need for over and over again. Mike Griffiths has put together a great checklist of non-functional requirements, but I felt that a number of them could be covered off and scheduled in standard stories. What follows are the ones that to me are about how a development team needs to think and behave over time:

Performance

This one shouldn’t come as a surprise. Quality software has to be fast. Or at least feel fast. As a front-end guy, this is the one I always feel first. “It’s not fast enough” is a battle I never want to get into. I had this reported as a bug against one of my projects, but the client wouldn’t specify what “fast enough” was; talk about moving goalposts. When you think about an app being performant, thing about specifying the following:

1. Response times

How long should your app take to load? What about screen refresh times or choreography?

2. Processing times

Can I get a spinning beachball please? How long is acceptable to perform key functions or export / import data?

3. Query and Reporting times

This could be covered off with general reporting times, but if you’re providing an API you should probably consider acceptable query times too.

Capacity and Scalability

How much do you need to cope with now, and how much do you think you’ll need to cope with in the future. Unless you are truly blessed with an incredibly forgiving audience (or a great illustrator), you won’t be able to get away with your own fail whale.

Your audience had better be pretty charmed by you if you can get away with regularly futzing on capacity

4. Throughput

Think about how many transactions your system needs to handle. A thousand a day? A million? When Amazon solved this for their needs, they decoupled systems and created a queue service that became the foundation of AWS.

5. Storage

How much data are you going to need to store to do the awesomeness you need it to do?

6. Growth Requirements

This is a tough one, because truly you don’t know how popular your app is going to be until it’s out there. But you can bet (or hope) that someone has made predictions about how wildly successful your app is going to be. Be wary of over engineering here, but at least make sure you aren’t constantly laying down track in front of a moving train.

Availability

Does your system need to be available all the time, and to everyone? Most of the time you can get away with the occasional vacation, but knowing what’s important helps you make good decisions. Quality software ain’t quality if its down when your precious users need it

Can you get away with a product that is rarely available?

7. Hours of operation

When does your app need to be available? If you need to do a database upgrade or a system backup, can you take the system offline while you do it?

8. Locations of operation

A few things to think about here: Geographic location, connection requirements and the restrictions of a local network prevail. If you are building a killer app for use behind the corporate firewall, you’d better make damn sure you aren’t using any exotic ports.

Maintainability

When you’re running a software project you are at constant war against software entropy. This is your chance to save yourself from having to live with shitty code, or leave shitty code for others to pile more shit onto. I’m passionate about this; can you tell?

Don’t live with broken windows

9. Architectural Standards

What’s important here is that the code is going to feel like a consistent body of work, not some ad-hoc glom of a bunch of developers’ whims. Define what the standards need to be else you’ll always be discussing the merits of different styles.

10. Coding Standards

Again – decide on the coding standards for the application and stick to them!

Recovery

You are prepared for a disaster, right? Right? And if a disaster happens, how long’s it going to take to get back up and running?

Backing up your stuff somewhere is smart, but how long will it take to get it out of storage?

11. Restore time

I know you don’t want to think about the worst happening, but if it does, ask yourself how long should it take to get back up and running?

12. Backup time

Speaking of which, how long does it take to back your data?

It doesn’t need to be all doom and gloom

I know when I first put together this list, I felt a bit deflated at all the extra specification I was laying onto my magic development world. In truth though it’s not all that bad. In fact I’ve done the hard legwork for you in editing down the list. If you just think through these things early on in development, you can be pretty sure you are going to save yourself either under- or over-engineering your app, and be much more likely to produce decent quality software that you can be proud of.

The post Making Quality Software: 12 non functional requirements every app should have. appeared first on Eyefodder.