How I Use Autotest

Eric Hodel | Mon, 26 Nov 2007 22:33:23 GMT

During the Q&A portion of my talk, I was asked a very important question about autotest, something like “How often do you save when using autotest?” I save all the time when using autotest. There was a followup question, something like “Don’t you get syntax errors?” and I don’t.

Before I wrote autotest I was making fine-grained saves that were syntactically correct. I wrote autotest to automate the running of tests so I wouldn’t have to choose which tests to run. My changes were so small that I spent an annoying fraction of my time editing my command line.

When I’m using autotest, rather than writing all of a method’s tests, I start by performing the setup (which should cause an error), add a flunk, and save, which is usually four lines:

  def test_blah
    result = @thingy.blah
    flunk
  end

When the tests run they’ll see the method blah doesn’t exist and fail, so I define the method, (the def blah and end lines), and save. Then I write an assertion and save the test, which will fail, so I implement what I need to make the assertion pass, and save. Now I repeatedly perform the minimum changes to go from failure to flunking until my test is complete, remove the flunk, and start over with the next method.

Each save I do is a handful of lines, so it’s easy to keep them syntactically correct. I know I’m going from working state to working state as I work towards my end goal so I can easily roll back my changes with undo.

Posted in ,  | 1 comment

Firebrigade Home Page Fixed

Eric Hodel | Tue, 18 Sep 2007 02:40:22 GMT

I denormalized a bit and beat the tests back into shape and the Firebrigade home page is back to life! Next I’m going to sand down a few rough edges on RubyGems to get a beta shipped suitable for feedback.

Firebrigade is now fully vladified using perforce.

Here’s the two setup steps you need for perforce:

  1. Require ‘vlad/perforce’ at the top of config/deploy.rb.
  2. Your .p4config goes in the scm/ directory of the server’s checkout.
  3. Run p4 client in the scm/ directory and set your View to //path/to/project/... //clientname/...

Hopefully we can fully automate this so the vlad:setup_app task can handle this automatically.

I also had to do some custom setup for firebrigade because it uses RubyInline and needs the INLINEDIR set. I just added extra stuff to the setup_app task:

namespace :vlad do remote_task :setup_app do cmds = [ "mkdir #{inline_dir}", "sudo chown www:www #{inline_dir}", ] run cmds.join(' && ') end end

Update: Automatic p4 setup is done!

Posted in ,  | 1 comment

Notes on Heckle

Eric Hodel | Mon, 12 Feb 2007 22:32:17 GMT

When I started heckle I was hoping for somehing better than rcov that would tell you that your code is poorly tested… But code so poorly engineered that changing anything as bound to make everything fall apart looks like well tested (read: heckle-proof) code in a sense. We both think that running just the unit tests that directly test the MUT (method under test) would help, but really it wouldn’t help enough. [...]

Suggestions? How would you differentiate between “well tested” and “tightly coupled crap”?

—Notes on Heckle via Polishing Ruby

I’m puzzled by this too. How can we detect well-tested vs tightly-coupled (and poorly tested) code automatically?

Posted in  | no comments

tinderbox version 1.0.0 has been released!

Eric Hodel | Wed, 31 Jan 2007 09:24:00 GMT

tinderbox version 1.0.0 has been released!

http://seattlerb.rubyforge.org/tinderbox

Description

Tinderbox tests projects and tries to make them break by running them on as many different platforms as possible.

Features & Problems

Changes:

1.0.0 / 2007-01-30

  • Tests gems in a sandbox
  • Submits results to Firebrigade
  • Birthday!

http://seattlerb.rubyforge.org/tinderbox

Posted in , , ,  | 4 comments

firebrigade_api version 1.0.0 has been released!

Eric Hodel | Wed, 31 Jan 2007 09:13:02 GMT

http://seattlerb.rubyforge.org/firebrigade_api

firebrigade_api is an API wrapper for http://firebrigade.seattlerb.org

Changes

Posted in , ,  | no comments

Test Profiling by Lines Logged

Eric Hodel | Mon, 22 Jan 2007 23:41:00 GMT

At work I've been cleaning up the tests and trying to make them run faster. One way of doing this is profiling the tests and fixing the slow spots. That only works so well, especially if there's lots of duplication or extra work in the tests. With Rails you can go about this a different way, since you have a second source of information on your tests' operation, the log file.

I added the following code to test/test_helper.rb: 

raise 'require\'d test/test_helper twice!, you broke it!' if
  ENV['RAILS_ENV'] == 'test'

At the very top, even above ENV["RALIS_ENV"] = "test" to make sure that the hack to Test::Unit::TestCase below happens only once, then the magic to figure out which test outputs which log lines: 

class Test::Unit::TestCase
  alias unlogged_run run
  def run(result, &block)
    RAILS_DEFAULT_LOGGER.debug "RUNNING #{self.class} #{@method_name}"
    unlogged_run result, &block
  end
end if ENV['PROFILE_LOG']

Each test run will be prefixed with text like RUNNING SomeTest test_blah which I can then run this script on: 

#!/usr/local/bin/ruby -w

test = nil
tests = Hash.new 0

File.open 'log/test.log' do |fp|
  fp.each_line do |line|
    if line.strip =~ /^RUNNING (.*)/ then
      test = $1
    else
      tests[test] += 1
    end
  end
end

tests.sort_by { |test, count| -count }.each do |test, count|
  puts "%5d %s" % [count, test]
end

To get the profile information, I run: 

rake log:clear; PROFILE_LOG=y rake && script/count_logs

When run on Firebrigade, I get the following output (top 10 only): 

  130 ProjectViewTest test_show
   74 DummyControllerTest test_error_500
   59 OwnerTest test_class_owner_count
   42 ProjectControllerTest test_index
   39 HomeControllerTest test_index_no_builds
   38 RestControllerTest test_add_build
   35 ProjectControllerTest test_show_no_versions
   35 ProjectControllerTest test_search_many_matches
   34 RestControllerTest test_add_project
   33 RestControllerTest test_add_version

Now I can quickly discover good candidates for refactoring. #test_error_500 above dumps an email into the logs, so it is a bogus result. That leaves ProjectViewTest#test_show as a candidate for simplification or refactoring.

Posted in , ,  | 2 comments

Ruby, Rails, Test::Rails Cheat Sheet

Eric Hodel | Thu, 24 Aug 2006 18:07:39 GMT

A cheat sheet that shows the assertions in Test::Unit, the ones added by Rails, and the ones further added by Test::Rails (part of the ZenTest gem).

Ruby, Rails, Test::Rails Cheat Sheet

—Ruby, Rails, Test::Rails Cheat Sheet via Nuby on Rails

Posted in , ,  | no comments

TDD and Creative Flow

Eric Hodel | Tue, 09 May 2006 00:05:52 GMT

A couple weeks ago I came across a discussion of whether test-first development was an impediment to creative flow based on an entry by Frank Sommers.

When I first started testing I found testing was an impediment to my flow. I didn’t know how to write my programs to be easily testable. My methods were too big, too coupled or just plain wrong.

What I did start to see was immediate feedback on whether I was going in the right or wrong direction. If I wrote a test and was quickly able to make it pass I knew I was doing something right. If I wrote a test and couldn’t see a good way of making it pass quickly I knew I was doing something wrong. Getting lost or bogged down quickly destroys flow, so an early-warning system was a big benefit.

When I started down the wrong path I knew before I was in too deep and could rip the code and the test out and either try again or work on some other aspect of the system and let the problem percolate in my brain.

Sommers cites Mihaly Csikszentmihalyi on four ingredients for creative flow:

  • The activity must present just the right amount of challenge: If the activity is too hard, we become frustrated; if too easy, boredom ensues.
  • There must be clear and unambiguous goals in mind.
  • There needs to be clear-cut and immediate feedback about the activity’s success.
  • Finally, our focus during the activity must center on the present – the activity itself.

I eventually came around to having TDD fulfill these requirements much better than the way I was writing code before.

TDD keeps me in the right challenge range. When things become too easy I need to refactor to get rid of the boredom. When things become too hard I’m doing something wrong.

The one I didn’t expect was what the clear and unambiguous goal is. It happens to be getting the next test to pass or fail as appropriate. When they don’t you know you’re doing something wrong.

The clear-cut and immediate feedback is even more immediate with testing. You nearly instantly know if you’re doing the right thing. Did the test fail? Is making the test pass easy? Is writing the next test easy?

Finally, I found myself spending more time writing code and less time chasing down bugs again and again. By testing I reduced distractions and kept developing code at the center of my activity. I no longer have to run through my code sprinkling puts or inspecting in the debugger only to do it over again on the next bug. My tests tell me what’s wrong long before I get to that stage.

Posted in  | 2 comments

ZenTest RDoc

Eric Hodel | Thu, 27 Apr 2006 02:57:29 GMT

The RDoc for your favorite testing toolset is now online at zentest.rubyforge.org/.

Strangely, my uploading tasks updates the RDoc every time I run it. Maybe I have the wrong task on the right hand side.

desc 'Upload RDoc to RubyForge'
task :upload => :rdoc do
  user = "#{ENV['USER']}@rubyforge.org" 
  project = '/var/www/gforge-projects/zentest'
  local_dir = 'doc'
  pub = Rake::SshDirPublisher.new user, project, local_dir
  pub.upload
end

Posted in , ,  | no comments

BDD vs. TDD

Eric Hodel | Tue, 25 Apr 2006 03:41:55 GMT

At both Canada on Rails and the Silicon Valley Ruby Conference I watched talks on Behavior Driven Development with RSpec, the former given by Dave Astels and the latter by Steven Baker. At SVRC one person asked why we need a new testing framework and Steven replied that BDD changes the way you think about TDD.

As I gained experience with TDD my failing test cases expressed the next piece of behavior I wanted my object to have. Learning how to do this took me about a year because I didn’t know how to think properly. BDD is designed to get you to think correctly about what you are testing, but it is hard to communicate how you should think about writing code.

With Test::Unit it is too easy to run off chasing edge cases that typically aren’t important. I did this back when I was new to testing and ended up with big, ugly tests that didn’t make my development much better. RSpec discourages edge-case chasing by expressing the tests at a higher level.

I think the only thing missing from the two presentations was a set of slides that show a small session of BDD like Steven displayed on classic TDD at Canada on Rails. The examples on the RSpec page are fully formed, so you can’t see the specification develop over time along with the code.

Posted in  | 3 comments

Older posts: 1 2