How I Use Autotest
drbrain |
During the Q&A portion of my talk, I was asked a very important question about autotest, something like “How often do you save when using autotest?” I save all the time when using autotest. There was a followup question, something like “Don’t you get syntax errors?” and I don’t.
Before I wrote autotest I was making fine-grained saves that were syntactically correct. I wrote autotest to automate the running of tests so I wouldn’t have to choose which tests to run. My changes were so small that I spent an annoying fraction of my time editing my command line.
When I’m using autotest, rather than writing all of a method’s tests, I start by performing the setup (which should cause an error), add a flunk, and save, which is usually four lines:
def test_blah
result = @thingy.blah
flunk
endWhen the tests run they’ll see the method blah doesn’t exist and fail, so I define the method, (the def blah and end lines), and save. Then I write an assertion and save the test, which will fail, so I implement what I need to make the assertion pass, and save. Now I repeatedly perform the minimum changes to go from failure to flunking until my test is complete, remove the flunk, and start over with the next method.
Each save I do is a handful of lines, so it’s easy to keep them syntactically correct. I know I’m going from working state to working state as I work towards my end goal so I can easily roll back my changes with undo.
Ruby, Rails, Test::Rails Cheat Sheet
drbrain |
A cheat sheet that shows the assertions in Test::Unit, the ones added by Rails, and the ones further added by Test::Rails (part of the ZenTest gem).
Ruby, Rails, Test::Rails Cheat Sheet
ZenTest RDoc
drbrain |
The RDoc for your favorite testing toolset is now online at zentest.rubyforge.org/.
Strangely, my uploading tasks updates the RDoc every time I run it. Maybe I have the wrong task on the right hand side.
desc 'Upload RDoc to RubyForge'
task :upload => :rdoc do
user = "#{ENV['USER']}@rubyforge.org"
project = '/var/www/gforge-projects/zentest'
local_dir = 'doc'
pub = Rake::SshDirPublisher.new user, project, local_dir
pub.upload
endautotest Sucks
drbrain |
You really might think this is a strange thing to say, despite how awesome some people say it is, it still sucks. It doesn’t suck because it doesn’t work well, it sucks because its insides still bear the scars of its birth.
My first version of autotest was written at OOPSLA 2005 after seeing Don Roberts and John Brandt show off a Smalltalk class browser that would automatically run tests whenever methods where changed.
At that time I was TDDing a personal Rails project and realized a tool to automatically run my tests as I made changes would probably speed up my development as much as switching to TDD from web browser reloading development did, so I wrote the first version of autotest.
That first version of autotest was probably about 100 lines of code. It was tied directly to my development process and got stuck occasionally, but it worked well for me.
Sometime in January I imported autotest into ZenTest and made it work for more generic ruby code. To do that I took my tiny ruby script and wrapped it up in a class. I pulled the Rails stuff out into a subclass and made a few other cleanups.
This didn’t do anything for the cleanliness of the code, but I did manage to add a bunch of tests that I hadn’t during my OOPSLA coding spree. I still had bugs though, and those took another four months to shake out with a few minor refactorings along the way.
Now autotest is nearly perfect functionally, but the implementation sucks. Some methods that do two or three things when they should be doing one. The test running algorithm is scattered across several methods.
autotest needs a major refactoring and I’m hoping to get to it in May. Refactoring will make the code cleaner, more straightforward and more maintainable. I may even do something to better support custom testing styles. I could add features to autotest, but that would only make the refactoring harder, so I won’t while adding those features will make the code more convoluted.
As autotest accumulated more suck it became less fun to work on. I don’t like the sound of software in pain so I’m going to listen and fix its suckage before adding any more fanciness.
Supercharged Rails Development
drbrain |
Geoff brings us a movie of autotest in action and has this to say:
My development process has recently been supercharged by autotest, a part of the ZenTest package.
[…]
The revolutionary part of this is that it speeds development by helping you develop without needing to open your web browser! I find myself thinking more about the functional issues that need to be solved rather than the placement of an image or the color of a link.
ZenTest Reaches 1000 Downloads
drbrain |
Today ZenTest crossed one thousand downloads!
The latest release of ZenTest contains zentest, Test::Rails and the incredibly cool (and my personal favorites) unit_diff and autotest.
ZenTest is my secret weapon for speedy, bug free development. autotest keeps me focused by intelligently rerunning my tests as I modify my code while unit_diff lets me easily decipher my failures. Test::Rails provides me a rich assertion library that makes writing tests for my Rails applications a breeze.
I’m not the only person who loves ZenTest though, Sean Carley first fell in love with unit_diff then took the step of integrating ZenTest with emacs while working with Pat Eyler doing Ping-Pong Pairing on their secret project.
Don’t have ZenTest installed? Install the gem!
<kbd>$ sudo gem install ZenTest Successfully installed ZenTest-3.2.0 Installing ri documentation for ZenTest-3.2.0... Installing RDoc documentation for ZenTest-3.2.0...</kbd>
Speeding up Test Runs with fork
drbrain |
Loading Rails takes a significant portion of your test run time, especially when you want to run only one test file or one test method. On my Powerbook loading Rails takes between four and six seconds. If you're frequently running unit tests this constant overhead can quickly become annoying.
When using autotest I may have to wait as much as ten seconds (five seconds between scans for changes, four seconds to load rails, one second to run the test) before I know if my changes fixed a problem or not. Ten seconds is past the threshold where I can keep paying attention which makes my mind wander. (A wandering mind is no good for productive work.) Also, those extra four seconds of loading Rails per test start to add up. I may load rails hundreds of times in a day just to run a tiny test.
There's one already existing way to reduce or eliminate that constant overhead of loading Rails. In development mode Rails reloads files to keep things running without restarting Rails on every change. I prefer to have an environment that is guaranteed to be clean when the tests start and reloading files removes this option.
Since I want Rails loaded without any application code I chose to create a process that would load rails then open up a server socket and wait for connections. When a connection comes in the process will fork to make a copy of the environment that can then load the application and run the tests.
A regular test run for just one file runs like this:
$ time ruby test/controllers/route_controller_test.rb Loaded suite test/controllers/route_controller_test Started ...................................................... Finished in 13.192465 seconds. 54 tests, 268 assertions, 0 failures, 0 errors real 0m17.884s user 0m8.147s sys 0m1.424s
The difference between the real time and the Test::Unit run time accounts for Rails and app loading overhead, about five seconds.
I've tentatively named the parent process spawner 'ruby_fork' and the client 'ruby_fork_client', so you start up the parent process:
$ RAILS_ENV='test' ruby_fork -r rubygems -e 'require_gem "rails"' /Users/drbrain/Links/ZT/bin/ruby_fork Running as PID 3570 on 9084
ruby_fork understands -r, -I and -e just like regular ruby so I can just load Rails and none of the rest of my application.
Then I run ruby_fork_client which takes its arguments and passes them across to the child process and then reads from the socket and prints to STDOUT.
$ time ruby_fork_client -r test/controllers/route_controller_test.rb Loaded suite /Users/drbrain/Links/ZT/bin/ruby_fork Started ...................................................... Finished in 12.442556 seconds. 54 tests, 268 assertions, 0 failures, 0 errors real 0m13.947s user 0m0.077s sys 0m0.022s
Now that extra time spent loading Rails is gone and I'm left with application loading and Test::Unit overhead which is miniscule in comparison.
ruby_fork is not Rails specific. The server and client can do anything they like, so this has applications beyond testing Rails (for example, handling incoming mail) or even Rails itself.
I'd like to release ruby_fork and ruby_fork_client as part of ZenTest but I'll be holding it until 3.3.0. Currently ZenTest is almost ready for release and ruby_fork and ruby_fork_client needs to act more like a regular invocation of ruby.
Pat Eyler on ZenTest, ZenTest-3.1.0
drbrain |
Pat Eyler has a new article in Linux Journal that covers the new additions to ZenTest.
We’ve just fixed a couple of his issues with ZenTest. ZenTest now ignores Emacs-generated autosave files and it has better error messages when you have missing files.
Go get ZenTest-3.1.0:
$ sudo gem install ZenTestAutotest is Better than Ever
drbrain |
Last night at Seattle.rb’s weekly hacking night I polished off several bugs and features for autotest in preparation for a release of ZenTest this week, possibly even today!
The most-interesting new feature (inspired by comments from Pat Eyler) is a mini continuous-integration mode I added. By running autotest -vcs=cvs, autotest will perform a cvs up every 5 minutes in the course of running its tests. Autotest also understands how to update svn and p4 repositories as well.
There is one open report related to testrb not being found when using the One-Click Ruby Installer. Not having a win32 machine, I can’t confirm whether the bug is ours, in the one-click installer, or a user configuration error.
Rails Functional TestCase
drbrain |
Its a shame that Rails doesn’t define its own Test::Unit::TestCase subclasses. I’ve taken that into my own hands. This one puts Test on the front because that’s the Test::Unit way.
require 'test/unit'
def Object.path2class(klassname)
klassname.split('::').inject(Object) { |k,n| k.const_get n }
end
class FunctionalTestCase < Test::Unit::TestCase
def setup
self.class.name =~ /\ATest(.*)\Z/
return unless $1
controller_klass = Object.path2class $1
@controller = controller_klass.new
controller_klass.send(:define_method, :rescue_action) { |e| raise e }
@request = ActionController::TestRequest.new
@response = ActionController::TestResponse.new
@deliveries = []
ActionMailer::Base.deliveries = @deliveries
end
def test_stupid
end
end
