Segment7: autotest Sucks

autotest Sucks

drbrain@segment7.net (Eric Hodel) — Tue, 25 Apr 2006 16:51:58 -0700

You really might think this is a strange thing to say, despite how awesome some people say it is, it still sucks. It doesn’t suck because it doesn’t work well, it sucks because its insides still bear the scars of its birth.

My first version of autotest was written at OOPSLA 2005 after seeing Don Roberts and John Brandt show off a Smalltalk class browser that would automatically run tests whenever methods where changed.

At that time I was TDDing a personal Rails project and realized a tool to automatically run my tests as I made changes would probably speed up my development as much as switching to TDD from web browser reloading development did, so I wrote the first version of autotest.

That first version of autotest was probably about 100 lines of code. It was tied directly to my development process and got stuck occasionally, but it worked well for me.

Sometime in January I imported autotest into ZenTest and made it work for more generic ruby code. To do that I took my tiny ruby script and wrapped it up in a class. I pulled the Rails stuff out into a subclass and made a few other cleanups.

This didn’t do anything for the cleanliness of the code, but I did manage to add a bunch of tests that I hadn’t during my OOPSLA coding spree. I still had bugs though, and those took another four months to shake out with a few minor refactorings along the way.

Now autotest is nearly perfect functionally, but the implementation sucks. Some methods that do two or three things when they should be doing one. The test running algorithm is scattered across several methods.

autotest needs a major refactoring and I’m hoping to get to it in May. Refactoring will make the code cleaner, more straightforward and more maintainable. I may even do something to better support custom testing styles. I could add features to autotest, but that would only make the refactoring harder, so I won’t while adding those features will make the code more convoluted.

As autotest accumulated more suck it became less fun to work on. I don’t like the sound of software in pain so I’m going to listen and fix its suckage before adding any more fanciness.

"autotest Sucks" by Jonathan Brodsky

Wed, 26 Apr 2006 12:25:05 -0700

I just started doing TDD last night for the first time, and auto test was a really nice way to run things.

thanks for writing such a nice tool. I am looking forward to it getting faster.

"autotest Sucks" by Eric Hodel

Wed, 26 Apr 2006 12:13:47 -0700

In perforce I’ve dropped the timeout to 2 seconds as 1 second churns the CPU a bit much. The five second thing was annoying.

Before pulling in ruby_fork I’ll need to do a bunch of refactoring. I already tried once and it got far too messy. Maybe I’ll get a chance to work on it at Mindcamp this weekend.

"autotest Sucks" by topfunky

Wed, 26 Apr 2006 11:56:30 -0700

I agree than autotest works well, and I also admire your goal to refactor it before adding more features.

That being said, it would be nice if the refactoring could also separate the change notification so that more fine-grained OS-specific tools could be used (like Mac OS X’s kpoll file save notifier). For other OS’es, the normal “check every 5 seconds” system could be used.

Other than using stakeout, I’m not sure how this would be implemented…maybe as part of an Observer pattern where the notifier was configurable.

On the other hand, it might be more worthwhile to put your development resources toward the forking system you have mentioned since an OS-specific change notifier would save a maximum of 5 seconds, whereas keeping Rails in memory could save 10-15 seconds for each run (but this only helps if you are using autotest for Rails).