Replace your test helpers with reusable API
drbrain |
test/test_helper.rb is a great idea Rails brought to the Ruby world as a place for functionality that helps you write better tests. There's now a standard place for you to implement common setup/teardown, shortcuts and custom assertions. However, a test helper is not the best place to store this functionality for a Ruby library.
One of the benefits you get out of writing tests is knowing where your API is clumsy and inadequate. If you have a test helper file full of methods to make your library easy to use in a test why is that not part of your library's API? Wouldn't your users also want a bunch of methods that make your library easier to use in their applications?
For example, in RubyGems we have a test helper that does this: gem_file = Gem::Builder.new(spec).build which is a little silly. Every time you create a Gem::Builder you want to build a gem. You don't create a Gem::Builder object for fun! To help out RubyGems users I added a new method: gem_file = Gem::Builder.build spec which immediately creates and builds them gem which is much nicer for everyone (but really, you should use Gem::PackageTask when building gems).
Whether you're writing a library or a Rails app, this kind of functionality belongs in your library (or application) code, not in the test helper where only your tests can benefit from it.
Even after you improve your API by moving helpful functionality back into your library there's still going to be some things that only make sense for tests. For example, you probably don't want to type t = Some::Deeply::Namespaced::Thing.new 1, 2, 3 many, many times in your tests, so you write a short wrapper method you can call like this: t = thing 1, 2, 3. Your tests may need setup and teardown to maintain a clean environment between tests, custom assertions for readability or you may want to include a pre-built stub or mock.
While this having this functionality in a test helper is fine for a Rails app, it shouldn't go in a library's test helper. When you keep testing functionality hidden in the test directory a user who wants to write a third-party extension for your gem can't access them. Why force a happy user to re-implement (possibly poorly or incorrectly) the work you've done to have nice, clean tests that are easy to read and write?
Instead of having a private test helper I have a public test case like MyGem::TestCase that lives in lib/my_gem/test_case.rb. This gives anyone who wants to extend my libraries a documented, ready-to-go API for writing tests for their extension.
My gem-specific test case typically contains all the requires needed to load the library (ideally require 'my_gem'), proper setup and teardown to sandbox the tests, any utility methods that don't belong in the library itself and possibly some custom assertions. This makes a brand new test easy to start:
require 'my_gem/test_case'
class TestMyGemSomeClass < MyGem::TestCase
def setup
super
# …
end
def test_something
# …
end
endThere is the minor downside that an extension writer must use minitest (my preferred testing library) to test their extension. Perhaps this inconvenience could be solved by a module providing setup, teardown and shortcuts that is included in the proper place for the extension writer's favorite testing library.
PS: Actually, Gem::Builder.build is Gem::Package.build since Gem::Format, Gem::Builder and Gem::Package are getting merged into one convenient class that deals with reading and writing gem files for RubyGems 2.0. This means there will only be one place to look for the API of messing with packages and it reduces the implementation of Gem::Installer a bit.
How to Sleep in Tests
drbrain |
And Use Other Kernel Methods
Ruby provides many methods for you in Kernel including such favorites as #sleep, #fork, #open, #system and #` that can be hard to test reliably and quickly.
When you write code that uses these methods you want your tests to be reliable and fast but not need to set up too much state, like having files to open, before running (most importantly because that's more typing, but also because it's more stuff that may break). You also don't want the tests to be slow so actually sleeping or running a command are undesirable.
Let's start with some sample code. Imagine you've got a severe case of NIH and you're implementing cron (poorly):
def run command, every
loop do
status = system command
raise "#{command} failed" unless status
sleep every
end
endAnd here's a test:
def test_run
e = assert_raises RuntimeError do
@cron.run 'false', 0
end
assert_equal 'false failed', e.message
endHow do we test the sleep behavior though? We'd need a command that failed on the second invocation. We'd also want to check the sleep duration somehow which would involve waiting.
There's another way though, since both #system and #sleep are in Kernel we can avoid calling the real implementations through the power of inheritance! If we have our tests inject an implementation of #sleep and #system into the implementation so the real methods never get called:
def test_run
def @cron.sleep time
@sleep = time
end
def @cron.system command
@commands ||= []
@commands << command
@commands.length <= 1
end
e = assert_raises RuntimeError do
@cron.run 'any old command', 2**30
end
assert_equal 'any old command failed', e.message
assert_equal 2**30, @cron.instance_variable_get(:@sleep)
assert_equal ['any old command', 'any old command'],
@cron.instance_variable_get(:@commands)
endThis test never calls Kernel#sleep nor Kernel#system, problem solved! As an additional benefit, this test will work on windows where the first will not because false may not exist.
Of course, you can always use a mocking or stubbing framework, but ruby is powerful enough on its own that you don't really need one.
Testing Effectively with Rails
drbrain |
I wrote a blog post for AT&T Interactive's Engineering Blog on how to test effectively with rails.
I know you all know how to write tests, so this blog post is not about that. I want to move your testing to the next level. Your current testing practices are OK, but some changes in how you approach testing will allow you to refactor and improve your code even faster and more easily than you currently do.
Check it out!
How I Use Autotest
drbrain |
During the Q&A portion of my talk, I was asked a very important question about autotest, something like “How often do you save when using autotest?” I save all the time when using autotest. There was a followup question, something like “Don’t you get syntax errors?” and I don’t.
Before I wrote autotest I was making fine-grained saves that were syntactically correct. I wrote autotest to automate the running of tests so I wouldn’t have to choose which tests to run. My changes were so small that I spent an annoying fraction of my time editing my command line.
When I’m using autotest, rather than writing all of a method’s tests, I start by performing the setup (which should cause an error), add a flunk, and save, which is usually four lines:
def test_blah
result = @thingy.blah
flunk
endWhen the tests run they’ll see the method blah doesn’t exist and fail, so I define the method, (the def blah and end lines), and save. Then I write an assertion and save the test, which will fail, so I implement what I need to make the assertion pass, and save. Now I repeatedly perform the minimum changes to go from failure to flunking until my test is complete, remove the flunk, and start over with the next method.
Each save I do is a handful of lines, so it’s easy to keep them syntactically correct. I know I’m going from working state to working state as I work towards my end goal so I can easily roll back my changes with undo.
Firebrigade Home Page Fixed
drbrain |
I denormalized a bit and beat the tests back into shape and the Firebrigade home page is back to life! Next I’m going to sand down a few rough edges on RubyGems to get a beta shipped suitable for feedback.
Firebrigade is now fully vladified using perforce.
Here’s the two setup steps you need for perforce:
- Require ‘vlad/perforce’ at the top of config/deploy.rb.
- Your .p4config goes in the scm/ directory of the server’s checkout.
- Run p4 client in the scm/ directory and set your View to //path/to/project/… //clientname/…
Hopefully we can fully automate this so the vlad:setup_app task can handle this automatically.
I also had to do some custom setup for firebrigade because it uses RubyInline and needs the INLINEDIR set. I just added extra stuff to the setup_app task:
namespace :vlad do
remote_task :setup_app do
cmds = [
"mkdir #{inline_dir}",
"sudo chown www:www #{inline_dir}",
]
end
Update: Automatic p4 setup is done!
Notes on Heckle
drbrain |
When I started heckle I was hoping for somehing better than rcov that would tell you that your code is poorly tested… But code so poorly engineered that changing anything as bound to make everything fall apart looks like well tested (read: heckle-proof) code in a sense. We both think that running just the unit tests that directly test the MUT (method under test) would help, but really it wouldn’t help enough. […]
Suggestions? How would you differentiate between “well tested” and “tightly coupled crap”?
—Notes on Heckle via Polishing Ruby
I’m puzzled by this too. How can we detect well-tested vs tightly-coupled (and poorly tested) code automatically?
tinderbox version 1.0.0 has been released!
drbrain |
tinderbox version 1.0.0 has been released!
http://seattlerb.rubyforge.org/tinderbox
Description
Tinderbox tests projects and tries to make them break by running them on as
many different platforms as possible.
Features & Problems
- Tests gems in a sandbox
- Submits gem test results to http://firebrigade.seattlerb.org
- Understands test/unit and RSpec
Changes:
1.0.0 / 2007-01-30
- Tests gems in a sandbox
- Submits results to Firebrigade
- Birthday!
firebrigade_api version 1.0.0 has been released!
drbrain |
http://seattlerb.rubyforge.org/firebrigade_api
firebrigade_api is an API wrapper for http://firebrigade.seattlerb.org
Changes
- Birthday!
- Ruby interface to http://firebrigade.seattlerb.org
- User-friendly cache for implementing Tinderbox
Test Profiling by Lines Logged
drbrain |
At work I've been cleaning up the tests and trying to make them run faster. One way of doing this is profiling the tests and fixing the slow spots. That only works so well, especially if there's lots of duplication or extra work in the tests. With Rails you can go about this a different way, since you have a second source of information on your tests' operation, the log file.
I added the following code to test/test_helper.rb:
raise 'require\'d test/test_helper twice!, you broke it!' if
ENV['RAILS_ENV'] == 'test'At the very top, even above ENV["RALIS_ENV"] = "test" to make sure that the hack to Test::Unit::TestCase below happens only once, then the magic to figure out which test outputs which log lines:
class Test::Unit::TestCase
alias unlogged_run run
def run(result, &block)
RAILS_DEFAULT_LOGGER.debug "RUNNING #{self.class} #{@method_name}"
unlogged_run result, &block
end
end if ENV['PROFILE_LOG']Each test run will be prefixed with text like RUNNING SomeTest test_blah which I can then run this script on:
#!/usr/local/bin/ruby -w
test = nil
tests = Hash.new 0
File.open 'log/test.log' do |fp|
fp.each_line do |line|
if line.strip =~ /^RUNNING (.*)/ then
test = $1
else
tests[test] += 1
end
end
end
tests.sort_by { |test, count| -count }.each do |test, count|
puts "%5d %s" % [count, test]
end
To get the profile information, I run:
rake log:clear; PROFILE_LOG=y rake && script/count_logs
When run on Firebrigade, I get the following output (top 10 only):
130 ProjectViewTest test_show 74 DummyControllerTest test_error_500 59 OwnerTest test_class_owner_count 42 ProjectControllerTest test_index 39 HomeControllerTest test_index_no_builds 38 RestControllerTest test_add_build 35 ProjectControllerTest test_show_no_versions 35 ProjectControllerTest test_search_many_matches 34 RestControllerTest test_add_project 33 RestControllerTest test_add_version
Now I can quickly discover good candidates for refactoring. #test_error_500 above dumps an email into the logs, so it is a bogus result. That leaves ProjectViewTest#test_show as a candidate for simplification or refactoring.
Ruby, Rails, Test::Rails Cheat Sheet
drbrain |
A cheat sheet that shows the assertions in Test::Unit, the ones added by Rails, and the ones further added by Test::Rails (part of the ZenTest gem).
Ruby, Rails, Test::Rails Cheat Sheet
