Using Queues

Eric Hodel | Fri, 11 Nov 2005 22:34:00 GMT

Posted in Ruby

I have a multithreaded program that updates RSS feeds for 43 People and I want it to finish without leaving any feeds on the Queue or in-process.

Originally I had a bunch of worker threads in a ThreadGroup and when I finished enqueueing records from the database I would repeatedly grab a thread out of the group and join it until there were no more threads left. Unfortunately attempting to join a thread waiting for an item to appear on the Queue will cause the thread to wait forever so I could never exit.

To exit cleanly I went with this approach:

    t = Thread.start do
      loop do
        break if @done and @queue.empty?
        begin
          feed = @queue.pop true
          secs = time { feed.refresh REFRESH_TTL }
          log "Updated feed #{feed.id} in #{secs}s" 
        rescue ThreadError
          sleep 0.5
        end
      end
    end

By passing true to Queue#pop you end up with a ThreadError if there is nothing to pull from the queue. I added a short delay to keep the process from busy-looping in case the DB reader thread is delayed. When the DB reader is done, it sets @done to true so the updater thread knows to finish.

Now I don’t have to worry about the updater script exiting while there are jobs in-flight.

Another way is to add a Queue for receipts and compare feeds issued for update to update receipts, but that isn’t nearly as simple as a delay and attempting to pop again.