Controlling Rails Process Size

Eric Hodel | Wed, 13 Sep 2006 07:44:00 GMT

Now that Ruby 1.8.5 is out setting process limits is easier than ever before! We used to use a small shell script run from cron to kill processes that got too big for their britches. Unfortunately this was difficult to do both well and simply (we chose simply).

Now, in Ruby 1.8.5, we have Process::setrlimit:

$ ri Process::setrlimit
----------------------------------------------------- Process::setrlimit
     Process.setrlimit(resource, cur_limit, max_limit)        => nil
     Process.setrlimit(resource, cur_limit)                   => nil
------------------------------------------------------------------------
     Sets the resource limit of the process. cur_limit means current 
     (soft) limit and max_limit means maximum (hard) limit.

     If max_limit is not given, cur_limit is used.

     resource indicates the kind of resource to limit. The list of 
     resources are OS dependent. Ruby may support following resources.

Process::RLIMIT_COREcore size (bytes) (SUSv3)
[...]
Process::RLIMIT_RSSresident memory size (bytes) (4.2BSD, GNU/Linux)

There's a bunch more in there, but those were the two I was most interested in using. I really don't want cores, and I don't want my application processes to grow too big, but I want cron jobs to get as big as they need to get.

At the top of config/environment.rb above everything else I have:

Process.setrlimit Process::RLIMIT_RSS, 1024*1024*150, Process::RLIM_INFINITY

And to eliminate core dumps, in config/environments/production.rb I have:

Process.setrlimit Process::RLIMIT_CORE, 0, Process::RLIM_INFINITY

Keeping the hard limit at infinity allows me to increase the limit at a later date, for example when running cron jobs. To give cron jobs different limits I've added a config/environments/cron.rb that slurps the production values then overrides as necessary:

eval File.read("#{RAILS_ROOT}/config/environments/production.rb")

CachedModel.use_local_cache = false

# Cron jobs can use a much memory as they want.
Process.setrlimit Process::RLIMIT_RSS, Process::RLIM_INFINITY

(That eval is in there because of that's the same hack that Rails::Initializer uses to expose config in environment files, bleh.)

Posted in Rails, Ruby | 1 comment | no trackbacks

render_tree for Rails

Eric Hodel | Fri, 08 Sep 2006 18:02:00 GMT

Spelunking deep in unfamiliar Rails view code? Flushing out the cobwebs and updating your code? Don't know what gets rendered when?

I have a solution for you:

class ActionView::Base

  alias plain_render render

  RENDERS = [:partial, :template, :file, :action, :text, :inline, :nothing,
             :update]

  def render(*args)
    @level ||= 0

    print '  ' * @level

    case args.first
    when String then
      p args.first
    when Hash then
      hash = args.first
      found = hash.keys & RENDERS
      if found.length == 1 then
        puts "%p => %p" % [found.first, hash[found.first]]
      else
        raise "Dunno: %p" % [hash]
      end
    else
      raise "Dunno: %p" % [args]
    end

    @level += 1
    result = plain_render(*args)
    @level -= 1
    result
  end

end

Drop that in test/render_tree.rb and require it in your tests when you want to see a tree like this:

$ ruby test/views/things_view_test.rb -n test_view
Loaded suite test/views/things_view_test
Started
"things/things-header"
  "things/sidebar"
    "widgets/forms/goal_form"
    :partial => "widgets/sidenav_boxes/invite_and_edit"
      "widgets/sidenav_boxes/_invite_and_edit"
    :partial => "widgets/forms/edit_worth_doing_form"
      "widgets/forms/_edit_worth_doing_form"
    :partial => "widgets/sidenav_boxes/tags"
      "widgets/sidenav_boxes/_tags"
    :partial => "widgets/sidenav_boxes/popular_places"
      "widgets/sidenav_boxes/_popular_places"
    :partial => "widgets/sidenav_boxes/google_ads"
      "widgets/sidenav_boxes/_google_ads"
    :partial => "widgets/sidenav_boxes/find_help"
      "widgets/sidenav_boxes/_find_help"
    :partial => "widgets/sidenav_boxes/people_who_reached_this_goal"
      "widgets/sidenav_boxes/_people_who_reached_this_goal"
    :partial => "widgets/sidenav_boxes/quotation"
      "widgets/sidenav_boxes/_quotation"
    :partial => "widgets/sidenav_boxes/goal_created_by"
      "widgets/sidenav_boxes/_goal_created_by"
"widgets/general/post_add_messages"
"things/shared_body"
  :partial => "widgets/goals_gallery_teaser"
    "widgets/_goals_gallery_teaser"
  :partial => "entries_bucket"
    "things/_entries_bucket"
  "widgets/forms/related_goals"
.
Finished in 1.205494 seconds.

1 tests, 7 assertions, 0 failures, 0 errors

(I think I'll end up throwing this into ZenTest.)

Posted in Hacking, Rails, Toys | 4 comments

Memory leaks, cached_model and backend jobs

Eric Hodel | Thu, 24 Aug 2006 21:08:36 GMT

One of my long-running problems with Rails (and Ruby in general) is that it’s difficult to debug memory leaks. I’ve had a number of cases where I’ve stuck something into a long-lived array or hash and discovered much later that my Ruby process was eating over 100 MB of RAM. While ps makes it easy to see when Ruby’s using lots of RAM, actually figuring out where it went is a lot harder.

[...] I asked the Seattle Ruby Group for help, and Ryan Davis gave me a quick little memory leak spotter that he uses. I made a few additions to it, and it helped me discover that my Typo development tree was leaking 1-3 strings per hit

—Memory leak profiling with Rails via scottstuff: Memory leak profiling with Rails

I just used Scott and Ryan’s script to find a memory leak in our backend jobs related to cached_model. We have the local cache enabled for for our sites because we reset the cache every page request. In our backend jobs we don’t clear the local cache so we end up with a Hash that holding every ActiveRecord object we’ve ever retrieved from the database.

So if the warnings in the documentation weren’t enough, here it is again. If you’re using CachedModel in a backend job, be sure to disable the local cache or call cache_reset periodically to allow items to be garbage collected.

Posted in Rails, Robot Co-op | no comments

Ruby, Rails, Test::Rails Cheat Sheet

Eric Hodel | Thu, 24 Aug 2006 18:07:39 GMT

A cheat sheet that shows the assertions in Test::Unit, the ones added by Rails, and the ones further added by Test::Rails (part of the ZenTest gem).

Ruby, Rails, Test::Rails Cheat Sheet

—Ruby, Rails, Test::Rails Cheat Sheet via Nuby on Rails

Posted in Rails, Testing, ZenTest | no comments

Subclassing vs include

Eric Hodel | Wed, 23 Aug 2006 18:15:00 GMT

In memcached Basics for Rails Rob Sanheim asked:

[W]hy make the cached model a class to extend instead of a module? Whether or not a model is cached should be an implementation detail, and shouldn’t define the hierarchy for a class. I know I would rather not use the power of (single!) inheritance just to cache something, when a mix-in should be plenty powerful to do it.

The short answer is:

Using a class the correct way to overlay features on top of another class.

Here’s my long answer:

When you use a class you get super, and super is a beautiful thing. It automatically walks your class’ ancestors and calls the right method in the right order.

A module doesn’t have this property so you can’t use it to overlay features on top of a class. The class’ implementation will always be called before the module’s implementation.

Now you’re going to say something about using alias to shuffle methods around. You could do that, but you’ll have to do this for each method you want to overlay (five in CachedModel) which involves lots of extra typing “do_the_thingy_without_the_stuff” that you could have had for free (and more-prettily) with “super”.

You also get another problem that may cause subtle bugs. When you use alias to overlay features the order of execution is dependent upon the order the files are required in! Intentionally writing code where the behavior may change file load order gives me the heebie-jeebies.

Having to do all that work to get the benefits of a subclass tells me that a module isn’t powerful enough to do what a subclass can, so a module isn’t the right way to add a cache.

I avoid* using modules to overlay features of a class, but do use them to add orthogonal or complementary features. Typically when I write a module it ends up being used like Comparable, Enumerable or Singleton. When I need to do something invasive a subclass is better.

Finally, making the argument that adding caching to ActiveRecord::Base via a subclass shouldn’t define the inheritance argument is very subjective. I could justify using a subclass by saying that ActiveRecord is a data storage class, and CachedModel is just another data store. If you want caching, inherit from CachedModel. If you don’t want caching, inherit from ActiveRecord::Base.

Except void or where prohibited by law. Your milage may vary. Break glass in case of emergency.

Posted in Rails, Ruby, Software | 9 comments

ar_mailer 1.1

Eric Hodel | Sat, 19 Aug 2006 22:41:15 GMT

ar_mailer allows you to queue ActionMailer emails in the database that will be sent with a separate process. ar_mailer gives drastic speed-ups when you need to send many emails.

To install ar_mailer: sudo gem install ar_mailer. You can also download ar_mailer from Rubyforge.

For instructions on how to convert from the regular delivery methods to using ar_mailer see the ar_mailer documentation.

ar_mailer-1.1.0 follows quick on the heels of ar_mailer 1.0. I’ve got several important features and bug fixes:

Features

• Added—chdir to set rails directory
• Added—environment to set RAILS_ENV
• Exits cleanly on TERM or INT signals
• Added FreeBSD rc.d script
• Exceptions during SMTP sending are now logged
• No longer waits if sending email took too long

Bugs fixed

• Fixed last send attempt in—mailq
• Better SMTP error handling
  • Messages are removed from the queue on 5xx errors
  • Added Net::SMTP.reset to avoid needing to recreate the connection

Posted in Rails, Software | no comments

cached_model 1.2

Eric Hodel | Fri, 18 Aug 2006 07:38:00 GMT

Geoff writes about our cached_model and memcache-client libraries:

The Robot Co-Op has made a few libraries available. cached_model makes it easy to cache single row queries from ActiveRecord tables. memcache-client is a pure Ruby client and is included with the installation of cached_model.

—memcached Basics for Rails via Nuby on Rails

He even includes a useful how-to document to get you up and running!

(Yes, that’s right, a brand new version of cached_model that works with Rails 1.1! If you’ve already downloaded 1.2.0, please upgrade to 1.2.1. I forgot that find_by_sql will return [], thanks to James Cox for pointing out my bug.)

Posted in Rails, Software | no comments

ar_mailer

Eric Hodel | Tue, 15 Aug 2006 23:39:00 GMT

Rubyforge Project: http://rubyforge.org/projects/rctools

Documentation: http://dev.robotcoop.com/Tools/ar_mailer

About

Even deliviring email to the local machine may take too long when you have to send hundreds of messages. ar_mailer allows you to store messages into the database for later delivery by a separate process, ar_sendmail.

Installing ar_mailer

Just install the gem:

$ sudo gem install ar_mailer

Converting to ar_mailer

Go to your Rails project:

$ cd your_rails_project

Create a new migration:

$ ar_sendmail --create-migration

You'll need to redirect this into a file. If you want a different name provide the —table-name option. Create a new model:

$ ar_sendmail --create-model

You'll need to redirect this into a file. If you want a different name provide the —table-name option. Change your email classes to inherit from ActionMailer::ARMailer instead of ActionMailer::Base:

--- app/model/emailer.rb.orig   2006-08-10 13:16:33.000000000 -0700
+++ app/model/emailer.rb        2006-08-10 13:16:43.000000000 -0700
@@ -1,4 +1,4 @@
-class Emailer < ActionMailer::Base
+class Emailer < ActionMailer::ARMailer

def comment_notification(comment)
  from comment.author.email

Edit config/environments/production.rb and set the delivery agent:

$ grep delivery_method config/environments/production.rb
ActionMailer::Base.delivery_method = :activerecord

Run ar_sendmail:

$ ar_sendmail

You can also run ar_sendmail from cron with -o, or as a daemon with -d. See ar_sendmail -h for full details.

Posted in Rails, Software | 2 comments

Freezing Rails Versions

Eric Hodel | Wed, 09 Aug 2006 18:58:00 GMT

Ryan Davis has a nice Rake tasks that lets you freeze a Rails release. Its probably the easiest way to upgrade to Rails 1.1.5.

Posted in Rails | no comments

Timezones 1, Rails 0

Eric Hodel | Tue, 04 Jul 2006 00:17:00 GMT

For Trackmap I need to take the time a picture was taken and a user supplied time zone and convert that to a correctly offset time. Since Flickr won't give me a time zone (cameras don't record them) I ask the user. If they say Mountain time I need to create a time that is correct for the server's clock (in my case Pacific Time).

Rails TimeZone

When I originally wrote the code back in November and December of 2005 I noticed that Rails provides the handy TimeZone class which worked great! Rails gave me handy offsets that I could use to adjust times correctly.

Then April rolled around, my clocks switched to daylight savings time, and Rails' TimeZone class broke. TimeZone thought the offset for Pacific time was -8 hours instead of -7 hours giving inaccurate conversions. I filed a ticket, but it was closed WONTFIX (the breakage remains undocumented). I was, however, pointed at the TZInfo Timezone plugin. Since April I haven't had much time to work on Trackmap, so I left it broken.

TZInfo Timezone Plugin

On Friday I switched over to the plugin and found it provides incorrect UTC offsets as a misfeature:

>> tzt = TimeZone['Pacific Time (US & Canada)']
=> #
>> tzt.utc_offset
=> -28800
>> Time.now.gmtoff
=> -25200

While the TZinfo Timezone plugin will convert times between known time zones, it doesn't provide a way of interpreting a user-input time so you get a correctly offset time, making it only half of a fix for Rails' TimeZone.

Interpreting Times

To handle user-input times you must use TZInfo::Timezone#period_for_utc or TZInfo::Timezone#period_for_local to retrieve the correct UTC offset. You can retrieve the TZInfo::Timezone object from TzinfoTimezone#tzinfo (which is all the utility the plugin provides).

Correct conversions require some work. I chose to do it this way:

1. Create a Time using the user-supplied time
2. Get the local TZInfo::TimezonePeriod for the timestamp
   • If a TimezonePeriod doesn't exist for the given time, get the utc period
   • If the TimezonePeriod is ambiguous, choose standard time
3. Use TimezonePeriod#to_utc to adjust the Time to a UTC Time

A TimezonePeriod won't exist during the standard time to daylight savings time switch, the user probably chose the wrong time zone. The TimezonePeriod is ambiguous during the daylight savings time to standard time switch since the user's clock traverses the same time period twice. I chose to have strange behavior over failing. If the user notices and emails me I can say they probably fat-fingered something.

Rails Strikes Back

Having time interpretation finished I thought I was done. My database has timestamp with time zone columns so the database and Rails should just figure things out. Turns out Rails still can't handle timestamp with time zone despite having a patch in the bug tracking database for three months.

Instead of applying the patch (which I'd have to do twice, once on my laptop and once online, and for every Rails upgrade until it gets checked in), I adapted it into a 75 line extension to the PostgreSQL connection adapter. Maybe I'll make a plugin out of it while I'm waiting.

I'd like to think I was unlucky and the patch just got lost in the shuffle, but a look at the bug database makes me think that lost tickets happen frequently. One quarter of Rails' defect tickets are open (1098 out of 4034) so I don't see how anybody can figure out what tickets are invalid, valid, duplicate, fixed but not closed, or outright bogus. Furthermore, the amount of bugmail generated must be overwhelming.

The Rails core team should do what Mozilla did back in its early days and organize a bug day. Get a bunch of people on IRC to crawl through the open tickets to validate, cross-reference, prioritize and categorize bugs. (Mozilla even made it a weekly event with prizes!) Having a clean bug database will prevent useful patches from falling on the floor and rotting to the point that their authors must re-write them.

Posted in Hacking, Rails, Trackmap | 4 comments