#sort_by and #sort_obj

Eric Hodel | Sun, 07 Oct 2007 08:42:32 GMT

I was speeding up RubyGems and found a bunch of places that still used #sort instead of #sort_by. #sort_by is faster than #sort because it performs fewer comparisons resulting in fewer method calls.

All these places were external to the thing I was sorting, so they’d need to know how to perform the sorting, which is just plain wrong. I did a bit of thinking, and created a #sort_obj method to return an object that can be used for sorting instead.

So Gem::Specification#<=> went from:

def <=>(other)
  platform_num = platform == Gem::Platform::RUBY ? -1 : 1
  other_platform_num = other.platform == Gem::Platform::RUBY ? -1 : 1

  [@name, @version, platform_num] <=>
    [other.name, other.version, other_platform_num]
end

To:

def sort_obj
  [@name, @version.to_ints, @platform == Gem::Platform::RUBY ? -1 : 1]
end

def <=>(other)
  sort_obj <=> other.sort_obj
end

So now instead of:

specs.sort_by do |spec|
  [@name, @version.to_ints, @platform == Gem::Platform::RUBY ? -1 : 1]
end

I can write:

specs.sort_by { |spec| spec.sort_obj }

Posted in ,  | no comments

Two Weeks of Vacation, Vlad, RubyGems

Eric Hodel | Fri, 17 Aug 2007 21:04:05 GMT

I’m not working this month so I can do whatever I want. So far that’s featured:

Building Vlad was a lot of fun. Ryan and I flew up Wilson Bilkovich, watched Bourne Ultimatum, then hacked for four days straight to bring Vlad into the world.

InfoQ published an interview with us about Vlad:

Capistrano, a popular deployment tool for Rails, is challenged by Vlad the Deployer, a tool which offers similar functionality with a much simpler implementation. We talked to the Ruby Hit Squad group that released version 1.0 of Vlad.

—Capistrano gets competition: Vlad the Deployer via InfoQ

The RubyGems hacking has been mostly bug fixes and refactoring so far. I’m working towards teaching RubyGems your platform so it can automatically install the correct version.

Here’s a sample of what’s I’ve done to RubyGems so far:

  • —sources is no longer remembered forever, use `gem sources` to manage the permanent list
  • The sources gem is gone, RubyGems uses a built-in list now (but can be upgraded in the future)
  • `gem list` respects its default of just gem names now
  • Only exact gem names are matched on install, “foo_bar 2.0” won’t install instead of “foo 1.0” if you run `gem install foo`
  • RubyGems requires only what it needs when you require 'rubygems'
  • Fewer bulk updates when updating the gem index
  • `gem dep -r` lists dependencies for remote gems
  • `gem info -r` shows information for remote gems
  • `gem -v` turns on “really verbose” mode (verbose mode is the default)
  • `gem_mirror`, `gem_server`, `gemlock`, `gemri`, `gemwhich`, `index_gem_repository.rb` have been merged into `gem`

Posted in , , ,  | 2 comments

SuperCaller version 1.0.0 has been released!

Eric Hodel | Sun, 05 Aug 2007 04:40:00 GMT

SuperCaller adds a beefed-up version of Kernel#caller and a beefed up version of Exception#backtrace. 
require 'super_caller'

def something() super_caller end
stack = something
p stack.first.file # => "-"
p stack.first.line # => 4
p stack.first.method_name # => nil
p stack.first.self # => main
p stack.first.sexp # => [:vcall, :super_caller]
p stack.first.source # => "def something\n  super_caller\nend"
Changes in 1.0.0 / 2007-06-30:
  • 1 major enhancement
    • Birthday!
http://seattlerb.org/SuperCaller

Posted in ,  | no comments

Inliners are Hard to Debug

Eric Hodel | Sun, 01 Jul 2007 08:40:00 GMT

So I wrote SuperCaller

require 'rubygems'
require 'super_caller'
require 'super_caller/exception'

class X
  def y; z; end
  def z; raise; end
end

begin
  X.new.y
rescue => e
  e.backtrace .each do |frame|
    puts frame
    puts frame.source || frame.sexp.inspect
    puts "---"
  end
end

Gives:

/usr/local/lib/ruby/gems/1.8/gems/SuperCaller-1.0.0/lib/super_caller/exception.rb:12:in `initialize'
def initialize(message)
  old_initialize(message)
  @backtrace = super_caller
end
---
test.rb:7:in `new'
[[:vcall, :raise]]
---
test.rb:7:in `z'
[[:vcall, :raise]]
---
test.rb:6:in `y'
def y
  z
end
---
test.rb:11
[[:call, [:call, [:const, :X], :new], :y]]
---

Coming Soon

Posted in ,  | no comments

I wrote an inliner

Eric Hodel | Wed, 27 Jun 2007 07:35:00 GMT 

$ ruby test.rb 
caller result: 12
caller:
def caller
  v1 = (2 + 3)
  x = callee(v1)
  (x + 2)
end
callee:
def callee(v)
  (v + 5)
end
 
inline callee into caller
caller result: 12
caller:
def caller
  v1 = (2 + 3)
  x = (inline_callee_v = v1
  (inline_callee_v + 5))
  (x + 2)
end
$ ruby -Ilib bm.rb 
Rehearsal -------------------------------------------
empty     0.090000   0.000000   0.090000 (  0.083842)
plain     1.030000   0.010000   1.040000 (  1.037302)
inlined   0.810000   0.000000   0.810000 (  0.821394)
---------------------------------- total: 1.940000sec
 
              user     system      total        real
empty     0.080000   0.000000   0.080000 (  0.084179)
plain     1.100000   0.000000   1.100000 (  1.105742)
inlined   0.900000   0.000000   0.900000 (  0.900799)

Posted in , ,  | no comments

Process Growth with Railsbench's GC patch

Eric Hodel | Tue, 17 Apr 2007 19:57:00 GMT

My friend Kevin Watt who runs Allpoetry had me help him diagnose problems with memory consumption on his Rails application. He was seeing processes suddenly jump in size by 46MB, when the processes were already over 100MB to begin with. After instrumenting to locate the growth in one innocuous method we went through his patches to Rails.

One of the patches Kevin was using was the Railsbench GC patch. It turns out that this patch has some very useful instrumenting of its own, so I had Kevin call GC.dump in that method, and he got the following values: 

HEAP[ 0]: size= 650000
HEAP[ 1]: size=1170001

This means there are slots for 1820001 objects in his process. If you end up creating that many objects Ruby will create a new chunk of object slots by multiplying the previous allocation by 1.8 and adding 1.

The next set of slots would contain slots for 2106002 objects. Each slot takes up 20 bytes of memory, so roughly 40MB of memory would be used. Kevin ended up pulling the patch and now has his processes holding on to around 70MB RSS each, rather than 100 or more.

The Railsbench page contains this note about memory consumption alongside the GC patch:

In addition, a patch for the ruby garbage collector is provided, which can be used to reduce the amount of time spent doing garbage collection, trading memory for speed, as usual (see file GCPATCH for details).

The initial size of the slots can be tuned, but we neglected to explore that option. In large-memory situations the patch's behavior is too negative. Be sure to follow the instructions to the GCPATCH file to get proper operation. Also, monitor your application's memory usage as the patch may cause your machine to use swap when too many objects are being used or if you've tuned improperly.

Posted in , ,  | 2 comments

render_tree for Rails

Eric Hodel | Fri, 08 Sep 2006 18:02:00 GMT

Spelunking deep in unfamiliar Rails view code? Flushing out the cobwebs and updating your code? Don't know what gets rendered when?

I have a solution for you: 

class ActionView::Base

  alias plain_render render

  RENDERS = [:partial, :template, :file, :action, :text, :inline, :nothing,
             :update]

  def render(*args)
    @level ||= 0

    print '  ' * @level

    case args.first
    when String then
      p args.first
    when Hash then
      hash = args.first
      found = hash.keys & RENDERS
      if found.length == 1 then
        puts "%p => %p" % [found.first, hash[found.first]]
      else
        raise "Dunno: %p" % [hash]
      end
    else
      raise "Dunno: %p" % [args]
    end

    @level += 1
    result = plain_render(*args)
    @level -= 1
    result
  end

end

Drop that in test/render_tree.rb and require it in your tests when you want to see a tree like this: 

$ ruby test/views/things_view_test.rb -n test_view
Loaded suite test/views/things_view_test
Started
"things/things-header"
  "things/sidebar"
    "widgets/forms/goal_form"
    :partial => "widgets/sidenav_boxes/invite_and_edit"
      "widgets/sidenav_boxes/_invite_and_edit"
    :partial => "widgets/forms/edit_worth_doing_form"
      "widgets/forms/_edit_worth_doing_form"
    :partial => "widgets/sidenav_boxes/tags"
      "widgets/sidenav_boxes/_tags"
    :partial => "widgets/sidenav_boxes/popular_places"
      "widgets/sidenav_boxes/_popular_places"
    :partial => "widgets/sidenav_boxes/google_ads"
      "widgets/sidenav_boxes/_google_ads"
    :partial => "widgets/sidenav_boxes/find_help"
      "widgets/sidenav_boxes/_find_help"
    :partial => "widgets/sidenav_boxes/people_who_reached_this_goal"
      "widgets/sidenav_boxes/_people_who_reached_this_goal"
    :partial => "widgets/sidenav_boxes/quotation"
      "widgets/sidenav_boxes/_quotation"
    :partial => "widgets/sidenav_boxes/goal_created_by"
      "widgets/sidenav_boxes/_goal_created_by"
"widgets/general/post_add_messages"
"things/shared_body"
  :partial => "widgets/goals_gallery_teaser"
    "widgets/_goals_gallery_teaser"
  :partial => "entries_bucket"
    "things/_entries_bucket"
  "widgets/forms/related_goals"
.
Finished in 1.205494 seconds.

1 tests, 7 assertions, 0 failures, 0 errors

(I think I'll end up throwing this into ZenTest.)

Posted in , ,  | 4 comments

Using RubyInline for Optimization

Eric Hodel | Sat, 02 Sep 2006 03:24:11 GMT

I wrote an article on using RubyInline for optimization where I take png.rb, sprinkle in a little profiling and a little C and make it go over 100 times faster.

Posted in ,  | 2 comments

mem_inspect and png

Eric Hodel | Thu, 31 Aug 2006 22:52:00 GMT

I'm pleased to announce to new libraries written by members of the Seattle Ruby Brigade, mem_inspect and png!

mem_inspect

mem_inspect is ObjectSpace.each_object on crack. mem_inspect gives you the contents of each slot in Ruby's heap. mem_inspect also includes viewers that let you visualize the contents of Ruby's heap.

To install: 

sudo gem install mem_inspect

Then you'll need to build a patched ruby: 

ruby_mem_inspect_build

You'll then have a ruby capable of running mem_inspect in mem_inspect_ruby_1_8.

You can make an image with: 

mem_inspect_ruby_1_8/ruby_mem_inspect -S ruby_mem_dump

Which will give you a PNG in your current directory named: mem_inspect.PID.TIMESTAMP.png

You'll get an image that looks something like this:

http://flickr.com/photos/drbrain/229482312/

Bigger:

http://flickr.com/photo_zoom.gne?id=229482312&size=o

To dump a PDF any time you want: 

require 'meminspect/png_viewer'

MemInspect::PNGViewer.new(1024, 768).draw

You can also dump to an AquaTerm plot window if you have RubyCocoa and AquaTerm installed. 

require 'meminspect/aquaterm_viewer'

MemInspect::AquatermViewer.new(1024, 768).draw

png

png is a pure-ruby PNG writing library written by Ryan Davis.

To install: 

sudo gem install png

To use: 

require 'png'

canvas = PNG::Canvas.new 200, 200

# Set a point to a color
canvas[100, 100] = PNG::Color::Black

# draw an anti-aliased line
canvas.line 50, 50, 100, 50, PNG::Color::Blue

png = PNG.new canvas
png.save 'blah.png'

Posted in , ,  | no comments

Reducing $SAFE

Eric Hodel | Thu, 31 Aug 2006 00:30:00 GMT

Ya you are correct, it won't let you change the safe level. I wonder how hard it would be to bypass it though using something like rubyinline?

—Re: $SAFE =4 safe enough? via snacktime 

require 'rubygems'
require 'inline'

class DeSafe
  inline do |builder|
    builder.prefix "RUBY_EXTERN int ruby_safe_level;"

    builder.c <<-EOC
      static void
      reduce() {
        ruby_safe_level = 0;
      }
    EOC
  end
end


$SAFE = ARGV.shift.to_i rescue 0

p $SAFE

DeSafe.new.reduce

p $SAFE
$ rm -fr ~/.ruby_inline/; ruby desafe.rb 4
desafe.rb:20:in `write': Insecure operation `write' at level 4 (SecurityError)
        from desafe.rb:20:in `p'
        from desafe.rb:20
$ rm -fr ~/.ruby_inline/; ruby desafe.rb 3
3
0

Posted in ,  | 4 comments

Older posts: 1 2 3 4