Memory Map

Eric Hodel | Wed, 30 Aug 2006 21:49:31 GMT

Here’s a picture of my Ruby process’ heap slots:

BLACK	unknown (char*, VALUE *, struct st_table, etc.)
GRAY	free memory
RED	nodes (code)
GREEN	Ruby objects
WHITE	Miscellaneous interpreter objects

If you look really closely in the original you can see a white pixel that is probably a struct SCOPE.

Posted in Hacking, Ruby | no comments

Memory Inspector

Eric Hodel | Wed, 30 Aug 2006 08:13:00 GMT

I've hacked into gc.c and added an alternative to ObjectSpace.each_object that gives you the contents of every heap slot in Ruby's memory space.

$ ./ruby -I ~/Work/p4/zss/src/RubyInline/dev/ ~/heapwalk.rb
[id, size, inspect]
0x001aa004 0x00000000                                                    :__free
0x001aa018 0x00000000                                                    :__free
0x001aa02c 0x00000000                                                    :__free
0x001aa040 0x00000000                                                    :__free
[the beginning is boring...]
0x001caa20 0x00000014                                           #<Class:Marshal>
0x001caa34 0x00000014                                                    :__node
0x001caa48 0x00000014                                                    :__node
0x001caa5c 0x0000000c                                                    Marshal
[but there's things like nodes, singleton classes, modules,]
0x001caae8 0x0000000c                #<NoMemoryError: failed to allocate memory>
0x001caafc 0x0000002d                                "failed to allocate memory"
0x001cab10 0x00000014                                                         []
[exceptions, strings, arrays and other objects,]
0x001ce5f8 0x00000014                                                  :__iclass
[included Modules,]
0x001dad30 0x00000014                                                   :__scope
[scopes, and some other stuff that's too hard to find]

It requires a hack to gc.c to gain access to the heap, but combined with Ryan's memory visualizer this might generate something usable.

Code will be available later I've audited my object size calculations (and have slept).

Posted in Hacking, Ruby | 3 comments

Finding primes with Rinda::TupleSpace

Eric Hodel | Sat, 19 Aug 2006 09:09:00 GMT

While at FOSCON 2006 I watched Lucas Carlson's presentation on Rinda and DRb, but he barely scratched the surface of Rinda's capabilities with his prime finding implementation.

Rinda::TupleSpace is more powerful than just a name service for DRb. Since a TupleSpace has synchronized access to all its items, you can use a TupleSpace to coordinate processes as well.

I started rewriting Lucas' DRb using prime finder at FOSCON, but wasn't able to finish it until now. This version coordinates all the activities of the prime finding via the TupleSpace. No locking is necessary in any of the code because the TupleSpace takes care of it for us. Read more...

Posted in DRb, Hacking, Ruby | no comments

Setting $stdout per-thread

Eric Hodel | Wed, 16 Aug 2006 18:58:00 GMT

cdfh on #ruby-lang asked how to redirect $stdout per-thread and I came up with this solution, redirect via a thread-local variable:

##
# Allows $stdout to be set via Thread.current[:stdout] per thread.

module ThreadOut

  ##
  # Writes to Thread.current[:stdout] instead of STDOUT if the thread local is
  # set.

  def self.write(stuff)
    if Thread.current[:stdout] then
      Thread.current[:stdout].write stuff 
    else
      STDOUT.write stuff
    end
  end
  
end

$stdout = ThreadOut

Simple test:

require 'stringio'
require 'threadout'

s = StringIO.new

Thread.start do 
  Thread.current[:stdout] = s
  puts 'redirected to StringIO'
end.join

Thread.start do
  puts 'no redirection'
end.join

puts s.string

Output:

no redirection
redirected to StringIO

Posted in Hacking, Ruby | no comments

Timezones 1, Rails 0

Eric Hodel | Tue, 04 Jul 2006 00:17:00 GMT

For Trackmap I need to take the time a picture was taken and a user supplied time zone and convert that to a correctly offset time. Since Flickr won't give me a time zone (cameras don't record them) I ask the user. If they say Mountain time I need to create a time that is correct for the server's clock (in my case Pacific Time).

Rails TimeZone

When I originally wrote the code back in November and December of 2005 I noticed that Rails provides the handy TimeZone class which worked great! Rails gave me handy offsets that I could use to adjust times correctly.

Then April rolled around, my clocks switched to daylight savings time, and Rails' TimeZone class broke. TimeZone thought the offset for Pacific time was -8 hours instead of -7 hours giving inaccurate conversions. I filed a ticket, but it was closed WONTFIX (the breakage remains undocumented). I was, however, pointed at the TZInfo Timezone plugin. Since April I haven't had much time to work on Trackmap, so I left it broken.

TZInfo Timezone Plugin

On Friday I switched over to the plugin and found it provides incorrect UTC offsets as a misfeature:

>> tzt = TimeZone['Pacific Time (US & Canada)']
=> #
>> tzt.utc_offset
=> -28800
>> Time.now.gmtoff
=> -25200

While the TZinfo Timezone plugin will convert times between known time zones, it doesn't provide a way of interpreting a user-input time so you get a correctly offset time, making it only half of a fix for Rails' TimeZone.

Interpreting Times

To handle user-input times you must use TZInfo::Timezone#period_for_utc or TZInfo::Timezone#period_for_local to retrieve the correct UTC offset. You can retrieve the TZInfo::Timezone object from TzinfoTimezone#tzinfo (which is all the utility the plugin provides).

Correct conversions require some work. I chose to do it this way:

1. Create a Time using the user-supplied time
2. Get the local TZInfo::TimezonePeriod for the timestamp
   • If a TimezonePeriod doesn't exist for the given time, get the utc period
   • If the TimezonePeriod is ambiguous, choose standard time
3. Use TimezonePeriod#to_utc to adjust the Time to a UTC Time

A TimezonePeriod won't exist during the standard time to daylight savings time switch, the user probably chose the wrong time zone. The TimezonePeriod is ambiguous during the daylight savings time to standard time switch since the user's clock traverses the same time period twice. I chose to have strange behavior over failing. If the user notices and emails me I can say they probably fat-fingered something.

Rails Strikes Back

Having time interpretation finished I thought I was done. My database has timestamp with time zone columns so the database and Rails should just figure things out. Turns out Rails still can't handle timestamp with time zone despite having a patch in the bug tracking database for three months.

Instead of applying the patch (which I'd have to do twice, once on my laptop and once online, and for every Rails upgrade until it gets checked in), I adapted it into a 75 line extension to the PostgreSQL connection adapter. Maybe I'll make a plugin out of it while I'm waiting.

I'd like to think I was unlucky and the patch just got lost in the shuffle, but a look at the bug database makes me think that lost tickets happen frequently. One quarter of Rails' defect tickets are open (1098 out of 4034) so I don't see how anybody can figure out what tickets are invalid, valid, duplicate, fixed but not closed, or outright bogus. Furthermore, the amount of bugmail generated must be overwhelming.

The Rails core team should do what Mozilla did back in its early days and organize a bug day. Get a bunch of people on IRC to crawl through the open tickets to validate, cross-reference, prioritize and categorize bugs. (Mozilla even made it a weekly event with prizes!) Having a clean bug database will prevent useful patches from falling on the floor and rotting to the point that their authors must re-write them.

Posted in Hacking, Rails, Trackmap | 4 comments

Keyword Assistant

Eric Hodel | Mon, 29 May 2006 05:22:18 GMT

I use Keyword Assistant with iPhoto to tag all my photos before uploading them to flickr. Unfortunately it has the unfortunate habit of running back to a compatibility mode when run on a version of iPhoto which it doesn't know about. This means Keyword Assistant won't make new keywords until the author releases a new version. There is a not so simple way around this restriction, I just opened up the KeywordAssistant binary which lives in /Applications /iPhoto.app /Contents /NetServices /Bundles /KeywordAssistant.NetService /Contents /MacOS/ and change the stored version to the current version of iPhoto. In this case I bumped it from 6.0.2 to 6.0.3. Naturally this kind of hackery might cause Keyword Assistant to do bad things like make iPhoto crash, so save a backup of that binary (and maybe even a backup of your photo library data files) before changing things around.

Posted in Hacking, Misc | no comments

IMAPCleanse 1.1.0

Eric Hodel | Fri, 05 May 2006 08:45:28 GMT

Now with imap_flag!

sudo gem install IMAPCleanse

I’ve even got fresh IMAPCleanse RDoc!

In short:

• imap_cleanse removes old, read, unflagged messages from my IMAP mailboxes.
• imap_flag flags messages I find interesting so I don’t have to!

They can even do this for you, too!

Next up is Part Three of my Plan for Total Email Domination wherein IMAPCleanse flags messages based on their interestingness.

Posted in Hacking, Misc | no comments

Marshal.load Speed

Eric Hodel | Wed, 03 May 2006 04:52:41 GMT

File.open 'dump' do |fp| Marshal.load fp end

Is much slower than:

File.open 'dump' do |fp| Marshal.load fp.read end

Because the former uses IO#getc and the latter operates directly on a String.

I learned this tonight while helping profile code at a Seattle.rb hacking night.

Posted in Hacking, Ruby | 2 comments

DRb via YAML

Eric Hodel | Sun, 30 Apr 2006 10:01:19 GMT

Much longer than it should be as the serialization of objects is buried inside a couple of huge methods. One tricky bit was having to mark DRbObject as not responding to #yaml_initialize, the other was marking a few undumpable classes as undumpable by YAML.

I’ll post it tomorrow when I have more brain.

Posted in Conference, DRb, Hacking | 1 comment

TupleSpace Replicator

Eric Hodel | Sun, 30 Apr 2006 02:32:00 GMT

At MindCamp 2.0 I was asked about how you would replicate a TupleSpace. I whipped up the following implementation:

class Rinda::TupleSpaceReplicator

  attr_accessor :debug

  attr_reader :local_tuplespace, :remote_tuplespace

  def initialize(remote_tuplespace, templates)
    @local_tuplespace = Rinda::TupleSpace.new
    @remote_tuplespace = remote_tuplespace
    @templates = templates

    @debug = false

    @replicators = ThreadGroup.new
  end

  def replicate
    @templates.each do |template|
      %w[write take].each do |type|
        make_replicator type, template
      end
    end
  end

  def make_replicator(event_type, template)
    thread = Thread.start do
      observer = @remote_tuplespace.notify event_type, template
      observer.each do |event, tuple|
        break if event == 'close'
        @local_tuplespace.send event, tuple
        $stderr.puts "master #{event}: #{tuple.inspect}" if @debug
      end
    end

    @replicators.add thread
  end

end

The core of this is the make_replicator method. It replicates tuples of the specified template from the remote tuplespace to the local tuplespace. Read more...

Posted in Conference, DRb, Hacking | no comments