Mail Hosting

Eric Hodel | Sat, 09 May 2009 04:28:43 GMT

I host my own mail using the following software:

OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.

Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).

Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)

In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:

DELIVER = /usr/local/libexec/dovecot/deliver

# ...

:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems

# ...

# last rule, delivers to INBOX
:0 w
| ${DELIVER}

My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).

I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.

To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.

My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).

Posted in ,  | no comments

Binding#remove_local_variable

Eric Hodel | Fri, 24 Apr 2009 00:29:07 GMT 

require 'rubygems'
require 'inline'

class Binding
  inline do |builder|
    builder.include '"node.h"'
    builder.include '"env.h"'

    ##
    # struct BLOCK isn't in any header, so include it here.

    builder.prefix <<-C
struct BLOCK {
  NODE *var;
  NODE *body;
  VALUE self;
  struct FRAME frame;
  struct SCOPE *scope;
  VALUE klass;
  NODE *cref;
  int iter;
  int vmode;
  int flags;
  int uniq;
  struct RVarmap *dyna_vars;
  VALUE orig_thread;
  VALUE wrapper;
  VALUE block_obj;
  struct BLOCK *outer;
  struct BLOCK *prev;
};
    C

    ##
    # :method: remove_local_variable
    #
    # Removes the local variable +name+ and replaces it with nil.
    #
    # Ordinarily if a local variable didn't exist it would raise
    # NoMethodError, but currently after #remove_local_variable it doesn't, it
    # just returns nil.
    #
    # In order to make this behave correctly, block->body would need to be
    # walked, duping nodes (so as not to affect future invocations of this
    # method) and replacing the LVAR nodes with VCALL nodes.
    #
    # That's just too much work for 17:15, though.  Maybe tomorrow.

    builder.c <<-C
VALUE remove_local_variable(VALUE name) {
  struct BLOCK *block;
  struct SCOPE *scope;
  ID name_id;
  ID *local_table;
  int i, n;
  VALUE entry;

  name_id = SYM2ID(name);

  Data_Get_Struct(self, struct BLOCK, block);

  scope = block->scope;
  local_table = scope->local_tbl;

  if (local_table) {
    n = *local_table++;
    for (i = 2; i < n; i++) { /* skip $_ and $~ */
      if (!rb_is_local_id(local_table[i])) continue; /* skip flip states */
      if (local_table[i] == name_id) {
        entry = scope->local_vars[i];
        local_table[i] = (ID)NULL;
        scope->local_vars[i] = Qnil;
        return entry;
      }
    }
  }

  return Qnil;
}
    C

  end
end

a = :my_value

p :lvar_a => a

b = binding
p :remove_local_variable_says => b.remove_local_variable(:a)

p :lvar_a => a # TODO raise exception

Posted in ,  | 2 comments

RubyGems code_swarm

Eric Hodel | Wed, 28 Jan 2009 00:59:32 GMT

I built a code_swarm visualization of the RubyGems repository history:


RubyGems code_swarm (HD) from Eric Hodel on Vimeo.

Coincidentally, you can find a code_swarm visualization for Rails and other projects by Ilya Grigorik. He points to Peter Burns’ fork on GitHub which has a much improved toolset for generating these types of visualizations.

I used the original repository which involves a lot more work. For RubyGems I did roughly the following:

  1. Get a log: svn log -v > rubygems.log
  2. Convert the log: python convert_logs.py -s rubygems.log -o rubygems.xml
  3. Munge the XML to remove the CVS conversion commits and hide the renames from trunk/rubygems/ to trunk/
  4. Run code_swarm: code_swarm rubygems.config
  5. Encode PNGs to an MP4: ffmpeg -f image2 -b 1500 -r 24 -i frames/rubygems-%05d.png -sameq rubygems.mp4
  6. Upload to Vimeo

I played around with the code_swarm configuration a bit and ended up with this config for code_swarm. My changes give a larger video with more lingering of files as they float around and bumps up font sizes for readability. For more active projects you may need to decrease FileLife and PersonLife. (Really, I’m not sure how they affect things.)

Posted in ,  | 3 comments

Friendly Ruby Objects

Eric Hodel | Thu, 18 Dec 2008 01:28:00 GMT

This post is intended to supplement the Ruby Quickref with the various ways you can make your objects play nicely with each other.

Most of the examples below are taken from RubyGems, some examples won't work until the next release of RubyGems.

Enumerable

The Enumerable module is based on the #each method and contains well known methods like #map and #each_with_index. If the enumerated objects implement the #<=> method you get a useful #sort, #min and #max.

Gem::SourceIndex has a Hash internally that it exposes via #each: 

class Gem::SourceIndex
  include Enumerable

  # ...

  def each(&block)
    @gems.each(&block)
  end
end

Which allows handy things like: 

dep = Gem::Dependency.new ARGV.shift, Gem::Requirement.default

found = Gem.source_index.any? do |name, spec|
  dep =~ spec
end

puts "found gem for #{dep.name}!" if found

Comparable

The Comparable module is based on the #<=> method and gives all the comparison methods.

Gem::Specification objects are sorted via name, version and platform: 

class Gem::Specification
  include Comparable

  def <=>(other)
    my_platform = Gem::Platform::RUBY == @platform ? -1 : 1
    other_platform = Gem::Platform::RUBY == other.platform ? -1 : 1

    [@name, @version, platform] <=>
      [other.name, other.version, other.platform]
  end
end

This is used in RubyGems to sort objects both for display on the screen like in gem list and internally when installing gems.

For a way to reduce the repetition in the above code and some other sorting speed-ups, see my post on #sort_by and #sort_obj.

#to_s and #inspect

Overriding #to_s and #inspect prevent your from puking all over the screen when somebody wants to look at your object. To a certain extent limiting the amount of information shown can aid debugging.

Gem::Specification's #to_s that gives only the two most-important attributes: 

class Gem::Specification
  def to_s
    "#<Gem::Specification name=#{@name} version=#{@version}>"
  end
end

Gem::Platform's #to_s gives a friendly string: 

class Gem::Platform
  def to_a
    [@cpu, @os, @version]
  end

  def to_s
    to_a.compact.join '-'
  end
end

Gem::Version's inspect ignores internal instance variables and only exposes @version: 

class Gem::Version
  def inspect # :nodoc:
    "#<#{self.class} #{@version.inspect}>"
  end
end

Case equality with #===, Matching with #=~

While #=== is more commonly overridden, it can also be useful to implement #=~ to allow your objects to be used in a very readable manner.

Gem::Platform overrides #===: 

class Gem::Platform
  def ===(other)
    return nil unless Gem::Platform === other

    # cpu
    (@cpu == 'universal' or other.cpu == 'universal' or @cpu == other.cpu) and

    # os
    @os == other.os and

    # version
    (@version.nil? or other.version.nil? or @version == other.version)
  end
end

In short, one platform matches another if they have the same cpu (architecture) or either is universal, they have the same os and their versions match if they have versions.

You can use this to group gems: 

platform_count = Hash.new 0

Gem.source_index.each do |name, spec|
  case spec.platform
  when Gem::Platform.new('linux') then
    platform_count['linux'] += 1
  # ...
  else
    platform_count['other'] += 1
  end
end

p platform_count

Gem::Dependency overrides #=~. It is a little strange because it converts the right-hand side to a Gem::Dependency object: 

class Gem::Dependency
  def =~(other)
    other = case other
            when self.class then
              other
            else
              return false unless other.respond_to? :name and
                                  other.respond_to? :version

              Gem::Dependency.new other.name, other.version
            end

    pattern = @name
    pattern = /\A#{Regexp.escape @name}\Z/ unless Regexp === pattern

    return false unless pattern =~ other.name

    reqs = other.version_requirements.requirements

    return false unless reqs.length == 1
    return false unless reqs.first.first == '='

    version = reqs.first.last

    version_requirements.satisfied_by? version
  end
end

You can use this as a filter: 

dep = Gem::Dependency.new(/ruby/, Gem::Requirement.default)

ruby_named = Gem.source_index.select do |name, spec|
  dep =~ spec
end

p ruby_named.map { |name, spec| name }

As a Hash key

Ruby uses #hash and #eql? to determine if two different objects really mean the same hash key.

Gem::Version is usable as a Hash key based on the internal version string: 

class Gem::Version
  def hash
    @version.hash
  end

  def eql?(other)
    self.class === other and @version == other.version
  end
end

In Gem::Version, the internal version string looks like "1.3" or "1.3.0". In this implementation the two versions would belong to different hash keys.

Using #eql? instead of #== to determine if two keys are the same is a nice distinction since it allows you to have interesting behaviors (but I'm not sure they are useful). For Gem::Version, a version of "1.3" is equal to "1.3.0", but the occupy different slots in a Hash.

Marshal

Ruby will marshal most objects automatically, but sometimes you want a custom format to ignore cached data that can be reconstructed or to reduce the size of the data you're saving out. There are two ways to do this, the older way is #_dump/::_load and the newer way is #marshal_dump/#marshal_load which takes priority. If you want to upgrade to the newer way you can leave ::_load to restore older marshaled objects.

Using the older way #_dump returns a String representation of the object (usually another Marshal string) and ::_load receives that String representation. Note that it's a class method, so ::_load is responsible for creating the object, which may be important in some instances.

Using the newer way, #marshal_dump returns an Object and #marshal_load receives that Object. The object is already allocated, but #initialize won't be called. The newer way can result in a smaller marshal dump size since it uses the existing symbol and object reference tables.

Gem::Specification uses #_dump/::load and is fairly complicated because I designed it to be backward and forward-compatible. This is a slightly-stripped-down version: 

class Gem::Specification

  CURRENT_SPECIFICATION_VERSION = 2

  # number of fields per version
  MARSHAL_FIELDS = { -1 => 16, 1 => 16, 2 => 16 }

  def self._load(str)
    array = Marshal.load str

    spec = Gem::Specification.new
    spec.instance_variable_set :@specification_version, array[1]

	# validate object
    current_version = CURRENT_SPECIFICATION_VERSION

    field_count = if spec.specification_version > current_version then
                    spec.instance_variable_set :@specification_version,
                                               current_version
                    MARSHAL_FIELDS[current_version]
                  else
                    MARSHAL_FIELDS[spec.specification_version]
                  end

    if array.size < field_count then
      raise TypeError, "invalid Gem::Specification format #{array.inspect}"
    end

	# restore object
    spec.instance_variable_set :@rubygems_version,          array[0]
    # ...
    spec.instance_variable_set :@platform,                  array[16].to_s
    spec.instance_variable_set :@loaded,                    false

    spec
  end

  def _dump(limit)
    Marshal.dump [
      @rubygems_version,
      @specification_version,
      # ...
      @new_platform,
    ]
  end
end

Gem::Version uses #marshal_dump/#marshal_load and ignores the internal instance variables, only dumping @version: 

class Gem::Version
  def marshal_dump
    [@version]
  end

  def marshal_load(array)
    self.version = array[0]
  end
end

Pretty-print with PP

With a little work PP can give you easily readable output for your objects, even output that you can copy and paste back into a script. Primarily you'll use the PrettyPrint#group, PrettyPrint#text, PrettyPrint#breakable and PP#pp methods inside a #pretty_print method on your object. You can find documentation for these methods using ri.

Here's #pretty_print from Gem::Dependency and Gem::Requirement: 

class Gem::Dependency
  def pretty_print(q)
    q.group 1, 'Gem::Dependency.new(', ')' do
      q.pp @name
      q.text ','
      q.breakable

      q.pp @version_requirements

      q.text ','
      q.breakable

      q.pp @type
    end
  end
end

class Gem::Requirement
  def pretty_print(q)
    q.group 1, 'Gem::Requirement.new(', ')' do
      q.pp as_list
    end
  end

  def as_list
    normalize
    @requirements.map do |op, version| "#{op} #{version}" end
  end
end

Together these make pretty, copy-pastable output: 

require 'pp'

gem 'ParseTree'

pp Gem.loaded_specs["ParseTree"].dependencies
[Gem::Dependency.new("RubyInline",
  Gem::Requirement.new([">= 3.7.0"]),
  :runtime),
 Gem::Dependency.new("sexp_processor",
  Gem::Requirement.new([">= 3.0.0"]),
  :runtime),
 Gem::Dependency.new("hoe", Gem::Requirement.new([">= 1.8.0"]), :development)]

Posted in ,  | 6 comments

Working Style

Eric Hodel | Fri, 22 Aug 2008 00:42:25 GMT

In Rubinius we’ve been slowly working towards using a VM written in C++ instead of the current C VM we’ve got (called “shotgun”). Most of the mechanics of the VM are already complete, so over the past couple weeks we’ve been wiring up the primitives necessary for our ruby kernel (equivalent to Ruby’s core library) to run.

This primitive work is rather tedious, and until Friday I found it highly unenjoyable. I couldn’t perceive how much closer I was to our goal of actually running ruby code on the new VM.

On Friday I recalled that we can run a compiled ruby file against the VM, which gave me a way to shoot towards my goal. Over the past couple days during our Rubinius Team Meeting we’ve gotten up to loading all of our kernel bootstrap and platform, and now we’re just starting to do work in the core libraries.

Switching my working style drastically improved my enjoyment of a process that I found quite annoying. I wish I’d figured this out a couple weeks ago.

Posted in ,  | no comments

PlayStation 3

Eric Hodel | Wed, 16 Apr 2008 09:11:50 GMT

I like my PlayStation 3. I bought it in November when my PS2 was dying of a hacking cough because it couldn’t read disks. I thought the PS3 was alright then, it didn’t make horrible noises and it let me check the Blu-Ray box on Netflix.

I’m semi-disappointed that the PS3 DVD player lost the L3/R3 buttons that subtitles and audio settings that the PS2 had. Now I have to bring up the onscreen menu to turn on and off the subtitle tracks in case I missed or misheard some dialogue.

Since I bought my PS3, the software updates have improved it into a decent console. The PS2 Hitman games now all work on the 80GB (which I bought) and the store was fixed so it isn’t totally stupid for the majority of its users.

The pricing on the store’s downloadable content is reasonable. I’ve bought PixelJunk Monsters, a tower-defense game, flOw, a game where you eat stuff by using the motion-sensors in the controller, Lemmings and PAIN, a physics game that involves slingshotting a character into stuff and watching them scream. Lemmings is slightly handicapped by the PS3 controller, but I really like PixelJunk Monsters and flOw, I’ve certainly gotten my $10 of enjoyment out of them. PAIN is too hard to unlock and doesn’t have enough single-player content to keep my interest.

The most recent PS3 update replaced the old store with a brand new one that’s faster to navigate, unless you want to look at something. It downloads item images as you navigate and doesn’t cache them between runs which is annoying. The new store now knows which demos you’ve downloaded which is a really nice feature, though.

The DualShock 3 controller is much nicer than the SIXAXIS, by both being slightly heavier and having slightly stiffer buttons. Compared to the PS2 controller, the vibration doesn’t seem as powerful and the motors seem noisier. (I’m not sure, as my PS2 is in pieces and I doubt I can keep it alive long enough to start a game.)

The one extra thing I bought for the PS3 is the NYKO Blu-Wave remote so I could control the PS3 with my universal remote (which doesn’t speak Bluetooth) through the Blu-Wave’s USB IR dongle. The remote can be found on Amazon for under $20.

Posted in  | no comments

Graphing Spam

Eric Hodel | Mon, 24 Dec 2007 11:49:00 GMT

For fun I decided to chart various statistics from my mail server’s logs. I ended up with these charts using John Barnette’s GChart, some regular expressions, and cron.

The details of parsing out the data from the log files and setting up a crontab is boring, so I’ll spare you that.

Using GChart is really cool and easy. Here’s what I used to generate the Amavis Statistics chart, the others are all nearly the same, just different labels and titles.

def graph_amavis(data)
  # These are the labels and colors I'm using 
  labels = %w[Banned Spam   Spammy Bad\ Header Clean]
  colors = %w[000000 ff0000 ff7f00 ffff00      00ff00]

  max = data.map { |vals| vals.max }.max

  # Since axis labels aren't yet supported by the API in 0.2.0, I use :extras
  extras = { 'chxt' => 'r', 'chxl' => "0:|#{axis_labels max}" }

  chart = GChart.line :title => 'Amavis Statistics', :data => data,
                      :labels => labels, :colors => colors, :extras => extras

  chart_path = File.join File.dirname(@file), 'amavis_statistics.png'

  chart.size = '750x400'# can't be > 300,000 pixels
  chart.write chart_path
end

That’s it! One minor gotcha I had was that I needed to transpose the data set after reading it in because it wasn’t in the right order for GChart to consume it, but with Array#transpose, it’s just an extra method call before graphing.

Also, I wrote this simple utility function to calculate some good-enough axis labels:

def axis_labels(max)
  axis_labels = []
  0.upto 10 do |i| axis_labels << (max * 0.1 * i).to_i end
  axis_labels.join '|'
end

The values aren’t prettily chosen, but they work well enough.

Posted in  | no comments

WTF: The Mythical Business Layer

Eric Hodel | Fri, 28 Sep 2007 07:11:43 GMT

Worse Than Failure made a post a few days ago about, among other things, built-in complexity, especially where its unnecessary and unsuited for getting things done.

Just look at the dreadful specs we’re given to work with:

When a Sale is Cleared, only Managers with Void Approval and Executives may issue a Cancellation Request. If the Propagation Status for the Transferable Receivable is not Pending and the Expense Allocation Type is Reversible, the Cancellation Request is issued for Processing; otherwise, it is issued for Approval.

I’m sure those of you who managed to make it through that spec did not have visions of IF-ELSE code blocks swirling through your head. I’ll bet some of you, without even seeing the rest of specs, excitedly envisioned a CancelationWorkflowProvider that inherited from the abstract RequestWorkflowProvider and implemented the IPermissionRequired, IPropogationStatusRequired, and IExpenseAllocationTypeRequired interfaces, and was powered by the all-encompassing WorkflowManager. Why? Because that’s so much more challenging than writing a simple IF-ELSE code block.

—The Mythical Business Layer via Worse Than Failure

While the post focuses on the “business layer” of an application, it is applicable to any development. There’s no need to write that extra library! Start with the core of you want to do, and grow, then refactor, then grow again. Reuse what already exists unless it can’t be molded to your will. If your application code gets too big pull a library out. Don’t write the library up-front, you don’t need it and you won’t need it.

Posted in  | 2 comments

Finding Random Reading

Eric Hodel | Mon, 20 Aug 2007 03:13:43 GMT

I’m really missing what reddit used to give me, which was things I liked to read that I didn’t know I wanted to read on the front page. Now reddit is full of dups and political stuff I don’t care about. It also has a recomendation feature never worked for me, I couldn’t tell the difference between it and the home page.

Google News solves the dup problem but has too much stuff I don’t care about. Sometimes it makes me laugh, but it still doesn’t tell me what to read, or even what I probably will like.

The recommendation service I love is Netflix’s, I’ve rated over 350 movies now and it is spookily good at picking movies I like. For example 11:14 has a silly-sounding plot summary:

Five seemingly random story lines intersect at precisely 11:14 p.m. in this innovative drama-thriller written and directed by newbie filmmaker Greg Marcks. Even though they’re strangers, Buzzy, Mark, Cheri, Jac and Eddie will become a part of one another’s lives—even if it kills them.

I forgot why I added it to my queue. When it arrived I thought it would be silly, but I really enjoyed it, and that wasn’t the first movie I’ve experienced this with. Also, it tells me to watch things like Afro Samurai and Tinker, Tailor, Soldier, Spy that I would never hear about or know about otherwise.

What I really want is Netflix for for my random web reading. I don’t care about what’s popular, I care about what is well-written and interesting. Does this kind of thing exist yet?

Until then, I think I’m going to switch to clicking wikipedia’s Random article button when I get bored.

Posted in  | 5 comments

Vacation!

Eric Hodel | Wed, 01 Aug 2007 19:18:10 GMT

Yesterday was my last day working for Lime Spot, so now I’m on vacation for the month of August.

Most of my time is going to be spent working on various bits of software, like adding the automatic platform selection to RubyGems and getting it ready for inclusion in ruby 1.9, cleaning up some RDoc and ruby documentation tickets and going through the rest of my open tracker items on RubyForge.

But today, its time to watch the Simpsons movie, then Taxi Driver!

Posted in  | 3 comments

Older posts: 1 2