Example domains and IPs

Eric Hodel | Fri, 02 Oct 2009 21:58:28 GMT

I'm always forgetting this information. Here's a handy reference for example domain names, example IPv4 addresses and example IPv6 addresses.

Example Domain names

From RFC 2606:

.test

For testing DNS related code

example

example.com

example.net

example.org

For documentation or as examples

IPv4 Addresses

From RFC 3330:

192.0.2.0/24 is used for documentation and example code.

198.18.0.0/15 is used for benchmark tests of network interconnect devices. See RFC 2544

IPv6 Addresses

From RFC 3849:

2001:DB8::/32 is used for documentation and examples.

Posted in Misc | no comments

Service Discovery Protocols

Eric Hodel | Wed, 19 Aug 2009 21:42:03 GMT

I’ve worked on two different service discovery libraries for ruby, one for SSPD and one for mDNS, and I’d like to share the differences between them with you.

Both SSDP and mDNS are designed to be used with self-configuring networks and to automatically discover and connect to other devices. Both use multicast to advertise and search.

SSPD

The Simple Service Discovery Protocol is used primarily by UPnP for network devices to discover each other. UPnP is designed for consumer electronics and appliances that need to communicate with each other automatically.

The description of SSPD fits on a twelve of pages when written in (relatively) plain English (section 1 in the UPNP Device Architecture). The protocol itself looks much like HTTP and can be implemented largely using simple text processing. (Strangely, SOAP is the higher-level communication layer for UPnP.)

Unfortunately there’s no way for multiple devices to share the same SSPD socket without being designed to work with each other. Since UPnP is focused on simple devices, this fits well with their design goals. Most UPnP devices are routers, media servers, game consoles, televisions, etc. If you built your own router and media server on the same host it will probably be impossible for both devices to show up on the network, nor will you be able to run a media server and media viewer on the same host without the two being designed to work with each other.

mDNS

Multicast DNS is used primarily by Apple and (basically) has a DNS server on each host. Hosts speak the DNS protocol to each other over multicast and have a central daemon that handles communication on the network much like the libc DNS resolver.

Even though there’s plenty of DNS clients around for code reuse, implementing a DNS client form scratch is a large undertaking (my SSDP implementation is around 750 lines with documentation, ruby’s Resolv DNS resolver is around 2200 lines and doesn’t include a server or multicast code). The entirety of the DNS system is spread across several RFCs, not including the documentation for mDNS itself.

Fortunately mDNS libraries provide a daemon for handling network communication and a client library you can use to register and browse for devices (the dnssd gem uses Apple’s DNS Service Discovery API). mDNS is focused on computers rather than devices, so multiple services can all coexist on the same host.

Unfortunately, due to the complexity of mDNS, there’s no single API for applications to use. Apple’s DNS-SD works across multiple platforms including windows. For unix-like operating systems there is also avahi which has a limited dnssd compatibility shim.

Posted in Misc | no comments

vi bindings for irb on OS X

Eric Hodel | Tue, 11 Aug 2009 20:56:22 GMT

OS X uses editline(3) instead of readline(3) so ~/.inputrc doesn’t do anything for irb or other tools using readline via the editline wrapper.

Instead, use ~/.editrc:

bind -v
bind \\t rl_complete

Which gives you vi bindings in irb.

Update

Now with tab completion thanks to Curt Sampson from an ancient netbsd-users email!

Posted in Misc, Ruby | no comments

Mail Hosting

Eric Hodel | Sat, 09 May 2009 04:28:43 GMT

I host my own mail using the following software:

• FreeBSD for my OS
• Postfix for SMTP
• OpenBSD’s spamd for greylisting
• amavisd for spam control (which uses SpamAssassin) and DKIM signing
• dovecot for IMAP and SASL
• procmail for mail filtering

OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.

Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).

Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)

In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:

DELIVER = /usr/local/libexec/dovecot/deliver

# ...

:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems

# ...

# last rule, delivers to INBOX
:0 w
| ${DELIVER}

My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).

I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.

To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.

My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).

Posted in FreeBSD, Misc | no comments

Binding#remove_local_variable

Eric Hodel | Fri, 24 Apr 2009 00:29:07 GMT

require 'rubygems'
require 'inline'

class Binding
  inline do |builder|
    builder.include '"node.h"'
    builder.include '"env.h"'

    ##
    # struct BLOCK isn't in any header, so include it here.

    builder.prefix <<-C
struct BLOCK {
  NODE *var;
  NODE *body;
  VALUE self;
  struct FRAME frame;
  struct SCOPE *scope;
  VALUE klass;
  NODE *cref;
  int iter;
  int vmode;
  int flags;
  int uniq;
  struct RVarmap *dyna_vars;
  VALUE orig_thread;
  VALUE wrapper;
  VALUE block_obj;
  struct BLOCK *outer;
  struct BLOCK *prev;
};
    C

    ##
    # :method: remove_local_variable
    #
    # Removes the local variable +name+ and replaces it with nil.
    #
    # Ordinarily if a local variable didn't exist it would raise
    # NoMethodError, but currently after #remove_local_variable it doesn't, it
    # just returns nil.
    #
    # In order to make this behave correctly, block->body would need to be
    # walked, duping nodes (so as not to affect future invocations of this
    # method) and replacing the LVAR nodes with VCALL nodes.
    #
    # That's just too much work for 17:15, though.  Maybe tomorrow.

    builder.c <<-C
VALUE remove_local_variable(VALUE name) {
  struct BLOCK *block;
  struct SCOPE *scope;
  ID name_id;
  ID *local_table;
  int i, n;
  VALUE entry;

  name_id = SYM2ID(name);

  Data_Get_Struct(self, struct BLOCK, block);

  scope = block->scope;
  local_table = scope->local_tbl;

  if (local_table) {
    n = *local_table++;
    for (i = 2; i < n; i++) { /* skip $_ and $~ */
      if (!rb_is_local_id(local_table[i])) continue; /* skip flip states */
      if (local_table[i] == name_id) {
        entry = scope->local_vars[i];
        local_table[i] = (ID)NULL;
        scope->local_vars[i] = Qnil;
        return entry;
      }
    }
  }

  return Qnil;
}
    C

  end
end

a = :my_value

p :lvar_a => a

b = binding
p :remove_local_variable_says => b.remove_local_variable(:a)

p :lvar_a => a # TODO raise exception

Posted in Misc, Ruby | 2 comments

RubyGems code_swarm

Eric Hodel | Wed, 28 Jan 2009 00:59:32 GMT

I built a code_swarm visualization of the RubyGems repository history:

RubyGems code_swarm (HD) from Eric Hodel on Vimeo.

Coincidentally, you can find a code_swarm visualization for Rails and other projects by Ilya Grigorik. He points to Peter Burns’ fork on GitHub which has a much improved toolset for generating these types of visualizations.

I used the original repository which involves a lot more work. For RubyGems I did roughly the following:

1. Get a log: svn log -v > rubygems.log
2. Convert the log: python convert_logs.py -s rubygems.log -o rubygems.xml
3. Munge the XML to remove the CVS conversion commits and hide the renames from trunk/rubygems/ to trunk/
4. Run code_swarm: code_swarm rubygems.config
5. Encode PNGs to an MP4: ffmpeg -f image2 -b 1500 -r 24 -i frames/rubygems-%05d.png -sameq rubygems.mp4
6. Upload to Vimeo

I played around with the code_swarm configuration a bit and ended up with this config for code_swarm. My changes give a larger video with more lingering of files as they float around and bumps up font sizes for readability. For more active projects you may need to decrease FileLife and PersonLife. (Really, I’m not sure how they affect things.)

Posted in Misc, Rubygems | 3 comments

Friendly Ruby Objects

Eric Hodel | Thu, 18 Dec 2008 01:28:00 GMT

This post is intended to supplement the Ruby Quickref with the various ways you can make your objects play nicely with each other.

Most of the examples below are taken from RubyGems, some examples won't work until the next release of RubyGems.

Enumerable

The Enumerable module is based on the #each method and contains well known methods like #map and #each_with_index. If the enumerated objects implement the #<=> method you get a useful #sort, #min and #max.

Gem::SourceIndex has a Hash internally that it exposes via #each:

class Gem::SourceIndex
  include Enumerable

  # ...

  def each(&block)
    @gems.each(&block)
  end
end

Which allows handy things like:

dep = Gem::Dependency.new ARGV.shift, Gem::Requirement.default

found = Gem.source_index.any? do |name, spec|
  dep =~ spec
end

puts "found gem for #{dep.name}!" if found

Comparable

The Comparable module is based on the #<=> method and gives all the comparison methods.

Gem::Specification objects are sorted via name, version and platform:

class Gem::Specification
  include Comparable

  def <=>(other)
    my_platform = Gem::Platform::RUBY == @platform ? -1 : 1
    other_platform = Gem::Platform::RUBY == other.platform ? -1 : 1

    [@name, @version, platform] <=>
      [other.name, other.version, other.platform]
  end
end

This is used in RubyGems to sort objects both for display on the screen like in gem list and internally when installing gems.

For a way to reduce the repetition in the above code and some other sorting speed-ups, see my post on #sort_by and #sort_obj.

#to_s and #inspect

Overriding #to_s and #inspect prevent you from puking all over the screen when somebody wants to look at your object. To a certain extent limiting the amount of information shown can aid debugging.

Gem::Specification's #to_s that gives only the two most-important attributes:

class Gem::Specification
  def to_s
    "#<Gem::Specification name=#{@name} version=#{@version}>"
  end
end

Gem::Platform's #to_s gives a friendly string:

class Gem::Platform
  def to_a
    [@cpu, @os, @version]
  end

  def to_s
    to_a.compact.join '-'
  end
end

Gem::Version's inspect ignores internal instance variables and only exposes @version:

class Gem::Version
  def inspect # :nodoc:
    "#<#{self.class} #{@version.inspect}>"
  end
end

Case equality with #===, Matching with #=~

While #=== is more commonly overridden, it can also be useful to implement #=~ to allow your objects to be used in a very readable manner.

Gem::Platform overrides #===:

class Gem::Platform
  def ===(other)
    return nil unless Gem::Platform === other

    # cpu
    (@cpu == 'universal' or other.cpu == 'universal' or @cpu == other.cpu) and

    # os
    @os == other.os and

    # version
    (@version.nil? or other.version.nil? or @version == other.version)
  end
end

In short, one platform matches another if they have the same cpu (architecture) or either is universal, they have the same os and their versions match if they have versions.

You can use this to group gems:

platform_count = Hash.new 0

Gem.source_index.each do |name, spec|
  case spec.platform
  when Gem::Platform.new('linux') then
    platform_count['linux'] += 1
  # ...
  else
    platform_count['other'] += 1
  end
end

p platform_count

Gem::Dependency overrides #=~. It is a little strange because it converts the right-hand side to a Gem::Dependency object:

class Gem::Dependency
  def =~(other)
    other = case other
            when self.class then
              other
            else
              return false unless other.respond_to? :name and
                                  other.respond_to? :version

              Gem::Dependency.new other.name, other.version
            end

    pattern = @name
    pattern = /\A#{Regexp.escape @name}\Z/ unless Regexp === pattern

    return false unless pattern =~ other.name

    reqs = other.version_requirements.requirements

    return false unless reqs.length == 1
    return false unless reqs.first.first == '='

    version = reqs.first.last

    version_requirements.satisfied_by? version
  end
end

You can use this as a filter:

dep = Gem::Dependency.new(/ruby/, Gem::Requirement.default)

ruby_named = Gem.source_index.select do |name, spec|
  dep =~ spec
end

p ruby_named.map { |name, spec| name }

As a Hash key

Ruby uses #hash and #eql? to determine if two different objects really mean the same hash key.

Gem::Version is usable as a Hash key based on the internal version string:

class Gem::Version
  def hash
    @version.hash
  end

  def eql?(other)
    self.class === other and @version == other.version
  end
end

In Gem::Version, the internal version string looks like "1.3" or "1.3.0". In this implementation the two versions would belong to different hash keys.

Using #eql? instead of #== to determine if two keys are the same is a nice distinction since it allows you to have interesting behaviors (but I'm not sure they are useful). For Gem::Version, a version of "1.3" is equal to "1.3.0", but the occupy different slots in a Hash.

#intialize_copy

#initialize_copy is called during #dup and #clone to copy object-specific state beyond instance variables. The object being cloned from is passed to the new instance. #initialize_copy could be used for cleaning out a per-object cache:

def initialize_copy(other)
  @cache = []
end

#exception

#exception is called on objects given to #raise to cast them to Exception objects. It must return a subclass of Exception. You can use this to turn arbitrary objects into exceptions, centralizing all your exception raising code:

class Result
  class Error < RuntimeError; end

  def initialize(json)
    @result = JSON.parse json
  end

  def exception(message = nil)
    Error.new "#{message} (#{@result['error']})"
  end

  def [](key)
    @result[key]
  end
end

r = Result.new open('http://example.com/api/blah').read

raise r if r['error']

Marshal

Ruby will marshal most objects automatically, but sometimes you want a custom format to ignore cached data that can be reconstructed or to reduce the size of the data you're saving out. There are two ways to do this, the older way is #_dump/::_load and the newer way is #marshal_dump/#marshal_load which takes priority. If you want to upgrade to the newer way you can leave ::_load to restore older marshaled objects.

Using the older way #_dump returns a String representation of the object (usually another Marshal string) and ::_load receives that String representation. Note that it's a class method, so ::_load is responsible for creating the object, which may be important in some instances.

Using the newer way, #marshal_dump returns an Object and #marshal_load receives that Object. The object is already allocated, but #initialize won't be called. The newer way can result in a smaller marshal dump size since it uses the existing symbol and object reference tables.

Gem::Specification uses #_dump/::load and is fairly complicated because I designed it to be backward and forward-compatible. This is a slightly-stripped-down version:

class Gem::Specification

  CURRENT_SPECIFICATION_VERSION = 2

  # number of fields per version
  MARSHAL_FIELDS = { -1 => 16, 1 => 16, 2 => 16 }

  def self._load(str)
    array = Marshal.load str

    spec = Gem::Specification.new
    spec.instance_variable_set :@specification_version, array[1]

	# validate object
    current_version = CURRENT_SPECIFICATION_VERSION

    field_count = if spec.specification_version > current_version then
                    spec.instance_variable_set :@specification_version,
                                               current_version
                    MARSHAL_FIELDS[current_version]
                  else
                    MARSHAL_FIELDS[spec.specification_version]
                  end

    if array.size < field_count then
      raise TypeError, "invalid Gem::Specification format #{array.inspect}"
    end

	# restore object
    spec.instance_variable_set :@rubygems_version,          array[0]
    # ...
    spec.instance_variable_set :@platform,                  array[16].to_s
    spec.instance_variable_set :@loaded,                    false

    spec
  end

  def _dump(limit)
    Marshal.dump [
      @rubygems_version,
      @specification_version,
      # ...
      @new_platform,
    ]
  end
end

Gem::Version uses #marshal_dump/#marshal_load and ignores the internal instance variables, only dumping @version:

class Gem::Version
  def marshal_dump
    [@version]
  end

  def marshal_load(array)
    self.version = array[0]
  end
end

Pretty-print with PP

With a little work PP can give you easily readable output for your objects, even output that you can copy and paste back into a script. Primarily you'll use the PrettyPrint#group, PrettyPrint#text, PrettyPrint#breakable and PP#pp methods inside a #pretty_print method on your object. You can find documentation for these methods using ri.

Here's #pretty_print from Gem::Dependency and Gem::Requirement:

class Gem::Dependency
  def pretty_print(q)
    q.group 1, 'Gem::Dependency.new(', ')' do
      q.pp @name
      q.text ','
      q.breakable

      q.pp @version_requirements

      q.text ','
      q.breakable

      q.pp @type
    end
  end
end

class Gem::Requirement
  def pretty_print(q)
    q.group 1, 'Gem::Requirement.new(', ')' do
      q.pp as_list
    end
  end

  def as_list
    normalize
    @requirements.map do |op, version| "#{op} #{version}" end
  end
end

Together these make pretty, copy-pastable output:

require 'pp'

gem 'ParseTree'

pp Gem.loaded_specs["ParseTree"].dependencies

[Gem::Dependency.new("RubyInline",
  Gem::Requirement.new([">= 3.7.0"]),
  :runtime),
 Gem::Dependency.new("sexp_processor",
  Gem::Requirement.new([">= 3.0.0"]),
  :runtime),
 Gem::Dependency.new("hoe", Gem::Requirement.new([">= 1.8.0"]), :development)]

Posted in Misc, Ruby | 6 comments

Working Style

Eric Hodel | Fri, 22 Aug 2008 00:42:25 GMT

In Rubinius we’ve been slowly working towards using a VM written in C++ instead of the current C VM we’ve got (called “shotgun”). Most of the mechanics of the VM are already complete, so over the past couple weeks we’ve been wiring up the primitives necessary for our ruby kernel (equivalent to Ruby’s core library) to run.

This primitive work is rather tedious, and until Friday I found it highly unenjoyable. I couldn’t perceive how much closer I was to our goal of actually running ruby code on the new VM.

On Friday I recalled that we can run a compiled ruby file against the VM, which gave me a way to shoot towards my goal. Over the past couple days during our Rubinius Team Meeting we’ve gotten up to loading all of our kernel bootstrap and platform, and now we’re just starting to do work in the core libraries.

Switching my working style drastically improved my enjoyment of a process that I found quite annoying. I wish I’d figured this out a couple weeks ago.

Posted in Misc, Rubinius | no comments

PlayStation 3

Eric Hodel | Wed, 16 Apr 2008 09:11:50 GMT

I like my PlayStation 3. I bought it in November when my PS2 was dying of a hacking cough because it couldn’t read disks. I thought the PS3 was alright then, it didn’t make horrible noises and it let me check the Blu-Ray box on Netflix.

I’m semi-disappointed that the PS3 DVD player lost the L3/R3 buttons that subtitles and audio settings that the PS2 had. Now I have to bring up the onscreen menu to turn on and off the subtitle tracks in case I missed or misheard some dialogue.

Since I bought my PS3, the software updates have improved it into a decent console. The PS2 Hitman games now all work on the 80GB (which I bought) and the store was fixed so it isn’t totally stupid for the majority of its users.

The pricing on the store’s downloadable content is reasonable. I’ve bought PixelJunk Monsters, a tower-defense game, flOw, a game where you eat stuff by using the motion-sensors in the controller, Lemmings and PAIN, a physics game that involves slingshotting a character into stuff and watching them scream. Lemmings is slightly handicapped by the PS3 controller, but I really like PixelJunk Monsters and flOw, I’ve certainly gotten my $10 of enjoyment out of them. PAIN is too hard to unlock and doesn’t have enough single-player content to keep my interest.

The most recent PS3 update replaced the old store with a brand new one that’s faster to navigate, unless you want to look at something. It downloads item images as you navigate and doesn’t cache them between runs which is annoying. The new store now knows which demos you’ve downloaded which is a really nice feature, though.

The DualShock 3 controller is much nicer than the SIXAXIS, by both being slightly heavier and having slightly stiffer buttons. Compared to the PS2 controller, the vibration doesn’t seem as powerful and the motors seem noisier. (I’m not sure, as my PS2 is in pieces and I doubt I can keep it alive long enough to start a game.)

The one extra thing I bought for the PS3 is the NYKO Blu-Wave remote so I could control the PS3 with my universal remote (which doesn’t speak Bluetooth) through the Blu-Wave’s USB IR dongle. The remote can be found on Amazon for under $20.

Posted in Misc | no comments

Graphing Spam

Eric Hodel | Mon, 24 Dec 2007 11:49:00 GMT

For fun I decided to chart various statistics from my mail server’s logs. I ended up with these charts using John Barnette’s GChart, some regular expressions, and cron.

The details of parsing out the data from the log files and setting up a crontab is boring, so I’ll spare you that.

Using GChart is really cool and easy. Here’s what I used to generate the Amavis Statistics chart, the others are all nearly the same, just different labels and titles.

def graph_amavis(data)
  # These are the labels and colors I'm using 
  labels = %w[Banned Spam   Spammy Bad\ Header Clean]
  colors = %w[000000 ff0000 ff7f00 ffff00      00ff00]

  max = data.map { |vals| vals.max }.max

  # Since axis labels aren't yet supported by the API in 0.2.0, I use :extras
  extras = { 'chxt' => 'r', 'chxl' => "0:|#{axis_labels max}" }

  chart = GChart.line :title => 'Amavis Statistics', :data => data,
                      :labels => labels, :colors => colors, :extras => extras

  chart_path = File.join File.dirname(@file), 'amavis_statistics.png'

  chart.size = '750x400'# can't be > 300,000 pixels
  chart.write chart_path
end

That’s it! One minor gotcha I had was that I needed to transpose the data set after reading it in because it wasn’t in the right order for GChart to consume it, but with Array#transpose, it’s just an extra method call before graphing.

Also, I wrote this simple utility function to calculate some good-enough axis labels:

def axis_labels(max)
  axis_labels = []
  0.upto 10 do |i| axis_labels << (max * 0.1 * i).to_i end
  axis_labels.join '|'
end

The values aren’t prettily chosen, but they work well enough.

Posted in Misc | no comments