My House Knows I'm Home

Eric Hodel | Fri, 02 Jul 2010 00:55:26 GMT

I bought a house last fall and upgraded it with Insteon switches (from MacHomeStore). I run Indigo on a Mac Mini to automate them all.

I wanted my house to know if it was or wasn't occupied so it could "do stuff" like turning off lights that were accidentally left on.

My girlfriend and I both have iPhones and while I could use the Find My iPhone feature, it isn't really accurate enough to know if I'm home and would probably drain the battery pretty quick if queried every five minutes.

I'm not sure if looking for the iPhones via ping or arp would work either, I've read the iPhone will shut off the WiFi adapter when it's not in use unless it's polling mail updates.

The iPhone has Bluetooth and it can be left on all the time without much battery loss. There's a handful of OS X apps that can execute a script when a bluetooth device comes in range but they only work for one device. Since my girlfriend also lives with me they won't work.

One of my coworkers got me to help him make a seemingly-abandoned Ruby Bluetooth library work on OS X using IOBluetooth.

The OS X side of the Bluetooth library started out with just the ability to scan for devices but I added remote name request, pairing, connection creation and signal strength.

It turns out that it's easy to see if a device is around once you have its address by asking for its name. There's no need to pair the device or create a connection (which will drain the battery faster).

Pairing and signal strength were implemented for the fun of walking around the house with the iPhone running a terminal session that monitored the signal strength and to see how well the Bluetooth signal works through my walls and floors.

Now that I could tell if the phones were nearby I needed to update Indigo with an occupied status. Indigo can monitor the occupied status and do something when it changes. The server is scriptable using AppleScript, so I used rubyosa to update Indigo with the presence of the phones. (The released rubyosa doesn't work on OS X 10.6, so I installed this version instead.)

All I needed now was the ability to run the script on a regular basis. I used Lingon to create a launchd agent that would run the script every five minutes. I told Indigo to turn off my interior lights when the house is no longer occupied, so if I leave the house (or turn the phones off) the lights automatically turn off.

I may also be able to do something more fancy like increase the poll interval once the house goes unoccupied to help it detect when I arrive home and do something like turn on the fireplace if it's cold, or turn on the entryway lights if it's dark.

Since I'm using Bluetooth my tools will work with any phone that can leave its radio on all the time. The phone doesn't need to be discoverable.

Hopefully I'll be able to release the Ruby Bluetooth library soon. I haven't been able to contact the original author and I'm not sure what the license is. I've replaced nearly all of the OS X portion of the library, but there's also extensive code for Linux and Windows.

Hopefully I'll also be able to convince Laurent to release an updated rubyosa that will work on OS X 10.6.

Posted in Misc | no comments

curb 0.7.3 performs as expected

Eric Hodel | Wed, 19 May 2010 20:32:31 GMT

Over loopback, curb runs about 2.4x the speed of net-http-persistent. This is the performance I expected curb to give. Having a pure-ruby library perform 2.4x worse than a C library makes net/http a fantastic library.

Remember, this benchmark and the previous one were designed to model a web-service type workload where many small requests are made. These benchmarks aren't designed to model downloading large files.

require 'rubygems'
require 'benchmark'
require 'net/http/persistent'
require 'curb'

# dd if=/dev/zero of=~/Sites/zeros-1k bs=1024 c=1
uri_1k  = URI.parse 'http://localhost/~drbrain/zeros-1k'
uri_2k  = URI.parse 'http://localhost/~drbrain/zeros-2k'
uri_10k = URI.parse 'http://localhost/~drbrain/zeros-10k'

uri = uri_2k

N = 100_000

Benchmark.bmbm do |bm|
  bm.report 'Net::HTTP::Persistent' do
    http_p = Net::HTTP::Persistent.new

    N.times do
      response = http_p.request uri
      response.body
    end
  end

  bm.report 'curb' do
    curl = Curl::Easy.new
    url = uri.to_s

    N.times do
      curl.url = url
      curl.perform
      curl.body_str
    end
  end
end

Rehearsal ---------------------------------------------------------
Net::HTTP::Persistent  54.070000   3.930000  58.000000 ( 71.651048)
curb                   11.800000   4.510000  16.310000 ( 30.930453)
----------------------------------------------- total: 74.310000sec

                            user     system      total        real
Net::HTTP::Persistent  54.130000   3.930000  58.060000 ( 72.430575)
curb                   11.860000   4.540000  16.400000 ( 30.804222)

Posted in Misc, Ruby | 2 comments

Example domains and IPs

Eric Hodel | Fri, 02 Oct 2009 21:58:28 GMT

I'm always forgetting this information. Here's a handy reference for example domain names, example IPv4 addresses and example IPv6 addresses.

Example Domain names

From RFC 2606:

.test

For testing DNS related code

example

example.com

example.net

example.org

For documentation or as examples

IPv4 Addresses

From RFC 3330:

192.0.2.0/24 is used for documentation and example code.

198.18.0.0/15 is used for benchmark tests of network interconnect devices. See RFC 2544

IPv6 Addresses

From RFC 3849:

2001:DB8::/32 is used for documentation and examples.

Posted in Misc | no comments

Service Discovery Protocols

Eric Hodel | Wed, 19 Aug 2009 21:42:03 GMT

I’ve worked on two different service discovery libraries for ruby, one for SSPD and one for mDNS, and I’d like to share the differences between them with you.

Both SSDP and mDNS are designed to be used with self-configuring networks and to automatically discover and connect to other devices. Both use multicast to advertise and search.

SSPD

The Simple Service Discovery Protocol is used primarily by UPnP for network devices to discover each other. UPnP is designed for consumer electronics and appliances that need to communicate with each other automatically.

The description of SSPD fits on a twelve of pages when written in (relatively) plain English (section 1 in the UPNP Device Architecture). The protocol itself looks much like HTTP and can be implemented largely using simple text processing. (Strangely, SOAP is the higher-level communication layer for UPnP.)

Unfortunately there’s no way for multiple devices to share the same SSPD socket without being designed to work with each other. Since UPnP is focused on simple devices, this fits well with their design goals. Most UPnP devices are routers, media servers, game consoles, televisions, etc. If you built your own router and media server on the same host it will probably be impossible for both devices to show up on the network, nor will you be able to run a media server and media viewer on the same host without the two being designed to work with each other.

mDNS

Multicast DNS is used primarily by Apple and (basically) has a DNS server on each host. Hosts speak the DNS protocol to each other over multicast and have a central daemon that handles communication on the network much like the libc DNS resolver.

Even though there’s plenty of DNS clients around for code reuse, implementing a DNS client form scratch is a large undertaking (my SSDP implementation is around 750 lines with documentation, ruby’s Resolv DNS resolver is around 2200 lines and doesn’t include a server or multicast code). The entirety of the DNS system is spread across several RFCs, not including the documentation for mDNS itself.

Fortunately mDNS libraries provide a daemon for handling network communication and a client library you can use to register and browse for devices (the dnssd gem uses Apple’s DNS Service Discovery API). mDNS is focused on computers rather than devices, so multiple services can all coexist on the same host.

Unfortunately, due to the complexity of mDNS, there’s no single API for applications to use. Apple’s DNS-SD works across multiple platforms including windows. For unix-like operating systems there is also avahi which has a limited dnssd compatibility shim.

Posted in Misc | no comments

vi bindings for irb on OS X

Eric Hodel | Tue, 11 Aug 2009 20:56:22 GMT

OS X uses editline(3) instead of readline(3) so ~/.inputrc doesn’t do anything for irb or other tools using readline via the editline wrapper.

Instead, use ~/.editrc:

bind -v
bind \\t rl_complete

Which gives you vi bindings in irb.

Update

Now with tab completion thanks to Curt Sampson from an ancient netbsd-users email!

Posted in Misc, Ruby | no comments

Mail Hosting

Eric Hodel | Sat, 09 May 2009 04:28:43 GMT

I host my own mail using the following software:

• FreeBSD for my OS
• Postfix for SMTP
• OpenBSD’s spamd for greylisting
• amavisd for spam control (which uses SpamAssassin) and DKIM signing
• dovecot for IMAP and SASL
• procmail for mail filtering

OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.

Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).

Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)

In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:

DELIVER = /usr/local/libexec/dovecot/deliver

# ...

:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems

# ...

# last rule, delivers to INBOX
:0 w
| ${DELIVER}

My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).

I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.

To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.

My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).

Posted in FreeBSD, Misc | no comments

Binding#remove_local_variable

Eric Hodel | Fri, 24 Apr 2009 00:29:07 GMT

require 'rubygems'
require 'inline'

class Binding
  inline do |builder|
    builder.include '"node.h"'
    builder.include '"env.h"'

    ##
    # struct BLOCK isn't in any header, so include it here.

    builder.prefix <<-C
struct BLOCK {
  NODE *var;
  NODE *body;
  VALUE self;
  struct FRAME frame;
  struct SCOPE *scope;
  VALUE klass;
  NODE *cref;
  int iter;
  int vmode;
  int flags;
  int uniq;
  struct RVarmap *dyna_vars;
  VALUE orig_thread;
  VALUE wrapper;
  VALUE block_obj;
  struct BLOCK *outer;
  struct BLOCK *prev;
};
    C

    ##
    # :method: remove_local_variable
    #
    # Removes the local variable +name+ and replaces it with nil.
    #
    # Ordinarily if a local variable didn't exist it would raise
    # NoMethodError, but currently after #remove_local_variable it doesn't, it
    # just returns nil.
    #
    # In order to make this behave correctly, block->body would need to be
    # walked, duping nodes (so as not to affect future invocations of this
    # method) and replacing the LVAR nodes with VCALL nodes.
    #
    # That's just too much work for 17:15, though.  Maybe tomorrow.

    builder.c <<-C
VALUE remove_local_variable(VALUE name) {
  struct BLOCK *block;
  struct SCOPE *scope;
  ID name_id;
  ID *local_table;
  int i, n;
  VALUE entry;

  name_id = SYM2ID(name);

  Data_Get_Struct(self, struct BLOCK, block);

  scope = block->scope;
  local_table = scope->local_tbl;

  if (local_table) {
    n = *local_table++;
    for (i = 2; i < n; i++) { /* skip $_ and $~ */
      if (!rb_is_local_id(local_table[i])) continue; /* skip flip states */
      if (local_table[i] == name_id) {
        entry = scope->local_vars[i];
        local_table[i] = (ID)NULL;
        scope->local_vars[i] = Qnil;
        return entry;
      }
    }
  }

  return Qnil;
}
    C

  end
end

a = :my_value

p :lvar_a => a

b = binding
p :remove_local_variable_says => b.remove_local_variable(:a)

p :lvar_a => a # TODO raise exception

Posted in Misc, Ruby | 2 comments

RubyGems code_swarm

Eric Hodel | Wed, 28 Jan 2009 00:59:32 GMT

I built a code_swarm visualization of the RubyGems repository history:

RubyGems code_swarm (HD) from Eric Hodel on Vimeo.

Coincidentally, you can find a code_swarm visualization for Rails and other projects by Ilya Grigorik. He points to Peter Burns’ fork on GitHub which has a much improved toolset for generating these types of visualizations.

I used the original repository which involves a lot more work. For RubyGems I did roughly the following:

1. Get a log: svn log -v > rubygems.log
2. Convert the log: python convert_logs.py -s rubygems.log -o rubygems.xml
3. Munge the XML to remove the CVS conversion commits and hide the renames from trunk/rubygems/ to trunk/
4. Run code_swarm: code_swarm rubygems.config
5. Encode PNGs to an MP4: ffmpeg -f image2 -b 1500 -r 24 -i frames/rubygems-%05d.png -sameq rubygems.mp4
6. Upload to Vimeo

I played around with the code_swarm configuration a bit and ended up with this config for code_swarm. My changes give a larger video with more lingering of files as they float around and bumps up font sizes for readability. For more active projects you may need to decrease FileLife and PersonLife. (Really, I’m not sure how they affect things.)

Posted in Misc, Rubygems | 3 comments

Friendly Ruby Objects

Eric Hodel | Thu, 18 Dec 2008 01:28:00 GMT

This post is intended to supplement the Ruby Quickref with the various ways you can make your objects play nicely with each other.

Most of the examples below are taken from RubyGems, some examples won't work until the next release of RubyGems.

Enumerable

The Enumerable module is based on the #each method and contains well known methods like #map and #each_with_index. If the enumerated objects implement the #<=> method you get a useful #sort, #min and #max.

Gem::SourceIndex has a Hash internally that it exposes via #each:

class Gem::SourceIndex
  include Enumerable

  # ...

  def each(&block)
    @gems.each(&block)
  end
end

Which allows handy things like:

dep = Gem::Dependency.new ARGV.shift, Gem::Requirement.default

found = Gem.source_index.any? do |name, spec|
  dep =~ spec
end

puts "found gem for #{dep.name}!" if found

Comparable

The Comparable module is based on the #<=> method and gives all the comparison methods.

Gem::Specification objects are sorted via name, version and platform:

class Gem::Specification
  include Comparable

  def <=>(other)
    my_platform = Gem::Platform::RUBY == @platform ? -1 : 1
    other_platform = Gem::Platform::RUBY == other.platform ? -1 : 1

    [@name, @version, platform] <=>
      [other.name, other.version, other.platform]
  end
end

This is used in RubyGems to sort objects both for display on the screen like in gem list and internally when installing gems.

For a way to reduce the repetition in the above code and some other sorting speed-ups, see my post on #sort_by and #sort_obj.

#to_s and #inspect

Overriding #to_s and #inspect prevent you from puking all over the screen when somebody wants to look at your object. To a certain extent limiting the amount of information shown can aid debugging.

Gem::Specification's #to_s that gives only the two most-important attributes:

class Gem::Specification
  def to_s
    "#<Gem::Specification name=#{@name} version=#{@version}>"
  end
end

Gem::Platform's #to_s gives a friendly string:

class Gem::Platform
  def to_a
    [@cpu, @os, @version]
  end

  def to_s
    to_a.compact.join '-'
  end
end

Gem::Version's inspect ignores internal instance variables and only exposes @version:

class Gem::Version
  def inspect # :nodoc:
    "#<#{self.class} #{@version.inspect}>"
  end
end

Case equality with #===, Matching with #=~

While #=== is more commonly overridden, it can also be useful to implement #=~ to allow your objects to be used in a very readable manner.

Gem::Platform overrides #===:

class Gem::Platform
  def ===(other)
    return nil unless Gem::Platform === other

    # cpu
    (@cpu == 'universal' or other.cpu == 'universal' or @cpu == other.cpu) and

    # os
    @os == other.os and

    # version
    (@version.nil? or other.version.nil? or @version == other.version)
  end
end

In short, one platform matches another if they have the same cpu (architecture) or either is universal, they have the same os and their versions match if they have versions.

You can use this to group gems:

platform_count = Hash.new 0

Gem.source_index.each do |name, spec|
  case spec.platform
  when Gem::Platform.new('linux') then
    platform_count['linux'] += 1
  # ...
  else
    platform_count['other'] += 1
  end
end

p platform_count

Gem::Dependency overrides #=~. It is a little strange because it converts the right-hand side to a Gem::Dependency object:

class Gem::Dependency
  def =~(other)
    other = case other
            when self.class then
              other
            else
              return false unless other.respond_to? :name and
                                  other.respond_to? :version

              Gem::Dependency.new other.name, other.version
            end

    pattern = @name
    pattern = /\A#{Regexp.escape @name}\Z/ unless Regexp === pattern

    return false unless pattern =~ other.name

    reqs = other.version_requirements.requirements

    return false unless reqs.length == 1
    return false unless reqs.first.first == '='

    version = reqs.first.last

    version_requirements.satisfied_by? version
  end
end

You can use this as a filter:

dep = Gem::Dependency.new(/ruby/, Gem::Requirement.default)

ruby_named = Gem.source_index.select do |name, spec|
  dep =~ spec
end

p ruby_named.map { |name, spec| name }

As a Hash key

Ruby uses #hash and #eql? to determine if two different objects really mean the same hash key.

Gem::Version is usable as a Hash key based on the internal version string:

class Gem::Version
  def hash
    @version.hash
  end

  def eql?(other)
    self.class === other and @version == other.version
  end
end

In Gem::Version, the internal version string looks like "1.3" or "1.3.0". In this implementation the two versions would belong to different hash keys.

Using #eql? instead of #== to determine if two keys are the same is a nice distinction since it allows you to have interesting behaviors (but I'm not sure they are useful). For Gem::Version, a version of "1.3" is equal to "1.3.0", but the occupy different slots in a Hash.

#intialize_copy

#initialize_copy is called during #dup and #clone to copy object-specific state beyond instance variables. The object being cloned from is passed to the new instance. #initialize_copy could be used for cleaning out a per-object cache:

def initialize_copy(other)
  @cache = []
end

#exception

#exception is called on objects given to #raise to cast them to Exception objects. It must return a subclass of Exception. You can use this to turn arbitrary objects into exceptions, centralizing all your exception raising code:

class Result
  class Error < RuntimeError; end

  def initialize(json)
    @result = JSON.parse json
  end

  def exception(message = nil)
    Error.new "#{message} (#{@result['error']})"
  end

  def [](key)
    @result[key]
  end
end

r = Result.new open('http://example.com/api/blah').read

raise r if r['error']

Marshal

Ruby will marshal most objects automatically, but sometimes you want a custom format to ignore cached data that can be reconstructed or to reduce the size of the data you're saving out. There are two ways to do this, the older way is #_dump/::_load and the newer way is #marshal_dump/#marshal_load which takes priority. If you want to upgrade to the newer way you can leave ::_load to restore older marshaled objects.

Using the older way #_dump returns a String representation of the object (usually another Marshal string) and ::_load receives that String representation. Note that it's a class method, so ::_load is responsible for creating the object, which may be important in some instances.

Using the newer way, #marshal_dump returns an Object and #marshal_load receives that Object. The object is already allocated, but #initialize won't be called. The newer way can result in a smaller marshal dump size since it uses the existing symbol and object reference tables.

Gem::Specification uses #_dump/::load and is fairly complicated because I designed it to be backward and forward-compatible. This is a slightly-stripped-down version:

class Gem::Specification

  CURRENT_SPECIFICATION_VERSION = 2

  # number of fields per version
  MARSHAL_FIELDS = { -1 => 16, 1 => 16, 2 => 16 }

  def self._load(str)
    array = Marshal.load str

    spec = Gem::Specification.new
    spec.instance_variable_set :@specification_version, array[1]

	# validate object
    current_version = CURRENT_SPECIFICATION_VERSION

    field_count = if spec.specification_version > current_version then
                    spec.instance_variable_set :@specification_version,
                                               current_version
                    MARSHAL_FIELDS[current_version]
                  else
                    MARSHAL_FIELDS[spec.specification_version]
                  end

    if array.size < field_count then
      raise TypeError, "invalid Gem::Specification format #{array.inspect}"
    end

	# restore object
    spec.instance_variable_set :@rubygems_version,          array[0]
    # ...
    spec.instance_variable_set :@platform,                  array[16].to_s
    spec.instance_variable_set :@loaded,                    false

    spec
  end

  def _dump(limit)
    Marshal.dump [
      @rubygems_version,
      @specification_version,
      # ...
      @new_platform,
    ]
  end
end

Gem::Version uses #marshal_dump/#marshal_load and ignores the internal instance variables, only dumping @version:

class Gem::Version
  def marshal_dump
    [@version]
  end

  def marshal_load(array)
    self.version = array[0]
  end
end

Pretty-print with PP

With a little work PP can give you easily readable output for your objects, even output that you can copy and paste back into a script. Primarily you'll use the PrettyPrint#group, PrettyPrint#text, PrettyPrint#breakable and PP#pp methods inside a #pretty_print method on your object. You can find documentation for these methods using ri.

Here's #pretty_print from Gem::Dependency and Gem::Requirement:

class Gem::Dependency
  def pretty_print(q)
    q.group 1, 'Gem::Dependency.new(', ')' do
      q.pp @name
      q.text ','
      q.breakable

      q.pp @version_requirements

      q.text ','
      q.breakable

      q.pp @type
    end
  end
end

class Gem::Requirement
  def pretty_print(q)
    q.group 1, 'Gem::Requirement.new(', ')' do
      q.pp as_list
    end
  end

  def as_list
    normalize
    @requirements.map do |op, version| "#{op} #{version}" end
  end
end

Together these make pretty, copy-pastable output:

require 'pp'

gem 'ParseTree'

pp Gem.loaded_specs["ParseTree"].dependencies

[Gem::Dependency.new("RubyInline",
  Gem::Requirement.new([">= 3.7.0"]),
  :runtime),
 Gem::Dependency.new("sexp_processor",
  Gem::Requirement.new([">= 3.0.0"]),
  :runtime),
 Gem::Dependency.new("hoe", Gem::Requirement.new([">= 1.8.0"]), :development)]

Posted in Misc, Ruby | 6 comments

Working Style

Eric Hodel | Fri, 22 Aug 2008 00:42:25 GMT

In Rubinius we’ve been slowly working towards using a VM written in C++ instead of the current C VM we’ve got (called “shotgun”). Most of the mechanics of the VM are already complete, so over the past couple weeks we’ve been wiring up the primitives necessary for our ruby kernel (equivalent to Ruby’s core library) to run.

This primitive work is rather tedious, and until Friday I found it highly unenjoyable. I couldn’t perceive how much closer I was to our goal of actually running ruby code on the new VM.

On Friday I recalled that we can run a compiled ruby file against the VM, which gave me a way to shoot towards my goal. Over the past couple days during our Rubinius Team Meeting we’ve gotten up to loading all of our kernel bootstrap and platform, and now we’re just starting to do work in the core libraries.

Switching my working style drastically improved my enjoyment of a process that I found quite annoying. I wish I’d figured this out a couple weeks ago.

Posted in Misc, Rubinius | no comments