My House Knows I'm Home
Eric Hodel | Fri, 02 Jul 2010 00:55:26 GMT
I bought a house last fall and upgraded it with Insteon switches (from MacHomeStore). I run Indigo on a Mac Mini to automate them all.
I wanted my house to know if it was or wasn't occupied so it could "do stuff" like turning off lights that were accidentally left on.
My girlfriend and I both have iPhones and while I could use the Find My iPhone feature, it isn't really accurate enough to know if I'm home and would probably drain the battery pretty quick if queried every five minutes.
I'm not sure if looking for the iPhones via ping or arp would work either, I've read the iPhone will shut off the WiFi adapter when it's not in use unless it's polling mail updates.
The iPhone has Bluetooth and it can be left on all the time without much battery loss. There's a handful of OS X apps that can execute a script when a bluetooth device comes in range but they only work for one device. Since my girlfriend also lives with me they won't work.
One of my coworkers got me to help him make a seemingly-abandoned Ruby Bluetooth library work on OS X using IOBluetooth.
The OS X side of the Bluetooth library started out with just the ability to scan for devices but I added remote name request, pairing, connection creation and signal strength.
It turns out that it's easy to see if a device is around once you have its address by asking for its name. There's no need to pair the device or create a connection (which will drain the battery faster).
Pairing and signal strength were implemented for the fun of walking around the house with the iPhone running a terminal session that monitored the signal strength and to see how well the Bluetooth signal works through my walls and floors.
Now that I could tell if the phones were nearby I needed to update Indigo with an occupied status. Indigo can monitor the occupied status and do something when it changes. The server is scriptable using AppleScript, so I used rubyosa to update Indigo with the presence of the phones. (The released rubyosa doesn't work on OS X 10.6, so I installed this version instead.)
All I needed now was the ability to run the script on a regular basis. I used Lingon to create a launchd agent that would run the script every five minutes. I told Indigo to turn off my interior lights when the house is no longer occupied, so if I leave the house (or turn the phones off) the lights automatically turn off.
I may also be able to do something more fancy like increase the poll interval once the house goes unoccupied to help it detect when I arrive home and do something like turn on the fireplace if it's cold, or turn on the entryway lights if it's dark.
Since I'm using Bluetooth my tools will work with any phone that can leave its radio on all the time. The phone doesn't need to be discoverable.
Hopefully I'll be able to release the Ruby Bluetooth library soon. I haven't been able to contact the original author and I'm not sure what the license is. I've replaced nearly all of the OS X portion of the library, but there's also extensive code for Linux and Windows.
Hopefully I'll also be able to convince Laurent to release an updated rubyosa that will work on OS X 10.6.
curb 0.7.3 performs as expected
Eric Hodel | Wed, 19 May 2010 20:32:31 GMT
Over loopback, curb runs about 2.4x the speed of net-http-persistent. This is the performance I expected curb to give. Having a pure-ruby library perform 2.4x worse than a C library makes net/http a fantastic library.
Remember, this benchmark and the previous one were designed to model a web-service type workload where many small requests are made. These benchmarks aren't designed to model downloading large files.
require 'rubygems'
require 'benchmark'
require 'net/http/persistent'
require 'curb'
# dd if=/dev/zero of=~/Sites/zeros-1k bs=1024 c=1
uri_1k = URI.parse 'http://localhost/~drbrain/zeros-1k'
uri_2k = URI.parse 'http://localhost/~drbrain/zeros-2k'
uri_10k = URI.parse 'http://localhost/~drbrain/zeros-10k'
uri = uri_2k
N = 100_000
Benchmark.bmbm do |bm|
bm.report 'Net::HTTP::Persistent' do
http_p = Net::HTTP::Persistent.new
N.times do
response = http_p.request uri
response.body
end
end
bm.report 'curb' do
curl = Curl::Easy.new
url = uri.to_s
N.times do
curl.url = url
curl.perform
curl.body_str
end
end
end
Rehearsal ---------------------------------------------------------
Net::HTTP::Persistent 54.070000 3.930000 58.000000 ( 71.651048)
curb 11.800000 4.510000 16.310000 ( 30.930453)
----------------------------------------------- total: 74.310000sec
user system total real
Net::HTTP::Persistent 54.130000 3.930000 58.060000 ( 72.430575)
curb 11.860000 4.540000 16.400000 ( 30.804222)
Example domains and IPs
Eric Hodel | Fri, 02 Oct 2009 21:58:28 GMT
I'm always forgetting this information. Here's a handy reference for example domain names, example IPv4 addresses and example IPv6 addresses.
Example Domain names
From RFC 2606:
- .test
- For testing DNS related code
- example
- example.com
- example.net
- example.org
- example.com
- For documentation or as examples
IPv4 Addresses
From RFC 3330:
192.0.2.0/24 is used for documentation and example code.
198.18.0.0/15 is used for benchmark tests of network interconnect devices. See RFC 2544
IPv6 Addresses
From RFC 3849:
2001:DB8::/32 is used for documentation and examples.
Service Discovery Protocols
Eric Hodel | Wed, 19 Aug 2009 21:42:03 GMT
I’ve worked on two different service discovery libraries for ruby, one for SSPD and one for mDNS, and I’d like to share the differences between them with you.
Both SSDP and mDNS are designed to be used with self-configuring networks and to automatically discover and connect to other devices. Both use multicast to advertise and search.
SSPD
The Simple Service Discovery Protocol is used primarily by UPnP for network devices to discover each other. UPnP is designed for consumer electronics and appliances that need to communicate with each other automatically.
The description of SSPD fits on a twelve of pages when written in (relatively) plain English (section 1 in the UPNP Device Architecture). The protocol itself looks much like HTTP and can be implemented largely using simple text processing. (Strangely, SOAP is the higher-level communication layer for UPnP.)
Unfortunately there’s no way for multiple devices to share the same SSPD socket without being designed to work with each other. Since UPnP is focused on simple devices, this fits well with their design goals. Most UPnP devices are routers, media servers, game consoles, televisions, etc. If you built your own router and media server on the same host it will probably be impossible for both devices to show up on the network, nor will you be able to run a media server and media viewer on the same host without the two being designed to work with each other.
mDNS
Multicast DNS is used primarily by Apple and (basically) has a DNS server on each host. Hosts speak the DNS protocol to each other over multicast and have a central daemon that handles communication on the network much like the libc DNS resolver.
Even though there’s plenty of DNS clients around for code reuse, implementing a DNS client form scratch is a large undertaking (my SSDP implementation is around 750 lines with documentation, ruby’s Resolv DNS resolver is around 2200 lines and doesn’t include a server or multicast code). The entirety of the DNS system is spread across several RFCs, not including the documentation for mDNS itself.
Fortunately mDNS libraries provide a daemon for handling network communication and a client library you can use to register and browse for devices (the dnssd gem uses Apple’s DNS Service Discovery API). mDNS is focused on computers rather than devices, so multiple services can all coexist on the same host.
Unfortunately, due to the complexity of mDNS, there’s no single API for applications to use. Apple’s DNS-SD works across multiple platforms including windows. For unix-like operating systems there is also avahi which has a limited dnssd compatibility shim.
vi bindings for irb on OS X
Eric Hodel | Tue, 11 Aug 2009 20:56:22 GMT
OS X uses editline(3) instead of readline(3) so ~/.inputrc doesn’t do anything for irb or other tools using readline via the editline wrapper.
Instead, use ~/.editrc:
bind -v
bind \\t rl_complete
Which gives you vi bindings in irb.
Update
Now with tab completion thanks to Curt Sampson from an ancient netbsd-users email!
Mail Hosting
Eric Hodel | Sat, 09 May 2009 04:28:43 GMT
I host my own mail using the following software:
- FreeBSD for my OS
- Postfix for SMTP
- OpenBSD’s spamd for greylisting
- amavisd for spam control (which uses SpamAssassin) and DKIM signing
- dovecot for IMAP and SASL
- procmail for mail filtering
OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.
Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).
Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)
In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:
DELIVER = /usr/local/libexec/dovecot/deliver
# ...
:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems
# ...
# last rule, delivers to INBOX
:0 w
| ${DELIVER}
My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).
I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.
To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.
My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).
Binding#remove_local_variable
Eric Hodel | Fri, 24 Apr 2009 00:29:07 GMT
require 'rubygems'
require 'inline'
class Binding
inline do |builder|
builder.include '"node.h"'
builder.include '"env.h"'
##
# struct BLOCK isn't in any header, so include it here.
builder.prefix <<-C
struct BLOCK {
NODE *var;
NODE *body;
VALUE self;
struct FRAME frame;
struct SCOPE *scope;
VALUE klass;
NODE *cref;
int iter;
int vmode;
int flags;
int uniq;
struct RVarmap *dyna_vars;
VALUE orig_thread;
VALUE wrapper;
VALUE block_obj;
struct BLOCK *outer;
struct BLOCK *prev;
};
C
##
# :method: remove_local_variable
#
# Removes the local variable +name+ and replaces it with nil.
#
# Ordinarily if a local variable didn't exist it would raise
# NoMethodError, but currently after #remove_local_variable it doesn't, it
# just returns nil.
#
# In order to make this behave correctly, block->body would need to be
# walked, duping nodes (so as not to affect future invocations of this
# method) and replacing the LVAR nodes with VCALL nodes.
#
# That's just too much work for 17:15, though. Maybe tomorrow.
builder.c <<-C
VALUE remove_local_variable(VALUE name) {
struct BLOCK *block;
struct SCOPE *scope;
ID name_id;
ID *local_table;
int i, n;
VALUE entry;
name_id = SYM2ID(name);
Data_Get_Struct(self, struct BLOCK, block);
scope = block->scope;
local_table = scope->local_tbl;
if (local_table) {
n = *local_table++;
for (i = 2; i < n; i++) { /* skip $_ and $~ */
if (!rb_is_local_id(local_table[i])) continue; /* skip flip states */
if (local_table[i] == name_id) {
entry = scope->local_vars[i];
local_table[i] = (ID)NULL;
scope->local_vars[i] = Qnil;
return entry;
}
}
}
return Qnil;
}
C
end
end
a = :my_value
p :lvar_a => a
b = binding
p :remove_local_variable_says => b.remove_local_variable(:a)
p :lvar_a => a # TODO raise exception
RubyGems code_swarm
Eric Hodel | Wed, 28 Jan 2009 00:59:32 GMT
I built a code_swarm visualization of the RubyGems repository history:
RubyGems code_swarm (HD) from Eric Hodel on Vimeo.
Coincidentally, you can find a code_swarm visualization for Rails and other projects by Ilya Grigorik. He points to Peter Burns’ fork on GitHub which has a much improved toolset for generating these types of visualizations.
I used the original repository which involves a lot more work. For RubyGems I did roughly the following:
- Get a log: svn log -v > rubygems.log
- Convert the log: python convert_logs.py -s rubygems.log -o rubygems.xml
- Munge the XML to remove the CVS conversion commits and hide the renames from trunk/rubygems/ to trunk/
- Run code_swarm: code_swarm rubygems.config
- Encode PNGs to an MP4: ffmpeg -f image2 -b 1500 -r 24 -i frames/rubygems-%05d.png -sameq rubygems.mp4
- Upload to Vimeo
I played around with the code_swarm configuration a bit and ended up with this config for code_swarm. My changes give a larger video with more lingering of files as they float around and bumps up font sizes for readability. For more active projects you may need to decrease FileLife and PersonLife. (Really, I’m not sure how they affect things.)
Friendly Ruby Objects
Eric Hodel | Thu, 18 Dec 2008 01:28:00 GMT
This post is intended to supplement the Ruby Quickref with the various ways you can make your objects play nicely with each other.
Most of the examples below are taken from RubyGems, some examples won't work until the next release of RubyGems.
Enumerable
The Enumerable module is based on the #each method and contains well known methods like #map and #each_with_index. If the enumerated objects implement the #<=> method you get a useful #sort, #min and #max.
Gem::SourceIndex has a Hash internally that it exposes via #each:
class Gem::SourceIndex
include Enumerable
# ...
def each(&block)
@gems.each(&block)
end
end
Which allows handy things like:
dep = Gem::Dependency.new ARGV.shift, Gem::Requirement.default
found = Gem.source_index.any? do |name, spec|
dep =~ spec
end
puts "found gem for #{dep.name}!" if found
Comparable
The Comparable module is based on the #<=> method and gives all the comparison methods.
Gem::Specification objects are sorted via name, version and platform:
class Gem::Specification
include Comparable
def <=>(other)
my_platform = Gem::Platform::RUBY == @platform ? -1 : 1
other_platform = Gem::Platform::RUBY == other.platform ? -1 : 1
[@name, @version, platform] <=>
[other.name, other.version, other.platform]
end
end
This is used in RubyGems to sort objects both for display on the screen like in gem list and internally when installing gems.
For a way to reduce the repetition in the above code and some other sorting speed-ups, see my post on #sort_by and #sort_obj.
#to_s and #inspect
Overriding #to_s and #inspect prevent you from puking all over the screen when somebody wants to look at your object. To a certain extent limiting the amount of information shown can aid debugging.
Gem::Specification's #to_s that gives only the two most-important attributes:
class Gem::Specification
def to_s
"#<Gem::Specification name=#{@name} version=#{@version}>"
end
end
Gem::Platform's #to_s gives a friendly string:
class Gem::Platform
def to_a
[@cpu, @os, @version]
end
def to_s
to_a.compact.join '-'
end
end
Gem::Version's inspect ignores internal instance variables and only exposes @version:
class Gem::Version
def inspect # :nodoc:
"#<#{self.class} #{@version.inspect}>"
end
end
Case equality with #===, Matching with #=~
While #=== is more commonly overridden, it can also be useful to implement #=~ to allow your objects to be used in a very readable manner.
Gem::Platform overrides #===:
class Gem::Platform
def ===(other)
return nil unless Gem::Platform === other
# cpu
(@cpu == 'universal' or other.cpu == 'universal' or @cpu == other.cpu) and
# os
@os == other.os and
# version
(@version.nil? or other.version.nil? or @version == other.version)
end
end
In short, one platform matches another if they have the same cpu (architecture) or either is universal, they have the same os and their versions match if they have versions.
You can use this to group gems:
platform_count = Hash.new 0
Gem.source_index.each do |name, spec|
case spec.platform
when Gem::Platform.new('linux') then
platform_count['linux'] += 1
# ...
else
platform_count['other'] += 1
end
end
p platform_count
Gem::Dependency overrides #=~. It is a little strange because it converts the right-hand side to a Gem::Dependency object:
class Gem::Dependency
def =~(other)
other = case other
when self.class then
other
else
return false unless other.respond_to? :name and
other.respond_to? :version
Gem::Dependency.new other.name, other.version
end
pattern = @name
pattern = /\A#{Regexp.escape @name}\Z/ unless Regexp === pattern
return false unless pattern =~ other.name
reqs = other.version_requirements.requirements
return false unless reqs.length == 1
return false unless reqs.first.first == '='
version = reqs.first.last
version_requirements.satisfied_by? version
end
end
You can use this as a filter:
dep = Gem::Dependency.new(/ruby/, Gem::Requirement.default)
ruby_named = Gem.source_index.select do |name, spec|
dep =~ spec
end
p ruby_named.map { |name, spec| name }
As a Hash key
Ruby uses #hash and #eql? to determine if two different objects really mean the same hash key.
Gem::Version is usable as a Hash key based on the internal version string:
class Gem::Version
def hash
@version.hash
end
def eql?(other)
self.class === other and @version == other.version
end
end
In Gem::Version, the internal version string looks like "1.3" or "1.3.0". In this implementation the two versions would belong to different hash keys.
Using #eql? instead of #== to determine if two keys are the same is a nice distinction since it allows you to have interesting behaviors (but I'm not sure they are useful). For Gem::Version, a version of "1.3" is equal to "1.3.0", but the occupy different slots in a Hash.
#intialize_copy
#initialize_copy is called during #dup and #clone to copy object-specific state beyond instance variables. The object being cloned from is passed to the new instance. #initialize_copy could be used for cleaning out a per-object cache:
def initialize_copy(other)
@cache = []
end
#exception
#exception is called on objects given to #raise to cast them to Exception objects. It must return a subclass of Exception. You can use this to turn arbitrary objects into exceptions, centralizing all your exception raising code:
class Result
class Error < RuntimeError; end
def initialize(json)
@result = JSON.parse json
end
def exception(message = nil)
Error.new "#{message} (#{@result['error']})"
end
def [](key)
@result[key]
end
end
r = Result.new open('http://example.com/api/blah').read
raise r if r['error']
Marshal
Ruby will marshal most objects automatically, but sometimes you want a custom format to ignore cached data that can be reconstructed or to reduce the size of the data you're saving out. There are two ways to do this, the older way is #_dump/::_load and the newer way is #marshal_dump/#marshal_load which takes priority. If you want to upgrade to the newer way you can leave ::_load to restore older marshaled objects.
Using the older way #_dump returns a String representation of the object (usually another Marshal string) and ::_load receives that String representation. Note that it's a class method, so ::_load is responsible for creating the object, which may be important in some instances.
Using the newer way, #marshal_dump returns an Object and #marshal_load receives that Object. The object is already allocated, but #initialize won't be called. The newer way can result in a smaller marshal dump size since it uses the existing symbol and object reference tables.
Gem::Specification uses #_dump/::load and is fairly complicated because I designed it to be backward and forward-compatible. This is a slightly-stripped-down version:
class Gem::Specification
CURRENT_SPECIFICATION_VERSION = 2
# number of fields per version
MARSHAL_FIELDS = { -1 => 16, 1 => 16, 2 => 16 }
def self._load(str)
array = Marshal.load str
spec = Gem::Specification.new
spec.instance_variable_set :@specification_version, array[1]
# validate object
current_version = CURRENT_SPECIFICATION_VERSION
field_count = if spec.specification_version > current_version then
spec.instance_variable_set :@specification_version,
current_version
MARSHAL_FIELDS[current_version]
else
MARSHAL_FIELDS[spec.specification_version]
end
if array.size < field_count then
raise TypeError, "invalid Gem::Specification format #{array.inspect}"
end
# restore object
spec.instance_variable_set :@rubygems_version, array[0]
# ...
spec.instance_variable_set :@platform, array[16].to_s
spec.instance_variable_set :@loaded, false
spec
end
def _dump(limit)
Marshal.dump [
@rubygems_version,
@specification_version,
# ...
@new_platform,
]
end
end
Gem::Version uses #marshal_dump/#marshal_load and ignores the internal instance variables, only dumping @version:
class Gem::Version
def marshal_dump
[@version]
end
def marshal_load(array)
self.version = array[0]
end
end
Pretty-print with PP
With a little work PP can give you easily readable output for your objects, even output that you can copy and paste back into a script. Primarily you'll use the PrettyPrint#group, PrettyPrint#text, PrettyPrint#breakable and PP#pp methods inside a #pretty_print method on your object. You can find documentation for these methods using ri.
Here's #pretty_print from Gem::Dependency and Gem::Requirement:
class Gem::Dependency
def pretty_print(q)
q.group 1, 'Gem::Dependency.new(', ')' do
q.pp @name
q.text ','
q.breakable
q.pp @version_requirements
q.text ','
q.breakable
q.pp @type
end
end
end
class Gem::Requirement
def pretty_print(q)
q.group 1, 'Gem::Requirement.new(', ')' do
q.pp as_list
end
end
def as_list
normalize
@requirements.map do |op, version| "#{op} #{version}" end
end
end
Together these make pretty, copy-pastable output:
require 'pp'
gem 'ParseTree'
pp Gem.loaded_specs["ParseTree"].dependencies
[Gem::Dependency.new("RubyInline",
Gem::Requirement.new([">= 3.7.0"]),
:runtime),
Gem::Dependency.new("sexp_processor",
Gem::Requirement.new([">= 3.0.0"]),
:runtime),
Gem::Dependency.new("hoe", Gem::Requirement.new([">= 1.8.0"]), :development)]
Working Style
Eric Hodel | Fri, 22 Aug 2008 00:42:25 GMT
In Rubinius we’ve been slowly working towards using a VM written in C++ instead of the current C VM we’ve got (called “shotgun”). Most of the mechanics of the VM are already complete, so over the past couple weeks we’ve been wiring up the primitives necessary for our ruby kernel (equivalent to Ruby’s core library) to run.
This primitive work is rather tedious, and until Friday I found it highly unenjoyable. I couldn’t perceive how much closer I was to our goal of actually running ruby code on the new VM.
On Friday I recalled that we can run a compiled ruby file against the VM, which gave me a way to shoot towards my goal. Over the past couple days during our Rubinius Team Meeting we’ve gotten up to loading all of our kernel bootstrap and platform, and now we’re just starting to do work in the core libraries.
Switching my working style drastically improved my enjoyment of a process that I found quite annoying. I wish I’d figured this out a couple weeks ago.

Articles