Mechanics of Programming
drbrain |
I sat down with Elise Worthy at RailsConf to pair with her on a Rails application that used the Wunderground API to retrieve the weather for multiple cities. If you don't know Elise, she's a Hungry Academy student formerly of Seattle. You can read about how she built her app on the Hungry Academy Blog. Since Elise is new to programming, pairing with her made me think about how I work on small tasks when implementing a larger feature.
When I first sat down, Elise had a class to retrieve the weather for Washington D.C., a controller and a view to display it. Our task was to add a search box so you could show the weather for any ZIP code in addition to Washington D.C.
By implementing weather for a fixed city first Elise had followed the most difficult lesson for me to learn, only implement one idea, or story, at a time. When I try to add too much I can't fit all the details in my head. By making the weather work for one fixed city she had a solid base of work she could revert to to try again if she got stuck.
I mostly helped her with the question of "what should I do next?" Elise knew which steps she needed to perform but didn't know what the best order was. I've found that working on the easiest task first is best. By picking the easiest task you have time to think about the hard tasks in the back of your mind. By the time you get around to them you may have made them easier or you may have come up with an easier way to implement them.
In the course of doing the easiest thing next we switched back and forth between the API wrapper, the controller and the view. Each change we made was very small. For example, first add the search form, next extract the ZIP code from params, then replace showing Washington D.C. with the user's choice.
Each step only changed a couple lines that we verified in the browser so we didn't get ahead of ourselves. By first changing from a fixed city to an arbitrary city we advanced towards our goal without confusing ourselves with extra details. While this broke one feature of the app, it was obvious that it would be easy to restore in the future when we were ready to address it.
I also find it useful to add TODO comments when I know I need to change or fix something but don't want to do it yet. The TODO item may be a distraction at this time or something difficult that a future step will make easier. I find it easy to forget some necessary cleanup without them. We added a few when Elise would think of something she needed to change but I didn't want to get us distracted.
There was one other tip I had for Elise, as she was adding support for both Washington D.C. and the user's ZIP her first idea was to use two instance variables to provide the data to the view, one for Washington D.C. and the other for the user's city. She also asked if I thought this was the best way.
I told her that it is easier for me to think of handling just one item or handling many items, but not just two items. By having only these two categories of items to display it makes it easier to maintain your code by preventing duplication.
The best thing that came out of pairing with Elise was that I got to think about how I program at the lowest level. While I got to show Elise some tips I've learned over the years, she showed me the details of my own internal process that I barely think about anymore.
Forever-valid SSL certificates
drbrain |
If your library uses X509 cryptography, naturally your tests will need a key and valid certificate to test against. Creating a key and certificate frequently can quickly drain your entropy pool which slows down your tests.
Instead of creating the key for every test startup you can create it once and load it off the disk like this:
class TestMyGem < MyGem::TestCase
private_key = File.expand_path '../../../test/private_key.pem', __FILE__
private_key = File.read private_key
PRIVATE_KEY = OpenSSL::PKey::RSA.new private_key
# …Sure, you can rebuild the certificate every time with a validity time of an hour, but why not create a forever-valid certificate to go with it? No reasonable person would ever use a key shipped with an open project anyhow. Here's how to generate such a key and certificate:
require 'openssl'
# purposefully short key length
key = OpenSSL::PKey::RSA.new 512
# bogus subject and issuer
name = OpenSSL::X509::Name.parse 'CN=nobody/DC=example'
cert = OpenSSL::X509::Certificate.new
cert.subject = name
cert.issuer = name
cert.version = 2
cert.serial = 1
cert.not_before = Time.now
# lasts as long as X509 allows
cert.not_after = Time.gm 9999, 12, 31, 23, 59, 59
cert.public_key = key.public_key
cert.sign key, OpenSSL::Digest::SHA1.new
open 'private_key.pem', 'w' do |io| io.write key.to_pem end
open 'public_cert.pem', 'w' do |io| io.write cert.to_pem endYou can load this certificate just like the key as described above:
public_cert = File.expand_path '../../../test/public_cert.pem', __FILE__
public_cert = File.read public_cert
PUBLIC_CERT = OpenSSL::X509::Certificate.new public_certWhy are you updating your VERSION twice?
drbrain |
I was looking at a project on GitHub and noticed two commits, one after the other, each updating the version of the project in different files.
Then I looked through GitHub's popular forked repositories list and picked out the first several ruby projects and poked through them to see how many made multiple commits to update the version.
While my sample size was only seven or eight ruby libraries, only one updated the version in one place. (Looking through other gems in the past has shown me that most gem maintainers update their versions in multiple files, so pointing out the guilty would be pointless.)
I'm continually puzzled that Ruby projects don't apply the DRY principle to what I have found to be the most error-prone portion of being a library maintainer, releasing the library correctly.
Since Ryan and I put Hoe together there have been several other projects that claim various benefits over Hoe, but I haven't seen a single one that has the "single, unambiguous, authoritative representation" for such a basic piece of information as the version.
Looking through the commit messages of the popular project, one project was released with differing versions between the gemspec and the VERSION constant. Why use a release tool (or process) that allows this to happen? Hoe doesn't! It checks to make sure the VERSION you release matches up with the VERSION in your source (the single, unambiguous authoritative representation) so you can't do that!
Please, please, look at the tools you use to manage your project. Do they make you do extra work like editing the same data twice? Do they prevent you from shooting yourself in the foot?
If they don't make your life easier then fix them or ditch them. While I know that Hoe is the best tool available for developing, maintaining and releasing gems, I know that not everyone seems to agree (even though they're wrong ☺). If you don't like Hoe or don't want to switch to Hoe the least you could do is fix the tool you use to help you instead of hinder you.
My House Knows I'm Home
drbrain |
I bought a house last fall and upgraded it with Insteon switches (from MacHomeStore). I run Indigo on a Mac Mini to automate them all.
I wanted my house to know if it was or wasn't occupied so it could "do stuff" like turning off lights that were accidentally left on.
My girlfriend and I both have iPhones and while I could use the Find My iPhone feature, it isn't really accurate enough to know if I'm home and would probably drain the battery pretty quick if queried every five minutes.
I'm not sure if looking for the iPhones via ping or arp would work either, I've read the iPhone will shut off the WiFi adapter when it's not in use unless it's polling mail updates.
The iPhone has Bluetooth and it can be left on all the time without much battery loss. There's a handful of OS X apps that can execute a script when a bluetooth device comes in range but they only work for one device. Since my girlfriend also lives with me they won't work.
One of my coworkers got me to help him make a seemingly-abandoned Ruby Bluetooth library work on OS X using IOBluetooth.
The OS X side of the Bluetooth library started out with just the ability to scan for devices but I added remote name request, pairing, connection creation and signal strength.
It turns out that it's easy to see if a device is around once you have its address by asking for its name. There's no need to pair the device or create a connection (which will drain the battery faster).
Pairing and signal strength were implemented for the fun of walking around the house with the iPhone running a terminal session that monitored the signal strength and to see how well the Bluetooth signal works through my walls and floors.
Now that I could tell if the phones were nearby I needed to update Indigo with an occupied status. Indigo can monitor the occupied status and do something when it changes. The server is scriptable using AppleScript, so I used rubyosa to update Indigo with the presence of the phones. (The released rubyosa doesn't work on OS X 10.6, so I installed this version instead.)
All I needed now was the ability to run the script on a regular basis. I used Lingon to create a launchd agent that would run the script every five minutes. I told Indigo to turn off my interior lights when the house is no longer occupied, so if I leave the house (or turn the phones off) the lights automatically turn off.
I may also be able to do something more fancy like increase the poll interval once the house goes unoccupied to help it detect when I arrive home and do something like turn on the fireplace if it's cold, or turn on the entryway lights if it's dark.
Since I'm using Bluetooth my tools will work with any phone that can leave its radio on all the time. The phone doesn't need to be discoverable.
Hopefully I'll be able to release the Ruby Bluetooth library soon. I haven't been able to contact the original author and I'm not sure what the license is. I've replaced nearly all of the OS X portion of the library, but there's also extensive code for Linux and Windows.
Hopefully I'll also be able to convince Laurent to release an updated rubyosa that will work on OS X 10.6.
curb 0.7.3 performs as expected
drbrain |
Over loopback, curb runs about 2.4x the speed of net-http-persistent. This is the performance I expected curb to give. Having a pure-ruby library perform 2.4x worse than a C library makes net/http a fantastic library.
Remember, this benchmark and the previous one were designed to model a web-service type workload where many small requests are made. These benchmarks aren't designed to model downloading large files.
require 'rubygems'
require 'benchmark'
require 'net/http/persistent'
require 'curb'
# dd if=/dev/zero of=~/Sites/zeros-1k bs=1024 c=1
uri_1k = URI.parse 'http://localhost/~drbrain/zeros-1k'
uri_2k = URI.parse 'http://localhost/~drbrain/zeros-2k'
uri_10k = URI.parse 'http://localhost/~drbrain/zeros-10k'
uri = uri_2k
N = 100_000
Benchmark.bmbm do |bm|
bm.report 'Net::HTTP::Persistent' do
http_p = Net::HTTP::Persistent.new
N.times do
response = http_p.request uri
response.body
end
end
bm.report 'curb' do
curl = Curl::Easy.new
url = uri.to_s
N.times do
curl.url = url
curl.perform
curl.body_str
end
end
endRehearsal ---------------------------------------------------------
Net::HTTP::Persistent 54.070000 3.930000 58.000000 ( 71.651048)
curb 11.800000 4.510000 16.310000 ( 30.930453)
----------------------------------------------- total: 74.310000sec
user system total real
Net::HTTP::Persistent 54.130000 3.930000 58.060000 ( 72.430575)
curb 11.860000 4.540000 16.400000 ( 30.804222)
Example domains and IPs
drbrain |
I'm always forgetting this information. Here's a handy reference for example domain names, example IPv4 addresses and example IPv6 addresses.
Example Domain names
From RFC 2606:
- .test
- For testing DNS related code
- example
- example.com
- example.net
- example.org
- example.com
- For documentation or as examples
IPv4 Addresses
From RFC 3330:
192.0.2.0/24 is used for documentation and example code.
198.18.0.0/15 is used for benchmark tests of network interconnect devices. See RFC 2544
IPv6 Addresses
From RFC 3849:
2001:DB8::/32 is used for documentation and examples.
Service Discovery Protocols
drbrain |
I’ve worked on two different service discovery libraries for ruby, one for SSPD and one for mDNS, and I’d like to share the differences between them with you.
Both SSDP and mDNS are designed to be used with self-configuring networks and to automatically discover and connect to other devices. Both use multicast to advertise and search.
SSPD
The Simple Service Discovery Protocol is used primarily by UPnP for network devices to discover each other. UPnP is designed for consumer electronics and appliances that need to communicate with each other automatically.
The description of SSPD fits on a twelve of pages when written in (relatively) plain English (section 1 in the UPNP Device Architecture). The protocol itself looks much like HTTP and can be implemented largely using simple text processing. (Strangely, SOAP is the higher-level communication layer for UPnP.)
Unfortunately there’s no way for multiple devices to share the same SSPD socket without being designed to work with each other. Since UPnP is focused on simple devices, this fits well with their design goals. Most UPnP devices are routers, media servers, game consoles, televisions, etc. If you built your own router and media server on the same host it will probably be impossible for both devices to show up on the network, nor will you be able to run a media server and media viewer on the same host without the two being designed to work with each other.
mDNS
Multicast DNS is used primarily by Apple and (basically) has a DNS server on each host. Hosts speak the DNS protocol to each other over multicast and have a central daemon that handles communication on the network much like the libc DNS resolver.
Even though there’s plenty of DNS clients around for code reuse, implementing a DNS client form scratch is a large undertaking (my SSDP implementation is around 750 lines with documentation, ruby’s Resolv DNS resolver is around 2200 lines and doesn’t include a server or multicast code). The entirety of the DNS system is spread across several RFCs, not including the documentation for mDNS itself.
Fortunately mDNS libraries provide a daemon for handling network communication and a client library you can use to register and browse for devices (the dnssd gem uses Apple’s DNS Service Discovery API). mDNS is focused on computers rather than devices, so multiple services can all coexist on the same host.
Unfortunately, due to the complexity of mDNS, there’s no single API for applications to use. Apple’s DNS-SD works across multiple platforms including windows. For unix-like operating systems there is also avahi which has a limited dnssd compatibility shim.
vi bindings for irb on OS X
drbrain |
OS X uses editline(3) instead of readline(3) so ~/.inputrc doesn’t do anything for irb or other tools using readline via the editline wrapper.
Instead, use ~/.editrc:
bind -v
bind \\t rl_completeWhich gives you vi bindings in irb.
Update
Now with tab completion thanks to Curt Sampson from an ancient netbsd-users email!
Mail Hosting
drbrain |
I host my own mail using the following software:
- FreeBSD for my OS
- Postfix for SMTP
- OpenBSD’s spamd for greylisting
- amavisd for spam control (which uses SpamAssassin) and DKIM signing
- dovecot for IMAP and SASL
- procmail for mail filtering
OpenBSD’s spamd sits in front of Postfix and greylists and blacklists for me (it runs on all my MXs to prevent spammers from sneaking around). spamd runs via a firewall rule that redirects unknown connections to the spamd daemon during the greylist period then later whitelists them for direct connection to Postfix. I’ve also added a few spam-collecting addresses to the spamtrap list to help with automatic blacklisting.
Postfix directs email through amavisd for spam control which allows me to bounce spam (from amavisd’s Postfix README). I have amavisd configured to use Postgresql to enable its pen pals feature which lowers spam scores for frequent correspondents (from amavisd’s SQL README and Postgresql README).
Postfix hands mail off to procmail via my .forward file (not mailbox_command) which is just "|/usr/local/bin/procmail -tf-". (It seems that other values you see around, like setting IFS, are to work around bugs in ancient versions of sendmail.)
In procmail, I use dovecot’s deliver to keep its indexes updated. A sample from my .procmailrc:
DELIVER = /usr/local/libexec/dovecot/deliver
# ...
:0 w
* List-Id:.*<rubygems-developers.rubyforge.org>
| ${DELIVER} -m Lists/Ruby/Rubygems
# ...
# last rule, delivers to INBOX
:0 w
| ${DELIVER}My outbound mail goes through the submission port using dovecot for SASL authentication and gets filtered by amavisd for the pen pals feature and DKIM signing (set up per amavisd’s DKIM documentation).
I’ve set up DKIM and SPF records for my domain in order to be a good internet citizen. You can get an SPF record pretty quick from the SPF record wizard. I have SpamAssassin using DKIM verification for improved filtering.
To ensure delivery of my mail, one of my backup MXs is at my home for connection redundancy and the second is Ryan Davis’ primary MX (we have mutual backups). A backup MXs out of my control prevents me from losing mail by screwing up both machines under my control.
My IMAP clients include Apple’s Mail for day-to-day mail reading, IMAPCleanse for cleaning out my lists and flagging threads I should follow up on. Soon I’ll be adding an IMAP to RSS tool for mail with lots of unimportant stuff (like Amazon, bank transfers, etc.).
Binding#remove_local_variable
drbrain |
require 'rubygems'
require 'inline'
class Binding
inline do |builder|
builder.include '"node.h"'
builder.include '"env.h"'
##
# struct BLOCK isn't in any header, so include it here.
builder.prefix <<-C
struct BLOCK {
NODE *var;
NODE *body;
VALUE self;
struct FRAME frame;
struct SCOPE *scope;
VALUE klass;
NODE *cref;
int iter;
int vmode;
int flags;
int uniq;
struct RVarmap *dyna_vars;
VALUE orig_thread;
VALUE wrapper;
VALUE block_obj;
struct BLOCK *outer;
struct BLOCK *prev;
};
C
##
# :method: remove_local_variable
#
# Removes the local variable +name+ and replaces it with nil.
#
# Ordinarily if a local variable didn't exist it would raise
# NoMethodError, but currently after #remove_local_variable it doesn't, it
# just returns nil.
#
# In order to make this behave correctly, block->body would need to be
# walked, duping nodes (so as not to affect future invocations of this
# method) and replacing the LVAR nodes with VCALL nodes.
#
# That's just too much work for 17:15, though. Maybe tomorrow.
builder.c <<-C
VALUE remove_local_variable(VALUE name) {
struct BLOCK *block;
struct SCOPE *scope;
ID name_id;
ID *local_table;
int i, n;
VALUE entry;
name_id = SYM2ID(name);
Data_Get_Struct(self, struct BLOCK, block);
scope = block->scope;
local_table = scope->local_tbl;
if (local_table) {
n = *local_table++;
for (i = 2; i < n; i++) { /* skip $_ and $~ */
if (!rb_is_local_id(local_table[i])) continue; /* skip flip states */
if (local_table[i] == name_id) {
entry = scope->local_vars[i];
local_table[i] = (ID)NULL;
scope->local_vars[i] = Qnil;
return entry;
}
}
}
return Qnil;
}
C
end
end
a = :my_value
p :lvar_a => a
b = binding
p :remove_local_variable_says => b.remove_local_variable(:a)
p :lvar_a => a # TODO raise exception
