Working Style
drbrain |
In Rubinius we’ve been slowly working towards using a VM written in C++ instead of the current C VM we’ve got (called “shotgun”). Most of the mechanics of the VM are already complete, so over the past couple weeks we’ve been wiring up the primitives necessary for our ruby kernel (equivalent to Ruby’s core library) to run.
This primitive work is rather tedious, and until Friday I found it highly unenjoyable. I couldn’t perceive how much closer I was to our goal of actually running ruby code on the new VM.
On Friday I recalled that we can run a compiled ruby file against the VM, which gave me a way to shoot towards my goal. Over the past couple days during our Rubinius Team Meeting we’ve gotten up to loading all of our kernel bootstrap and platform, and now we’re just starting to do work in the core libraries.
Switching my working style drastically improved my enjoyment of a process that I found quite annoying. I wish I’d figured this out a couple weeks ago.
Return of Syck
drbrain |
From #rubinius, the royal struggle between syck and RbYAML.
Defiler: re rbyaml also
Defiler: Why aren't we using syck though?
Defiler: rbyaml is totally not compliant with what MRI is expecting to
have happen
drbrain: syck period, or the syck extension from MRI?
Defiler: "Why are we using rbyaml" is what I should have said, I guess
drbrain: Back in Ye Olden Times of Late Last December
drbrain: We had a wrapper for the syck library that was not MRI's syck
extension
drbrain: For in Ye Olden Days, Fair Subtend was not so Fair
drbrain: Thus, a Newcomer was Raised to Ye Throne of YAML
drbrain: RbYAML
drbrain: Now, with the waxing power of Subtend, the MRI syck extension
may reclaim its Rightful Crown
.au Sprint
drbrain |
I spent Saturday and Sunday at the Sydney Rubinius sprint, and it was quite productive. Not only were local beers sampled, but we also got some good work done.
Lincoln Stoll helped me shake the last bugs out of RubyGems, so we integrated it into Rubinius. We decided to make it a subcommand rbx gem like rbx compile or rbx describe. There are still a few things broken in RubyGems, namely installing gems with extensions because mkmf.rb doesn’t work in Rubinius.
Lincoln also pointed out and gave me patches for a few backwards-compatibility problems with RDoc, so now both RubyGems and RDoc work on Rubinius.
Upon my return next week I’ll be cooking up a release of RubyGems with a handful of bug fixes and persistent connection support for RubyGems’ HTTP fetching which will make those incremental updates quite a bit faster.
RubyGems in Rubinius
drbrain |
Last night I checked an improved version of Zlib into Rubinius, and now gems can be installed:
$ ~/Links/RUBINIUS/shotgun/rubinius \
-Ilib bin/gem install rake --no-ri --no-rdoc -i ~/tmp/rbxgems
Successfully installed rake-0.8.1
1 gem installed
$ GEM_HOME=~/tmp/rbxgems ~/Links/RUBINIUS/shotgun/rubinius \
-Ilib ~/tmp/rbxgems/bin/rake --help
rake [-f rakefile] {options} targets...
Options are ...
[...]
Building the ri and RDoc information doesn't work correctly yet on Rubinius, but Rubinius is now using RDoc to generate some documentation, particularly the VM opcode information (run `rake doc:vm` in a Rubinius checkout).
Next I'm going to work on the RDoc/ri end of things some more, including improving or importing more documentation into Rubinius, and make a release of my recent RDoc changes.
Rubinius' Foreign Function Interface
drbrain |
I really, really, really love Rubinius’ Foreign Function Interface (FFI) since it allows you to replace C code with Ruby code. Earlier today I wrote Socket::getaddrinfo in C for Rubinius, and just now I finished a rewrite using FFI and Ruby. I’ve commented the code for clarity.
def self.getaddrinfo(host, service, family = nil, socktype = nil,
protocol = nil, flags = nil)
service = service.to_s
# MemoryPointer.new is kind-of like malloc(3), but understands what's inside
hints_p = MemoryPointer.new Socket::Foreign::AddrInfo.size
# Socket::Foreign::AddrInfo is a struct addrinfo wrapper with friendly accessors
hints = Socket::Foreign::AddrInfo.new hints_p
hints[:ai_family] = family || 0
hints[:ai_socktype] = socktype || 0
hints[:ai_protocol] = protocol || 0
hints[:ai_flags] = flags || 0
# getaddrinfo(3) asks for a struct addrinfo **.
# This creates a pointer to a pointer
res_p = MemoryPointer.new :pointer
# call out to C
err = Socket::Foreign.getaddrinfo host, service, hints_p, res_p
# check for errors
raise SocketError, Socket::Foreign.gai_strerror(err) unless err == 0
# now we read out the pointer that getaddrinfo() passed us, and cast it
# to a struct addrinfo *
res = Socket::Foreign::AddrInfo.new res_p.read_pointer
addrinfos = []
loop do
addrinfo = []
# Extract data
addrinfo << Socket::Constants::AF_TO_FAMILY[res[:ai_family]]
ai_sockaddr = res[:ai_addr].read_string res[:ai_addrlen]
sockaddr = Socket::Foreign::unpack_sa_ip ai_sockaddr, true
addrinfo << sockaddr.pop # port
addrinfo.concat sockaddr # hosts
addrinfo << res[:ai_family]
addrinfo << res[:ai_socktype]
addrinfo << res[:ai_protocol]
addrinfos << addrinfo
# struct addrinfo is a linked list, so if we've hit the end, stop
break unless res[:ai_next]
# otherwise, down the linked-list
res = Socket::Foreign::AddrInfo.new res[:ai_next]
end
return addrinfos
ensure
# like a C code, we have to free our MemoryPointer objects
hints_p.free if hints_p
if res_p then
# also, we have to do any C-side cleanup
Socket::Foreign.freeaddrinfo res_p.read_pointer
res_p.free
end
endgetaddrinfo(3), freeaddrinfo(3) and gai_strerror(3) are wrapped up by FFI like this:
attach_function "gai_strerror", :gai_strerror, [:int], :string
attach_function "getaddrinfo", :getaddrinfo,
[:string, :string, :pointer, :pointer], :int
attach_function "freeaddrinfo", :freeaddrinfo, [:pointer], :voidThe first argument is the C function name, the second is the Ruby name, the third is the input arguments, and the fourth is the return type. Currently, FFI can only wrap up C functions with six or fewer args.
The AddrInfo struct is wrapped up like this:
class AddrInfo < FFI::Struct
config("rbx.platform.addrinfo", :ai_flags, :ai_family, :ai_socktype,
:ai_protocol, :ai_addrlen, :ai_addr, :ai_canonname, :ai_next)
endThe config method pulls pre-generated struct information out of a Rubinius config file and hooks up accessors to each of the struct’s fields. The accessors know which offset into the struct the data lives at and what type to convert data from and to when working with the struct. The information is collected at Rubinius build time by a small bit of C code.
I still have some confusion between passing an FFI::Struct like Socket::Foreign::AddrInfo vs. passing a MemoryPointer instance (which is what an FFI wrapped function understands) to an FFI-wrapped function, so we’re going to clean up that part of the API to make it more natural. Instead you’ll be able to initialize an FFI::Struct directly and pass it to the FFI-wrapped function. This will make the code quite a bit cleaner.
RubyGems on Rubinius
drbrain |
Friday I passed an important milestone for RubyGems on Rubinius, test/test_gem.rb ran to completion! Here are the dots to prove it:
$ ~/Links/RUBINIUS/shotgun/rubinius -Ilib:test test/test_gem.rb Loaded suite test/test_gem Started ................................. Finished in 18.667198 seconds. 33 tests, 56 assertions, 0 failures, 0 errors
These tests include building, installing, and activating a Gem, so it should be possible now to use RubyGems in Rubinius. Unfortunately, the `gem` command doesn't work yet. That will be among my next tasks.
The final holdup for these RubyGems tests was a Dir.glob that handled {}. Finishing this off should also allow rake to work, according Evan's recent work on rake.
A full run of all RubyGems tests reports:
100 tests, 221 assertions, 6 failures, 13 errors
A full run with MRI is 533 tests, 1816 assertions. The 433 missing tests are due to a missing feature of Module#module_function in Rubinius that prevents the remaining test files from loading.
If you'd like to duplicate my results, here's how:
- Checkout Rubinius
- Build Rubinius by running rake
- Checkout RubyGems' trunk
- Run /path/to/rubinius/shotgun/rubinius -Ilib:test test/test_gem.rb
"What should I work on in Rubinius?"
drbrain |
Occasionally, people drop into the Rubinius IRC channel and ask what they should work on. The answer in the past has been, “write some specs” which can be some terribly boring work and is not very motivational. Fortunately, I have a solution for that.
I think the best way to get involved in Rubinius is to take your favorite Ruby project and try to run it on Rubinius. Whatever problem you run into first, fix that (including writing specs, of course).
My work on running RubyGems in Rubinius has involved running the RubyGems tests, seeing where they fail, then moving over to Rubinius and fixing whatever failure I’m having. This way I’m highly motivated because my end goal, making RubyGems work, is easily in sight. (Of course, when RubyGems is working, I’ll need a new goal, maybe RDoc).
Rubinius
drbrain |
I’m down in San Francisco this week working on Rubinius with Evan, Ryan, Wilson and Brian, which has been my Top Secret job at Engine Yard for a little over a month. Also joining us has been Josh, Kevin and Nathan (among others) have stopped by to hang out and hack too.
Primarily I’ve been working on getting RubyGems working on Rubinius, along with build system and other cleanups. I’ve started by trying to get just test_gem running, and I’ve become hung up waiting for Kernel#eval and Kernel#binding, which will be finished with the compiler2 work.
