Robot Co-op Hardware
Eric Hodel | Thu, 16 Mar 2006 02:12:00 GMT
There’s been interest in the hardware that has driven the sites of The Robot Co-op over 2.5 million requests/day so here it is:
| Quantity | CPU | Memory | Disks | Functions |
|---|---|---|---|---|
| 4 | Dual 3GHz Xeon | 6GB | 70GB RAID 1 | Apache, FastCGI, MogileFS storage node, memcached, image serving |
| 1 | Dual 3GHz Xeon | 2GB | 70GB RAID 1 | Staging, mail, backend jobs |
| 1 | Dual Opteron 246 | 12GB | 5×73GB in RAID 5 | MySQL |
The four web servers are more fluke than planning, we don’t need the capacity they have just yet. We started with two webservers, a database server and a staging/mail/backend server, all dual 3GHz Xeons. We then added a third webserver and after that the Opteron MySQL box. The old database server was recently repurposed as a webserver.
Site traffic is currently spread across all four web boxes as each box runs all of our sites by a hardware load balancer of unknown manufacture. Eventually we’ll switch to running the 43 Things on a pair of machines and all other sites on the remaining machines.
Images are routed through a separate IP directly to WEBrick running a custom HTTPServlet that interacts with MogileFS to serve and resize images.
2.5 million
Eric Hodel | Tue, 07 Mar 2006 06:40:00 GMT
On Saturday March 4th the sites of The Robot Co-op handled 2,587,240 requests through Rails. That number includes redirects and error pages but excludes images, CSS and static javascript (we dynamically generate some JS for blog posting).
attr vs method vs define_method
Eric Hodel | Tue, 07 Mar 2006 00:59:00 GMT
There are four different ways to define a method in Ruby. The two most common is the def keyword and the Module#attr family of methods. The last two ways use Module#define_method, define_method with a block and define_method with a Method object.
Ruby's interpreter handles methods created with each of these constructs differently and you can really notice it when benchmarking them:
user system total real attr_writer 0.880000 0.010000 0.890000 ( 0.919265) regular method 1.370000 0.000000 1.370000 ( 1.485922) define_method w/method 2.470000 0.010000 2.480000 ( 2.636708) define_method w/block 3.030000 0.020000 3.050000 ( 3.268494)Read more...
Memcache-client benchmarks
Eric Hodel | Mon, 06 Mar 2006 22:15:00 GMT
Stefan Kaes writes about using memcached for Ruby on Rails session storage wherein he finds that memcache-client beats out all comers in his session store benchmarks:
memcache-client-1.0.3 provides much better performance: much faster than either the old implementation, pstore or ActiveRecordStore, but also faster than my optimized SQLSessionStore using MysqlSession.
String interpolation in ruby2c
Eric Hodel | Sat, 04 Mar 2006 07:27:00 GMT
The toolset zenspider and I built into ParseTree for rewriting Ruby’s AST allows us to make features work that otherwise would be very difficult by rewriting them into easier to implement features.
One feature that makes Ruby so clean is string interpolation but currently ruby2c doesn’t support it. In Ruby there are two types of String nodes, a plain :str node ([:str, "mystring"]) for regular strings and a :dstr node that contains all the parts of the dynamic string which the interpreter concatenates ([:dstr, "val: ", [:vcall, :val]]).
Implementing :dstr in a runtime would be too hard to be worthwhile so I used SexpProcessor to rewrite a :dstr into a series of calls to #<< and #to_s that append to the leading :str node.
Now a dynamic string like:
[:dstr,
"a",
[:vcall, :x],
[:str, "b"],
[:vcall, :y],
[:str, "c"]]Gets transformed into this ugly sexp:
s(:call,
s(:call,
s(:call,
s(:call,
s(:str, "a"),
:<<,
s(:arglist,
s(:call, s(:call, nil, :x, nil), :to_s, nil))),
:<<,
s(:arglist,
s(:call, s(:str, "b"), :to_s, nil))),
:<<,
s(:arglist,
s(:call, s(:call, nil, :y, nil), :to_s, nil))),
:<<,
s(:arglist,
s(:call, s(:str, "c"), :to_s, nil)))I don’t know about the inside-outness of it though, I may have to fix that.
Fast memcached on OS X
Eric Hodel | Fri, 03 Mar 2006 00:39:00 GMT
For those of you experiencing a painfully slow memcached on OS X, Noah M. Daniels wrote up instructions on making memcached fast for OS X:
Two simple changes:
First, in memcached.c (in the memcached source directory) add (anywhere above line 105, which reads #ifdef TCP_NOPUSH) the line:
#undef TCP_NOPUSH
I just added it on the line above the #ifdef line.
Rebuild memcached (just do a make && sudo make install, don’t need to re-run configure if you’ve already done it)
then, set the environment variable EVENT_NOKQUEUE to 1
in csh and derivatives: setenv EVENT_NOKQUEUE 1
in sh and derivatives (like bash): export EVENT_NOKQUEUE=1
then start memcached, and it should be fast (it certainly made a difference here)
Markov Chain
Eric Hodel | Sun, 26 Feb 2006 00:39:00 GMT
The back of my brain has been wanting a Markov chain based random-text generator for some time. I found some decently explanatory pseudo-code in the Generating Text section of the online Programming Pearls by Jon Bently.
What I really wanted was something that illustrated the concept well so I could tell what it was doing. The perl, python and ruby versions I found weren’t simple enough for me to see that. I also wanted to adjust the amount of state in the chain, but the only implementations that supported that were far too complicated to figure out.
Working from Jon Bently’s pseudo-code I ended up with the following implementation. Read more...
Bob's Favorite Scripting Language
Eric Hodel | Sun, 26 Feb 2006 00:39:00 GMT
Perl is so incomprehensible. I definitely like Ruby better than Perl as my favorite scripting language….
—Bob
MySQL is Dum
Eric Hodel | Sun, 26 Feb 2006 00:39:00 GMT
How hard is it to write data until the disk fills up? Creating more than 4GB of data isn’t that difficult, so shouldn’t the database be able to work with that much data out of the box?
How hard is it to write data to a table in-order when using INSERT INTO SELECT?
UPDATE: Seriously!
<samp>mysql> delete from [...] where timestamp < [...]; Query OK, 35759328 rows affected (57 min 13.77 sec)</samp>
Rubygems + ri
Eric Hodel | Thu, 23 Feb 2006 11:42:00 GMT
I’ve almost finished doing what has previously been claimed as impossible. I’ve cleanly integrated ri and Rubygems so that you can use ri to search your installed gems’ documentation.
The first part was simple, tell Rubygems to generate ri data for its gems. Rather than have Rubygems install a gem’s ri data mixed-in with the standard library’s data it installs it into a per-gem directory.
The unfinished part is getting a patch into Ruby that makes ri go looking in the gem ri data directories. That patch is in [ruby-core:7423]. Hopefully I can push it into 1.8 so it will be usable with Rubygems 0.9.
Articles