Segment7: Robot Co-op Hardware

Robot Co-op Hardware

drbrain@segment7.net (Eric Hodel) — Wed, 15 Mar 2006 18:12:00 -0800

There’s been interest in the hardware that has driven the sites of The Robot Co-op over 2.5 million requests/day so here it is:

Quantity	CPU	Memory	Disks	Functions
4	Dual 3GHz Xeon	6GB	70GB RAID 1	Apache, FastCGI, MogileFS storage node, memcached, image serving
1	Dual 3GHz Xeon	2GB	70GB RAID 1	Staging, mail, backend jobs
1	Dual Opteron 246	12GB	5×73GB in RAID 5	MySQL

The four web servers are more fluke than planning, we don’t need the capacity they have just yet. We started with two webservers, a database server and a staging/mail/backend server, all dual 3GHz Xeons. We then added a third webserver and after that the Opteron MySQL box. The old database server was recently repurposed as a webserver.

Site traffic is currently spread across all four web boxes as each box runs all of our sites by a hardware load balancer of unknown manufacture. Eventually we’ll switch to running the 43 Things on a pair of machines and all other sites on the remaining machines.

Images are routed through a separate IP directly to WEBrick running a custom HTTPServlet that interacts with MogileFS to serve and resize images.

"Robot Co-op Hardware" by Michael Shigorin

Tue, 04 Apr 2006 10:28:28 -0700

> Most importantly, lighttpd doesn’t > have a mod_deflate equivalent It’s also a weird code, every person on our team who was first admired with it later regretted it for various reasons (not handling ”+” in filenames or whatever architectural problems).

No one has got off nginx as a frontend/static server though, it’s very robust. Definitely worth looking at.

"Robot Co-op Hardware" by Doug D

Mon, 27 Mar 2006 15:02:57 -0800

The load balancer is most likely a Cisco CSS series device. Cisco is our preferred vendor for almost all of our network devices. (I work for Rackspace =D)

"Robot Co-op Hardware" by djwm

Fri, 17 Mar 2006 18:22:44 -0800

regarding system load. it a very unrepresentive metric. you can have a box with a load of 500 which will be running almost like it has no work to do at all.

A good example is a forking ftp server. The ftp processes are lightweight and just do i/o in little spurts. High load, but FreeBSD just kicks ass for this…

or a box with a flat load of 1.00 with a single process which sits on cpu all the time and is running at 100% of the systems capacity. any kqueueified single process daemon, such as thttpd.

i find mysql is very good at operating under load on freebsd6. I maintain a bacula backup server for our business and the some of tables used for the backup db have a lot of rows (10million+). I can run 40 concurrent backup jobs into this db – AND run a table optimise query at the same time and the box sits about 60% CPU, disc on 20MB/s.

Did I mention, the box is a 2GHz P4 with 2GB RAM and a ATA100 drive? :)

... and its my KDE desktop too.

FreeBSD++

"Robot Co-op Hardware" by Chuq

Fri, 17 Mar 2006 12:04:02 -0800

Eric, interesting that you guys don’t have a backup database server. I’m assuming that you guys are using a battery backed RAID controller at very least?

"Robot Co-op Hardware" by Paul

Fri, 17 Mar 2006 12:00:16 -0800

Curious are you using Apache2 + mod_fcgi or mod_fastcgi? Worker or prefork? And I doubt it, but any PHP running around on those servers?

Great to read though.

"Robot Co-op Hardware" by Todd Huss

Fri, 17 Mar 2006 09:04:22 -0800

Great post, thanks for the info! It’s interesting to compare setups, I’ve written about our setup here:

http://gabrito.com/post/website-hardware-and-our-setup

"Robot Co-op Hardware" by Zack Chandler

Thu, 16 Mar 2006 22:56:02 -0800

I use Rackspace also – great company although pricey. I didn’t know they did freeBSD though.

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 20:19:30 -0800

Keith P: I’ll post a rough outline of our software configuration after I’m feeling well again.

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 19:57:09 -0800

Keith V: No, we only use it for an image store.

Sessions are stored in memcached.

"Robot Co-op Hardware" by Keith Pitty

Thu, 16 Mar 2006 18:58:57 -0800

How many FastCGI server processes per web server?

"Robot Co-op Hardware" by Keith Veleba

Thu, 16 Mar 2006 16:37:47 -0800

Do you replicate session information with MogileFS?

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 15:54:20 -0800

Tom: We let Rackspace handle all the hardware and the console doesn’t say what it is. I think it is a Cisco, but I can’t say for certain. (Note: F5 is in my backyard, and I practice Jujutsu with a guy who works there.)

"Robot Co-op Hardware" by John Röthlisberger

Thu, 16 Mar 2006 15:16:29 -0800

Thanks Eric. It is fantastic to see these real-life figures as they are so much more useful than theoreticals.

"Robot Co-op Hardware" by Tom

Thu, 16 Mar 2006 14:21:45 -0800

Why is the manufacturer of the hardware load balancer unknown? (Disclaimer: I am an engineer at a company that makes hardware load balancers)

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 09:33:04 -0800

jtoy: I use NFS mode with a WEBrick server to handle the trackers’ usage checks. Fortunately FreeBSD has a decent NFS implementation. IO::AIO and FreeBSD hate each other for some unknown reason so I can’t use Perlbal.

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 09:27:09 -0800

Brandt: I tested HTT performance and found about a 20% improvement.

"Robot Co-op Hardware" by jtoy

Thu, 16 Mar 2006 07:51:31 -0800

Hi, did you need to modigy MogileFS to get it to work on FreeBSD? I thought that it is currently only working on linux.

Thanks for the information.

"Robot Co-op Hardware" by john

Thu, 16 Mar 2006 06:28:21 -0800

just fyi: in the case of the database box, loads can still get quite high without affecting performance, mostly because 64bit boxes are monsters when it comes to context switching.

At various times, we’ve had here many (>10, up to 25) 64bit (16GB) MySQL slaves with loads >>5 that keep with with replication fine, and still serve SELECT traffic for the site.

"Robot Co-op Hardware" by Brandt

Thu, 16 Mar 2006 05:36:33 -0800

eric: i don’t think HTT gives you the extra capacity you think it does. for example (this is what i did), use ab (or similar load-generating tool) to generate a steady load of 2.0 on your box, once with HTT enabled and once with it turned off. i believe you’ll find (as i did) that you get roughly the same number of requests/second served regardless of HTT setting. that’s because you don’t actually double the number of compute cores when you enable HTT, it just looks that way to the OS.

so, if you want to keep your servers running at 50% of max capacity, you really want to be adding nodes when your load average hits 1.0, because a load of 2.0 is really 100% capacity.

"Robot Co-op Hardware" by john

Thu, 16 Mar 2006 04:42:37 -0800

I apologize for the questions, and if it works for you, rock on. :) And since the DB can all fit into RAM for you, disk I/O won’t be an issue of course.

My experience has been with the db being much bigger than RAM, and very IO-bound to disk, and so when I see RAID5 I get twitchy flashbacks of some very long and tough nights. :)

Thanks for humoring my questions.

"Robot Co-op Hardware" by Adam

Thu, 16 Mar 2006 01:29:50 -0800

FYI,

There is a patch to add mod_deflate to Lighttpd. Its listed as a patch for 1.4.10. Might want to give it a shot just for kicks.

"Robot Co-op Hardware" by Eric Hodel

Thu, 16 Mar 2006 00:39:09 -0800

john: All FreeBSD 6 on UFS2 as I prefer my safety belts. The Opteron is in amd64 mode. The database sits all in RAM so the machine is nearly idle.

I’m not into premature optimization and simplicity suggests spreading writes across as many spindles as possible. If it becomes a problem I’ll address it then.

Bob tuned innodb based on wikipedia’s configuration.

"Robot Co-op Hardware" by Observer

Wed, 15 Mar 2006 23:40:23 -0800

What’s with all the “Why this and not that?” questions?

If it works for them, it is a perfectly valid setup.

"Robot Co-op Hardware" by john

Wed, 15 Mar 2006 22:30:20 -0800

what OS and filesystem for the Opteron, how much you giving to Innodb’s bufferpool, and what does iostat look like ? I assume since you’ve got the monster hunk of RAM that you’re doing some decent qps on it.

question: Are you disk I/O bound at all ?

In my experience, RAID5 can give some really awful performance, and RAID10 is about the best you’ll get. I’d be interested to hear the actual numbers.

"Robot Co-op Hardware" by Eric Hodel

Wed, 15 Mar 2006 21:14:12 -0800

Most importantly, lighttpd doesn’t have a mod_deflate equivalent (only compression of static content). Compressed pages save lots of money.

I also hate extra software so I’d have to either deal with the pain of extra software to worry about or reconfigure all our internal tools that use apache to use lighttpd.