Segment7: Robot Co-op Software

Robot Co-op Software

drbrain@segment7.net (Eric Hodel) — Mon, 20 Mar 2006 11:28:00 -0800

I’ve seen a lot of comments asking for information on our software setup, so here it is. If you’d like more detail just ask, I’ll fill you in as best I can either in a comment or in a future post.

UPDATE: Added link to Wikipedia’s MySQL configuration

All our online machines run FreeBSD 6.0-RELEASE. We use Amanda for backups which gets rsynced off-site. Subversion is used for revision control of all our configuration data.

We have our own CVSup mirror and package build machine which builds packages for all our boxes to keep things in-sync and to decrease the load on FreeBSD’s CVSup mirrors. We also NFS-mount the src and ports directories to reduce wasted disk space. portaudit is used for vulnerability monitoring and portupgrade for performing package upgrades.

Critical processes are watched by dwatch. Clocks are synchronized with ntpd.

Webservers (x4)

The webservers all run PAE kernels with 6GB of ram. 2GB is allocated to memcached and the rest gets chewed up by Rails processes. Each of our sites gets its own set of processes and we have 25 total Rails processes running per host.

For pages our webservers are all Apache 1.3 with mod_fastcgi. Apache logs are rotated by cronolog and processed by AWStats (but I might switch to Visitors).

We’re using Rails (of course) to generate our pages. We use a library very similar to CachedModel to fetch ActiveRecord objects out of memcached. We don’t use any page or fragment caching.

For images our webservers are WEBrick using a Ruby MogileFS library I wrote (but have not yet released). Image resizing is performed by RMagick, occasionally on-the-fly.

We use the lang/ruby18-nopthreads port of Ruby because we experienced an incredible load increase with the default pthread version.

Each machine runs memcached with 2GB of cache (so be sure to set kern.maxdsiz and kern.maxssiz in /boot/loader.conf appropriately). We store sessions, ActiveRecords and random other expensive-to-compute data in memcache.

Each machine is a MogileFS tracker node and file store. All our images get stored in MogileFS. We use NFS mode due to problems with components of Perlbal on FreeBSD. (With the way we’ve implemented image serving and FreeBSD’s NFS implementation it works out fine.)

Our webservers all run sendmail set up as a smart host pointing to a machine running postfix. This keeps outbound mail sending fast and reduces the amount of maintenance.

Database

We have one database server, it runs MySQL 4.1.x and isn’t all that special. Bob tuned it based on Wikipedia’s MySQL configuration.

Miscellaneous

We have one machine that does miscellaneous jobs. It runs postfix for inbound and outbound mail, runs crons that update memcached, analyzes log files and a small handful of other unimportant things.

"Robot Co-op Software" by Greg

Wed, 22 Mar 2006 11:30:28 -0800

Why are you using WEBrick to serve your images (why not something else)? I’d be really interested in your feedback on that because I’ll be needing to do something similar.

"Robot Co-op Software" by Eric Hodel

Tue, 21 Mar 2006 10:43:29 -0800

I just added a new post about Version Control and Sysadmin.

"Robot Co-op Software" by http://blog.inlet-media.de

Tue, 21 Mar 2006 08:19:46 -0800

Can you tell me how you use Subversion for version control your configuration data? Did you commit the whole /etc directory?

"Robot Co-op Software" by James

Mon, 20 Mar 2006 18:46:47 -0800

http://www.freebsd.org/cgi/cvsweb.cgi/ports/graphics/GraphicsMagick/

"Robot Co-op Software" by Eric Hodel

Mon, 20 Mar 2006 17:15:17 -0800

John: So far there isn’t a port for GraphicsMagick in FreeBSD’s ports tree, so I haven’t even considered it. We very rarely generate extra images and keep the generated images forever so I haven’t worried about it.

"Robot Co-op Software" by john

Mon, 20 Mar 2006 17:03:08 -0800

FWIW: we use GraphicsMagick where I work, and it’s much faster and more stable than ImageMagick. it doesn’t have all of the features as IM, but has all that we care about.

also, when compiled with gcc4, it gives us almost a 20% performance boost with running ‘convert’ on large images.

you might not have a requirement to process images very fast, tho, since you mentioned that sometimes you’re doing them on-the-fly. :)

"Robot Co-op Software" by Eric Hodel

Mon, 20 Mar 2006 16:06:15 -0800

Cesar: I use FreeBSD for everything. I’ll adhere to Don’t Repeat Yourself and Ya’ Ain’t Gonna Need It over switching to Linux for the DB. It would just be too much work to need to upgrade the software twice.

"Robot Co-op Software" by Cesar

Mon, 20 Mar 2006 15:34:11 -0800

Eric, do you use FreeBSD for MySQL? Have you tested Linux vs FreeBSD for database performance?, I’ve heard that MySQL has performance problems on FreeBSD.

"Robot Co-op Software" by Eric Hodel

Mon, 20 Mar 2006 14:11:01 -0800

Todd: So far we’re using AWStats just for ballpark figures.

"Robot Co-op Software" by Eric Hodel

Mon, 20 Mar 2006 14:10:31 -0800

Alex: No, but I did just see MiniMagick. I’ll check both of them out.

"Robot Co-op Software" by Alex

Mon, 20 Mar 2006 13:51:41 -0800

Have you looked at imlib2-ruby at all? [http://www.pablotron.org/software/imlib2-ruby/doc/]. imlib2 is supposed to be an incredibly fast library. Perhaps you don’t do enough image work for it to be worth it, but if it’s less of a memory hog than rmagick then it might be very useful for any app that does a lot of image resizing or basic manipulation.

"Robot Co-op Software" by Todd Huss

Mon, 20 Mar 2006 13:33:27 -0800

In my experience AwStats tends to really suffer from crawler inflation by overcount page views, unique visitors, and grossly overcounting referrals. Most log based analyzers overcount real user traffic. It’s fine for ballpark stuff (which is good enough for most sites) but once a site gets big enough there’s really no substitute for a good client based analytics solution like Google Analytics, Omniture, or Fireclick when it comes to accuracy.

"Robot Co-op Software" by Eric Hodel

Mon, 20 Mar 2006 13:14:40 -0800

Chadwick: Yes, we use trac internally.

"Robot Co-op Software" by Chadwick

Mon, 20 Mar 2006 13:02:46 -0800

Thanks for posting this info Eric. Also of note, didn’t you once mention that you’re using Trac for internal PM?