Segment7: Version Control and Sysadmin

Version Control and Sysadmin

drbrain@segment7.net (Eric Hodel) — Tue, 21 Mar 2006 10:19:00 -0800

Every part of the system configuration you change belongs under version control (with a few exceptions). If you’re going to be making changes to your configuration you might do something wrong and need to roll back. You might wonder why or who made a change in the future. Version control will perform CYA duties for you.

There are a few things you probably don’t want under version control. /etc/master.passwd shouldn’t be flying across the wire (and Kerberization or similar works much better for distributing passwords). Sudo will get mad if you go and touch /usr/local/etc/sudoers inappropriately, especially if it has the wrong owners.

Configuration files in /etc, /usr/local/etc, /boot, custom rc.d and periodic scripts, anything you’re going to change, add or even break needs to be under version control. (I haven’t figured out a good way of putting crontabs under version control, ideas?)

For The Robot Co-op, each machine’s configuration is in its own branch in a subversion repository to allow care-free copying of changes between machines. A change to the httpd.conf on one machine is a commit and a couple of merges away from being accurately changed on all the machines. No typos from multiple manual changes.

"Version Control and Sysadmin" by Doug

Wed, 22 Mar 2006 15:05:19 -0800

Eric, I’m looking at /etc/crontab on a FreeBSD-6.0 Release box, and the sixth field in the line for each job is the user name of the account that the command should run under. Will this not let you consolidate your cron schedules in that one file for revision control?

"Version Control and Sysadmin" by Eric Hodel

Wed, 22 Mar 2006 10:01:40 -0800

cfengine sounds like something I really need to use. It would be great if you’d write up the version control integration.

"Version Control and Sysadmin" by Scott Woods

Wed, 22 Mar 2006 07:05:01 -0800

While it’s a bit heavier-handed, if you have the desire to do this on more than a couple machines, you can combine version control with cfengine to get a very organized and powerful system (and scratch the common crontab issue itch).

You can keep the master set of files under version control, and cfengine handles the distribution and comparisons on various machines. When a file goes out of sync and needs to be updated, you get notified.

cfengine only uses the files that are committed to the repository, so you’re free to test and experiment with your local copy.

If a set of machines have the same file in common, they can all just belong to the same class, and you only need one copy the file in version control.

Furthermore, you can define actions to perform after a certain file has been updated, such as “crontab /var/spool/cron/root”, or “service httpd reload”.

The standard cfengine setup doesn’t typically integrate the version control, but I’d be happy to put a page up somewhere describing the method that we use.

"Version Control and Sysadmin" by Eric Hodel

Tue, 21 Mar 2006 21:00:13 -0800

I like to use version control as a log system rather than a backup system so meaningful commit messages are more important. I don’t want something automatically checking in to subversion, I find that too scary and not useful enough for tracing changes to meaningful messages. (Who added this job 3 weeks ago?)

Wrapping up crontab with a forced commit would be too annoying, especially if you get it wrong and have to go back in and change it.

"Version Control and Sysadmin" by Larry Wright

Tue, 21 Mar 2006 20:01:30 -0800

You can add a cron job to the users whose crontabs you want backed up to do the ‘crontab -l | >> jobs.list’ and then automatically check it in if ‘svn diff’ shows a difference.

I’m not entirely sure what you mean by “automatic”. Unless you’re scheduling your svn commits (in the manner I just described), then it still involves a manual step.

I do agree with your advice though. This is one of the big advantages of having plain-text configuration files. Try doing this with the Windows registry….

"Version Control and Sysadmin" by Eric Hodel

Tue, 21 Mar 2006 19:16:59 -0800

Larry: Right, but its not automatic. I’d like something that won’t involve extra work.

"Version Control and Sysadmin" by Larry Wright

Tue, 21 Mar 2006 19:00:23 -0800

You can always just ‘crontab -l >> jobs.list’ and version that file (at least under linux, it’s been a while since I’ve used a BSD).

"Version Control and Sysadmin" by Eric Hodel

Tue, 21 Mar 2006 13:48:38 -0800

FreeBSD doesn’t have a cron.d equivalent. There is a /etc/crontab, but that gets run as root and there’s several jobs that I’d rather have run as a non-root user.

Fortunately /var gets backed up by amanda so I don’t have to worry about a total loss of the crontabs.

"Version Control and Sysadmin" by Adam Keys

Tue, 21 Mar 2006 12:40:43 -0800

A few years ago I admin’d some systems (Redhat Enterprise I think) that had an /etc/cron.d/ directory. Any file you dropped in there was treated as a crontab. So I was able to put cronjobs for separate tasks in separate files. Maybe that would make it easier to version control cronjobs?