Version Control and Sysadmin

Eric Hodel | Tue, 21 Mar 2006 18:19:00 GMT

Posted in

Every part of the system configuration you change belongs under version control (with a few exceptions). If you’re going to be making changes to your configuration you might do something wrong and need to roll back. You might wonder why or who made a change in the future. Version control will perform CYA duties for you.

There are a few things you probably don’t want under version control. /etc/master.passwd shouldn’t be flying across the wire (and Kerberization or similar works much better for distributing passwords). Sudo will get mad if you go and touch /usr/local/etc/sudoers inappropriately, especially if it has the wrong owners.

Configuration files in /etc, /usr/local/etc, /boot, custom rc.d and periodic scripts, anything you’re going to change, add or even break needs to be under version control. (I haven’t figured out a good way of putting crontabs under version control, ideas?)

For The Robot Co-op, each machine’s configuration is in its own branch in a subversion repository to allow care-free copying of changes between machines. A change to the httpd.conf on one machine is a commit and a couple of merges away from being accurately changed on all the machines. No typos from multiple manual changes.

9 comments

Comments RSS FEED

A few years ago I admin’d some systems (Redhat Enterprise I think) that had an /etc/cron.d/ directory. Any file you dropped in there was treated as a crontab. So I was able to put cronjobs for separate tasks in separate files. Maybe that would make it easier to version control cronjobs?

Adam Keys said about 2 hours later

FreeBSD doesn’t have a cron.d equivalent. There is a /etc/crontab, but that gets run as root and there’s several jobs that I’d rather have run as a non-root user.

Fortunately /var gets backed up by amanda so I don’t have to worry about a total loss of the crontabs.

Eric Hodel said about 4 hours later

You can always just ‘crontab -l >> jobs.list’ and version that file (at least under linux, it’s been a while since I’ve used a BSD).

Larry Wright said about 9 hours later

Larry: Right, but its not automatic. I’d like something that won’t involve extra work.

Eric Hodel said about 9 hours later

You can add a cron job to the users whose crontabs you want backed up to do the ‘crontab -l | >> jobs.list’ and then automatically check it in if ‘svn diff’ shows a difference.

I’m not entirely sure what you mean by “automatic”. Unless you’re scheduling your svn commits (in the manner I just described), then it still involves a manual step.

I do agree with your advice though. This is one of the big advantages of having plain-text configuration files. Try doing this with the Windows registry….

Larry Wright said about 10 hours later

I like to use version control as a log system rather than a backup system so meaningful commit messages are more important. I don’t want something automatically checking in to subversion, I find that too scary and not useful enough for tracing changes to meaningful messages. (Who added this job 3 weeks ago?)

Wrapping up crontab with a forced commit would be too annoying, especially if you get it wrong and have to go back in and change it.

Eric Hodel said about 11 hours later

While it’s a bit heavier-handed, if you have the desire to do this on more than a couple machines, you can combine version control with cfengine to get a very organized and powerful system (and scratch the common crontab issue itch).

You can keep the master set of files under version control, and cfengine handles the distribution and comparisons on various machines. When a file goes out of sync and needs to be updated, you get notified.

cfengine only uses the files that are committed to the repository, so you’re free to test and experiment with your local copy.

If a set of machines have the same file in common, they can all just belong to the same class, and you only need one copy the file in version control.

Furthermore, you can define actions to perform after a certain file has been updated, such as “crontab /var/spool/cron/root”, or “service httpd reload”.

The standard cfengine setup doesn’t typically integrate the version control, but I’d be happy to put a page up somewhere describing the method that we use.

Scott Woods said about 21 hours later

cfengine sounds like something I really need to use. It would be great if you’d write up the version control integration.

Eric Hodel said about 24 hours later

Eric, I’m looking at /etc/crontab on a FreeBSD-6.0 Release box, and the sixth field in the line for each job is the user name of the account that the command should run under. Will this not let you consolidate your cron schedules in that one file for revision control?

Doug said 1 day later

Comments are disabled