Introduction to Rdist

"Using rdist doesn't make sense unless you are dealing with a sufficiently large number of machines---let's say at least two." (Markus Moster)

Introduction

In a Nutshell

Synchronizing files and file hierarchies across multiple machines is an ever-recurring task in almost any reasonably-sized computer network environment. Since the arrival of BSD 4.3 a widely ignored tool called rdist is available to deal with this task. At least among machines running Un*x-style operating systems, that is.

Simple Real World Examples

What does rdist provide? The basic functionality is best demonstrated with some simplified real-world examples.

First consider a WAN connected through low-bandwidth (64kbit/s) links. At about 40 sites there exist general-purpose servers. These servers provide some software repository to say twenty clients each. Whenever a user feels like it s/he may install software from that repository on his/her desktop client. The software in the repository is frequently updated and contains about 500 MB worth of software. Using rdist you simply define one server as the "master" or "reference" machine. Whenever you've modified the repository on that machine you run rdist, for large modifications preferably overnight, and it updates the remote servers without further sysadmin intervention. Since rdist will only update files that have changed this provides a very bandwidth and time preserving way to deal with this distribution issue. If you ever fear that for some reason a repository got out of sync you re-run rdist. Unless things actually have changed this will have little impact on the servers or the bandwidth available in between. Of course the same approach works for the document roots of multiple web servers behind a load balancer, too.

Consider the same servers again. Most of their configuration is the same on all machines, or at least should be. If you set them up the same and then use rdist to distribute all configuration files that shouldn't be adjusted individually you won't have to update 40 /etc/profile's by hand (eventually getting things out of sync anyway) but rather do that only once and then run rdist. If you want to test new configurations you do so on only one machine. If things work out you run rdist from that machine, otherwise you can easily do a rollback by running rdist from one of the remaining functional machines. This has proven to be particularly useful in cluster environments where cluster members must be kept identically configured and no excuses. Guess what Markus Moster (see the quote above) is habitually working on.

If you are responsible for quality control in a software development team you may want to test the installation packages you get from the development team in a clearly defined test environment. Of course you might want to reinstall those machines either from backup or through an install server (like a Sun[TM] Solaris[TM] JumpStart[TM]). You might however want to find out what system files the package installation has messed up. If you define a reference file system somewhere rdist lets you do all this with a reasonable amount of trouble. And imagine the educational value of test machines that are always in a clearly defined state every morning.

In a security-sensitive environment where you need to maintain a consistent user base on multiple machines but can afford neither the security issues of NIS nor the hassles and Solaris-onliness of NIS+ you can easily set up one machine as the "master" passwd server and distribute whatever /etc/passwd, /etc/shadow, /etc/groups and such it has through cron to all other machines. New user accounts get created on that machine or people change their password on that machine and maybe a minute later the new account or password is active on all machines.

How `rdist` Works

How does rdist provide this functionality? Well, assume that you have a machine master that wants to copy some files to a list of machines slave_1 to slave_n. Here's what happens:

You invoke rdist, possibly with some options.
If you didn't specify directly what to copy where the first thing rdist does is to read a Distfile. This file tells rdist what to copy where.
Now rdist starts a "daemon" program called rdistd on all slaves. By default it does so using rsh but anybody working in a remotely security-sensitive environment will tell it to use the Secure Shell (ssh) instead.
Once the connection between the master rdist and slaves rdistds is up and running the master starts to stat(2) all files and file hierarchies specified in the distfile. It asks the slaves to do so as well. Whenever the stat(2) results prove different the file is sent from the master to the slave where it is installed.
Once the master is through it tells the slave rdistd to terminate which will also shut down the connection between the machines.

This general behaviour can be modified in various ways. Among others you can

do a "dry run", i.e. only list what would be done if you hadn't told rdist to hold back,
force a full comparison of file contents instead of relying on stat(2) only,
skip files that appear to be more up-to-date on the slave than on the master and/or
make rdist ignore certain file attributes like file owner and group.

Setting Up `rdist`

Here's an outline how to set up rdist to work in your environment. If you want to use it in conjunction with ssh make sure that ssh is running first.

Get, compile and install an up-to-date version of rdist. Sources should be available from your favourite FTP site. At the time of this writing the latest version is 6.1.5. Note that many Un*xen come with an older version you shouldn't use. Make sure you install the "daemon" binary rdistd on all slave machines, preferably somewhere within the default PATH or at least in the same directory on all machines. If you install rdistd outside the default PATH you need to specify where to find it with the "-p path-to-rdistd" option.
Set up the underlying transport and authentication mechanism. Choose between standard security-free rsh and non-standard secure ssh.
1. By default rdist uses the traditional rlogin/rsh for this. If that's what you want you need to add the master machine to the ~/.rhosts or /etc/hosts.equiv or whatever file on all slave machines. And of course the slave machines have to run the rshd, which is usually started via inetd. Make sure that an "rsh slave hostname" will output the hostname of the slave.
2. If you're in any way concerned about security you should set up ssh so that you can connect from the master to the slave machines. Make sure that the master knows about all slaves in its known_hosts file by ssh'ing to all slaves once. To use rdist with ssh you need to specify ssh as the underlying transport mechanism using the option "-P path-to-ssh".
Now see if rdist works. First create some temporary file in /tmp to distribute:
```
	  master$ cd /tmp
	  master$ echo "Hello, world!" >distribute-me
	
```
To test our setup use the "-c" option which allows to specify a mini "distfile" on the command line and send it to a slave. Don't forget to add the "-p path-to-rdistd" and/or "-P path-to-ssh" options here if necessary.
```
	  master$ rdist -c distribute-me slave:/var/tmp
	
```
This will distribute our file /tmp/distribute-me from the master to the /var/tmp directory on our slave.
If you want to make life easier you may want to create a simple shell script that takes care of those "-p" and "-P" options. Try this one:
```
	  #! /bin/sh
	  SSH="`which ssh`"
	  RDISTD="`which rdistd`"
	  
	  rdist -p "$RDISTD" -P "$SSH" "$@"
	
```
Adjust as needed, especially if you don't have ssh and/or rdistd in your path at all.

Now the environment is ready to use rdist. At least for one-shot jobs, that is. The real power of rdist is in its distfiles however, so read on.

Elementary Distfiles

Basics

A distfile defines what files and file hierarchies to copy where and with which options. You may specify a particular distfile using the "-f distfile" option. Otherwise rdist will default to a default filename "Distfile" in the current working directory.

The syntax is somewhat similar to a standard makefile. Here's a distfile that'd copy our previous test file /tmp/distribute-me to some slave machine slave_1.

      # Distfile 1
      /tmp/distribute-me -> slave_1
          install /var/tmp ;

This distfile tells rdist to install our file on slave_1 in its /var/tmp directory. In detail the first line specifies the source file and the destination machine. The second line tells that the file specified needs to be installed in the /var/tmp directory of the slave.

Now we may want to specify multiple files and/or multiple destination machines. Doing so we need to put parentheses around them, like this:

      # Distfile 2
      ( /var/www /var/ftp ) -> ( slave_1 slave_2 slave_3 )
          install -o remove /var ;

This will copy the contents of both /var/www and /var/ftp to all three slave machines. The -o option to install allows us to specify a wide range of additional options. In this particular case we tell rdist to remove all files on the slaves that don't exist on the master. Use this option with care, a typo in the wrong spot may wipe out lots of data on the slaves.

If you just want to see what would happen if you ran rdist without actually doing the update you may add an option "verify" to the install command. That'll make rdist do a dry run that doesn't modify anything on the target machine. See below how you add this option on the command line.

Synchronizing directory trees with lots of small files will take some time even if no files need to be copied at all. To speed things up we may define makefile-style targets that can be invoked on the command line. So we redo our previous distfile again.

      # Distfile 3
      www: /var/www -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;
      
      ftp: /var/ftp -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;

Different to make an invocation of rdist that doesn't specify a particular target or set of targets will have all targets installed, not just the first one. So if we invoke rdist with this distfile it'll behave exactly like No.2. But if we apply an additional argument "www" or "ftp" to the end of our rdist invocation only the target of that name will be installed.

There are other commands besides the "install" command we've seen. You may exclude files and/or file patterns from distribution using the "exclude" and "exclude_pat" commands. The former takes a list of file names as arguments and excludes them from installation:

      # Distfile 4
      ftp: /var/ftp -> ( slave_1 slave_2 slave_3 )
               install -o remove /var ;
               except /var/ftp/Distfile /var/ftp/incoming ;

This will prevent rdist to install Distfile and the incoming directory.

Even more powerful is the except_pat command. It will exclude files according to a ed(1) style regular expression:

      # Distfile 5
      etc: /etc -> ( archivehost )
               install -o remove /archive/master/etc ;
               except_pat ~$ ;

Note that you need to escape the dollar character (marking the end of line) in this pattern. It will skip all files ending with a tilde character.

Variables

Similar to makefiles we may use variables in our distfiles. These variables provide a means to "recycle" definitions of file and target groups to simplify the maintenance of distfiles. Here's a very simple example that shows what variables may be good for:

      # Distfile 6
      HOSTS= ( slave_1 slave_2 slave_3 )
      FILES= (
               /etc/hosts
               /etc/inet/ntp.conf 
               /etc/services 
             ) 
      
      base-configs: ${FILES} -> ${HOSTS}
      install -owhole / ;

So far this doesn't look particularly useful. It does however show how we use variables---we'll see what they're good for in the following section. The syntax for both variable definition and variable expansion is very much like in a makefile again, except that we must use curly braces around the variable name, not parentheses. Yes, you may skip the braces if your variable name is only a single character long.

Set Operations

The distfiles so far have basically distributed files using a one-to-one scheme. Especially in the last example we may however see a hint of the problems to come: What if we want to maintain configuration files for multiple machines running different Un*x versions? We may use set operations to define which files are host specific, OS specific or generally used. Here is another (simplified) example:

      # Distfile 7
      
      # Consider your lists of hosts here
      SOLARISHOSTS= ( sun1 sun2 sun3 )
      FREEBSDHOSTS= ( chuck1 chuck2 )
      NETBSDHOSTS=  ( pdp1 pdp2 )
      OPENBSDHOSTS= ( marvin1 marvin2 )
      
      LEAFNODES=    ( sun1 sun2 )
      
      # The list of files going to /etc (BSD) or /etc/inet (Solaris) here.
      INETCFG=         ( /etc/services /etc/protocols /etc/hosts /etc/ntp.conf )
      INSECUREINETCFG= ( /etc/inetd.conf )

      
      # No serviceable parts beyond this line (read on, though)
      
      BSDHOSTS=     ( ${FREEBSDHOSTS} + ${NETBSDHOSTS} + ${OPENBSDHOSTS} )
      HOSTS=        ( ${SOLARISHOSTS| + ${BSDHOSTS} )
      
      etc: ( ${INETCFG} + ${INSECUREINETCFG} ) -> ${SOLARISHOSTS}
               install /etc/inet ;
      
      etc: ${INETCFG} -> ${BSDHOSTS}
               install /etc;
      
      etc: ${INSECUREINETCFG} -> ( ${BSDHOSTS} - ${OPENBSDHOSTS} )
               install /etc ; 
      
      etc: /etc/notrouter -> ( ${SOLARISHOSTS} & ${LEAFNODES} )
               install /etc ;

This example shows what you can do with those set operations. Even though we maintain a single master copy of various configuration files we place them on on the destination machines in different directories according to their architecture and/or operating system. In the case of the "inetd.conf" we explicitly exclude the extra-paranoid OpenBSD installations because they don't run an inetd in our example. And for the Solaris workstations we make sure they don't try to route anything by installing the file "/etc/notrouter" on them.

The exact syntax for set operations is this: Every individual object (like a file or target machine name) is considered a set. So is a sequence of objects surrounded in parentheses. So is a variable reference referring to an set. We may use the set operations "+", "&" and "-" on multiple sets to compute the union, intersection and exclusion of those sets, respectively.

Use of these set operations may not be nested within an individual expression. You may however use intermediate variables to work around this. Yes, this is considered a silly limitation even within the man page.

The "`special`" And "`cmdspecial`" Commands

It may happen that you want to execute some sort of command on the target machines whenever you have changed a file. There are two commands to do so. The "special" command is executed for every file that has been updated by the rule it has been given while the "cmdspecial" command will be executed once when at least one of the files specified in a rule has been updated.

The command specified is run on the target machine from within the home directory of the user you connect to. It is run through the Bourne shell so you may use whatever the shell has to offer. The "special" command has the environment variable "FILE" set to the path of the file on the master machine and "REMFILE" variable set to the path of the file on the target machine. The "cmdspecial" executes with the environment variable "FILES" containing a colon-separated list of remote file names that have been updated.

The command string needs to be quoted using double quotes. I haven't bothered to figure out the exact details about quoting mechanisms. Instead I rather send a full-blown shell script to the target machine and then execute it.

Here is an example. It logs whatever file is updated to the targets syslogd using logger. Once it has finished it also appends all the new files to a tar file.

    ( /usr /bin /sbin ) -> ( test1 test2 test3 )
	install / ;
        special "logger Updated $REMFILE" ;
	cmdspecial "tar rf /tmp/rdist-update.tar `echo $FILES | tr : ' '`"

Command Line Options

You may specify several command line options to rdist. Among the more useful are these:

-D: Create debugging output.
-d var=value: Set a variable. This overrides variable definitions in the distfile.
-f distfile: Specify an alternate distfile.
-M num: Specify the number of target machines to be updated in parallel. Default value is 4.
-m machine: Restrict updates to the machine given. May be used multiple times.
-n: Only show the rules that would be applied to the machines. Best used in conjunction with -m.
-o distopts
: Specify additional options to all install commands. Multiple options may be separated by comma. This is particularly useful with the verify and compare options.
-p remote-path-to-rdistd: Tell rdist where to look for rdistd on the target machines.
-P path-to-rsh: Tell rdist where to find rsh or whatever replacement (like ssh) on the master machine.
-V: Make rdist output its version. Use this in shell scripts where it depends on the PATH environment variable which version of rdist is being run.

There are several other options to rdist which are of less importance than the ones shown here. Consult the manual pages for the rest of them.

Advanced Features

Now that we've mastered the basics of rdist we take a look at some additional features that are not of fundamental importance to the everyday use of rdist but still prove useful for more special purposes.

Additional `install` Options

The "`quiet`" Option

If you want to make the install command be less talkative you specify an additional option "quiet". As a personal preference however I rather keep the output from an rdist run in a file and either "grep -v" that file afterwards or run rdist again to get a short "everything is up to date"-style output.

Modifying the Update Conditions

By default rdist always updates files whenever the stat(2) syscall returns differences in file size, time stamps, permissions or owners. In case of the former two criteria the file content is updated, in case of the latter two only the permissions and ownership. This default behaviour can be modified using the following options to install.

"younger": Don't update the destination file if it is younger than the source file. So this option should really be named "notyounger" or "older" or whatever...
"compare": Force a content comparison of the source and destination file. This is obviously bandwidth intensive so only use it if absolutely necessary.
"ignlnks": By default rdist will complain about stale symlinks. This option will turn these warnings off.
"chknfs": Do not update files located on an NFS-mounted volume.
"chkreadonly": If the destination file is on a RO-mounted volume or file system then don't bother about it.
"nodescend": Don't recursively update directories. Normally, rdist will update the whole directory tree if it updates a directory. With this option it will only update the directory itself, i.e. its attributes.
"nochkowner" and "nochkgroup": Don't check the owner and group of a file, respectively. This is particularly useful if you run rdist as ordinary user on the destination machines.
"numchkowner" and "numchkgroup": By default rdist checks the user and group names. These options make it compare the numeric UID and GID, respectively. This is particularly useful if you want to provide a cross-architecture installation server or such.
"nochkmode": Don't check the permissions on the target files.

Modifying the Update Behaviour

"whole"

Normally the destination file path is the concatenation of the destination directory specified (or "~" if none was specified) and the last component of the source path. A rule of the form

          # Distfile 8
          /var/tmp/distribute-me -> slave_1
              install /export/home ;

would have the file installed as /export/home/distribute-me on slave_1. If however you gave the install command the whole option you would install the file as /export/home/var/tmp/distribute-me.

Whenever you use this option make double sure things go where you want. Especially if you're using it in conjunction with "remove"!

"savetargets"

Whenever a file is updated, keep the old file as a backup copy with an extension ".OLD".

"sparse"

Deal sparse files efficiently. Don't forget this if you distribute ndbm files and such.

Message Logging and Change Notification

It is possible to make rdist generate various kinds of logging output. By default it is already writing copious output to stdout when running. You may however want it to log even more things to stdout, the master or slave machines syslog facility or log file or an e-mail address. Since these features are somewhat beyond the scope of an "introductory" text and I personally rather use sed or perl to extract whatever I want from the output I won't go into detail about these. Take a look at the man page to find out about this---it has a section dedicated to this issue.

There is also a mechanism available to generate lists of files that have changed since some reference file has been last modified. Again, see the man page. I personally rather run rdist with the -overify option instead.

Strategic Considerations and Best Practices

Now that we've made it through the features of rdist it is time to consider approaches how to use it best. Here comes a list of issues worth to consider:

In any security-related context run rdist through ssh.
The easiest way to see if an rdist run completed successfully is to run it again, grepping for anything but "*: updating host *" and "*: updating of * finished". If there is anything like that there has been some problem.
The default number of machines to update in parallel is just four. If your master machine can handle some load but you want to keep the load on the slaves low consider updating way more machines in parallel using the "-M" option.
You may want to distribute things from a dedicated "master image" server. This provides some sort of repository mechanism, which may be helpful especially together with some sort of version control environment (from CVS to some backup software). But you may also consider using a peer machine as "master", which lets you test your changes on that machine and then push it to all the slaves in an easier way. In this second case consider to add the master machine to the list of machines to update. This will let you push changes from all machines, using whatever machine you want as "master".
Probably the toughest decision to make is if you want to arrange your Distfile targets by directory or by some sort of "package" unit. Arranging by directory allows easy use of the "-oremove" option but effectively forces you to keep the distributed directories on all machines exactly the same. Using a "package" style approach lets you easily define which files need to go to which machine but makes it almost impossible to remove garbage files.

Conclusion

The features rdist provides are simple but extremely useful to handle groups of machines that share some common files that occasionally need to be updated in a systematic way. Considering the simplicity and scope of rdist it is surprising that it isn't way more widely known and used. I sincerely hope that this little documentation (which has become slightly longer than the original man page) will help to change this.