multx - Execute Commands on Multiple Remote Linux/UNIX Hosts

multx (MULTiple eXecution) is a shell script that invokes the same command(s) on multiple target hosts. multx can execute these command(s) either sequentially (foreground) or in parallel (background). multx has only a few command line arguments, so ∫knowledge dt is tiny.

It's useful when you must execute commands on multiple *NIX hosts and you don't have access to a working configuration management tool that you know how to use.

In order to use multx without having to (interactively) type passphrases or passwords (except for the initial setup, of course!), the use of two other shell scripts is strongly recommended. The three scripts are:

• multx - The script that actually executes the commands on the remote hosts. To be most useful, use multx together with the two scripts below. While multx can be used in its unaltered form, it is exceedingly likely that customizing multx to reflect your hostnames/IP addresses will make it much easier to use. That is:

  multx is best used a template to be customized as required.

• start-ssh-agent - The script that sets up ssh-agent and the environment.
• stop-ssh-agent - The script that stops ssh-agent and cleans up the environment.

Download these shell_scripts from the github repository.

Note: once ssh-agent is running with the correct configuration, all other commands that are based on ssh will work without being prompted for authentication as well. For example, scp and rsync.

The Default multx Usage Message

Preparation: Configuring ssh and ssh-agent

Before one can use multx without typing passphrases or passwords every time (strongly recommended), ssh must be configured on all the target systems and ssh-agent on at least one initiating system. See manage_ssh_agent for details on how to do this.

Simple Examples of Running multx

Note that typing passwords or passphrases is not required in any of the examples below. Also, it would be just as easy to run remote commands on 100 target hosts as on a few.

1. Running hostname on the group "F" hosts (foreground):
   
   $ multx -F hostname [i: Wed Jan 10 08:46:26 AEDT 2024 : multx end] [i: Wed Jan 10 11:08:29 AEDT 2024 : multx begin] [i: fred === <ssh fred 'hostname'> ===] fred [i: wilma === <ssh wilma 'hostname'> ===] wilma [i: Wed Jan 10 11:08:30 AEDT 2024 : multx end]
2. Running hostname on the group "F" hosts (background):
   
   $ multx -Fb hostname [i: Wed Jan 10 11:08:30 AEDT 2024 : multx begin] [i: fred === <ssh fred 'hostname'> ===] [i: wilma === <ssh wilma 'hostname'> ===] fred wilma [i: Wed Jan 10 11:08:30 AEDT 2024 : multx end]
   Note that the order of the output is non-deterministic when using the background option.
3. Creating a file with date on all hosts (background):
   
   $ multx -ab 'date > /tmp/junk' i: Wed Jan 10 11:08:30 AEDT 2024 : multx begin] [i: fred === <ssh fred 'date > /tmp/junk'> ===] [i: wilma === <ssh wilma 'date > /tmp/junk'> ===] [i: barney === <ssh barney 'date > /tmp/junk'> ===] [i: betty === <ssh betty 'date > /tmp/junk'> ===] [i: pebbles === <ssh pebbles 'date > /tmp/junk'> ===] [i: bamm-bamm === <ssh bamm-bamm 'date > /tmp/junk'> ===] [i: Wed Jan 10 11:08:31 AEDT 2024 : multx end]
4. Using sed to create/overwrite a modified file on the remote hosts (background)"
   
   $ multx -ab 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish' [i: Wed Jan 10 11:08:31 AEDT 2024 : multx begin] [i: fred === <ssh fred 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: wilma === <ssh wilma 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: barney === <ssh barney 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: betty === <ssh betty 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: pebbles === <ssh pebbles 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: bamm-bamm === <ssh bamm-bamm 'sed s/20[[:digit:]][[:digit:]]/1492/ /tmp/junk > /tmp/rubbish'> ===] [i: Wed Jan 10 11:08:31 AEDT 2024 : multx end]
5. Listing the contents of a file with cat in (foreground):
   
   $ multx -a 'cat /tmp/rubbish' [i: Wed Jan 10 11:08:31 AEDT 2024 : multx begin] [i: fred === <ssh fred 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:30 AEDT [i: wilma === <ssh wilma 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:30 AEDT [i: barney === <ssh barney 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:31 AEDT [i: betty === <ssh betty 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:31 AEDT [i: pebbles === <ssh pebbles 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:31 AEDT [i: bamm-bamm === <ssh bamm-bamm 'cat /tmp/rubbish'> ===] Wed 10 Jan 2024 11:08:31 AEDT [i: Wed Jan 10 11:08:33 AEDT 2024 : multx end]
6. Checking if all hosts have the same version of a file (background, quiet mode):
   
   $ multx -abq 'echo $(cksum /etc/hosts) $(hostname) $(uname)' | sort -f 3897348360 158 /etc/hosts bamm-bamm Linux 3897348360 158 /etc/hosts barney Linux 3897348360 158 /etc/hosts fred Linux 3897348360 158 /etc/hosts pebbles Linux 3897348360 158 /etc/hosts wilma Linux 9384084572 1234 /etc/hosts betty Linux [i: Wed Jan 10 11:08:33 AEDT 2024 : multx begin] [i: Wed Jan 10 11:08:33 AEDT 2024 : multx end]
   It is very easy to confirm that all these files are the same (or not). Note that 'sort -f' runs on the initiating system, so the status/information ('i:') lines appear at the end of the output.
7. Same as above, but display only the ones with incorrect configuration:
   
   $ multx -abq 'echo $(cksum /etc/hosts) $(hostname) $(uname)' | grep -v '^3897348360 158 ' [i: Wed Jan 10 09:05:06 AEDT 2024 : multx begin] 9384084572 1234 /etc/hosts betty Linux [i: Wed Jan 10 09:05:06 AEDT 2024 : multx end]
8. Checking if all hosts have the correct date (background, quiet mode):
   
   $ multx -abq 'echo $(date) $(hostname)' | sort -f Wed 10 Jan 2024 08:57:10 AEDT bamm-bamm Wed 10 Jan 2024 08:57:10 AEDT barney Wed 10 Jan 2024 08:57:10 AEDT betty Wed 10 Jan 2024 08:57:10 AEDT pebbles Wed 10 Jan 2024 08:57:10 AEDT wilma Wed 10 Jan 2024 23:14:12 AEDT fred [i: Wed Jan 10 08:57:10 AEDT 2024 : multx begin] [i: Wed Jan 10 08:57:10 AEDT 2024 : multx end]
   It looks like the system time is incorrect on fred.
9. Displaying a particular line of interest from a particular file (background, quiet mode):
   
   $ multx -abq 'echo $(hostname) : $(grep ^users: /etc/group)' | sort -f bamm-bamm : users:x:100: barney : users:x:100: betty : users:x:100: fred : users:x:100: pebbles : users:x:100: wilma : users:x:100: [i: Wed Jan 10 08:57:10 AEDT 2024 : multx begin] [i: Wed Jan 10 08:57:10 AEDT 2024 : multx end]
10. Running commands as root (with sudo) on the target host(s). Note that it is assumed that the user can execute sudo on the target host(s) without entering a password.

    In this example, 'hostname' is run as your user ID and calling sudo runs 'id' as root (background, quiet mode). Double quotes are particularly useful here:
    

    $ multx -Fbq 'printf "%-10s: %s\n" $(hostname) "$(sudo id)"' | sort -f fred : uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 wilma : uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [i: Wed Jan 10 08:57:10 AEDT 2024 : multx begin] [i: Wed Jan 10 08:57:11 AEDT 2024 : multx end]

    On some older Red Hat systems (and perhaps some others?), sudo on the target(s) sometimes fails with a tty error, for example:

    $ multx -Fq 'sudo id' [i: Tue Apr 4 10:46:05 AEST 2017 : multx begin] sudo: sorry, you must have a tty to run sudo sudo: sorry, you must have a tty to run sudo [i: Tue Apr 4 10:46:07 AEST 2017 : multx end]

    One solution is to tell multx to pass ssh the option(s) it needs to allow this access using '-s ssh_option(s)'. Typically, "-s -t" or "-s -tt"):

    $ multx -s -t -Fq 'sudo id' [i: Tue Apr 4 10:46:16 AEST 2017 : multx begin] uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [i: Tue Apr 4 10:46:18 AEST 2017 : multx end]

    Of course one can do much more interesting things than just 'id'. One can do literally anything. For example, manage configuration files, stop/start/restart services, shutdown/reboot target host(s), etc.

    $ multx -Fbq sudo systemctl restart httpd.service [i: Wed Jan 10 08:57:11 AEDT 2024 : multx begin] [i: Wed Jan 10 08:57:12 AEDT 2024 : multx end]
11. Adding and/or deleting particular host(s) from host group(s):
    
    $ multx -FRbq -h pebbles -x barney -x wilma hostname - or, equivalently: $ multx -FRbq -h pebbles -x 'barney|wilma' hostname [i: Wed Jan 10 08:57:12 AEDT 2024 : multx begin] betty fred pebbles [i: Wed Jan 10 08:57:13 AEDT 2024 : multx end]
12. Specifying a different user on the target system(s) using "-l <username>":
    
    $ multx -l oracle -F ls /oracle

Hopefully, the examples above convey the basic idea. Of course, multx can do many more things than the simple examples above!

Saving Time and Effort

Executing commands on target servers as a user that can sudo to root without a password is highly recommended whenever maintaining files that are not manageable by that user. Otherwise, the user's password will have to be typed for every sudo command on every target.

Rolling Out a New File

1. Use the -i option to execute a command on the initiating (local) host once for each target host. In this case, copy a particular local file to each of the remote host(s):
   
   $ multx -abqi 'scp -p resolv.conf_new ${MULTXTARGET}:/tmp/resolv.conf_new'
2. Make a backup of the original file (optional):
   
   $ multx -abq 'sudo cp -p /etc/resolv.conf /etc/resolv.conf_save'
3. Install the new file:
   
   $ multx -abq 'sudo cp /tmp/resolv.conf_new /etc/resolv.conf'
4. After a suitable period of time... Cleanup (optional):
   
   $ multx -abq 'sudo rm /etc/resolv.conf_save /tmp/resolv.conf_new'

Executing Commands on the Initiating (Local) System

1. Call rsync for each F target to syncronize your user's bin directory to each target.
   $ multx -ibq -F 'rsync -az ~/bin/ ${MULTXTARGET}:bin'
2. Call rsync for each R target to syncronize all id* files in your .ssh directory to each target - except don't rsync files from the initiating system to itself, if it's in the specified target(s).
   $ multx -ibq -R -x "\b$(hostname)\b" 'rsync -az ~/.ssh/id* ${MULTXTARGET}:.ssh'
3. ${MULTXTARGET} and/or ${MT} can be used multiple times in the same command. This example consists of two steps:
   1. Create a directory named the same as the target host for each target host of interest.
   2. Copy all crontab files in /var/spool/cron/tabs on each target host into the corresponding directory.
   $ multx -a -ibq 'mkdir ${MULTXTARGET}' $ multx -a -ibq 'ssh ${MT} sudo tar -C /var/spool/cron/tabs -cf - . | tar -C ${MT} -xvf -'

Change the Same File on Multiple Hosts

1. Create a new version of the file on every target system:
   
   On group F of hosts:
   $ multx -Fbq 'sed "/^ *- seeds:/s/\".*\"/\"alma90,alma94\"/" /etc/cassandra/default.conf/cassandra.yaml > /tmp/new.yaml'
   
   On group R of hosts:
   $ multx -Rbq 'sed "/^ *- seeds:/s/\".*\"/\"alma94,alma90\"/" /etc/cassandra/default.conf/cassandra.yaml > /tmp/new.yaml'
2. Check the new version on both host groups F and R (optional, but strongly recommended):
   
   $ multx -FR 'diff /etc/cassandra/default.conf/cassandra.yaml /tmp/new.yaml' or, alternately: $ multx -FRqb 'echo $(diff /etc/cassandra/default.conf/cassandra.yaml /tmp/new.yaml) $(hostname)' | sort -f
3. Update the configuration file with the modified version on both host groups:
   
   $ multx -FRbq 'sudo cp /tmp/new.yaml /etc/cassandra/default.conf/cassandra.yaml'

The only limit is your imagination!

Store Some Info from Each Target Host in a Named File on the Initiating Host

Use the '-o outfile_template'. Note that one must prevent the initiating system's shell from attempting to evaluate ${MULTXTARGET} by 'escaping' the $. This can be done by preceding the $ with a backslash ('\') - \${MULTXTARGET} - or by surrounding it with single quotes - '${MULTXTARGET}'.

This should produce files like the ones below on the initiating host - each of which contains the information requested. In this example, the information requested is a sorted list of all the packages installed on each of the target hosts.

Customizing multx to Your Environment(s)

Remember: multx is best used as a template to be customized as required.

If, on the off chance, your hostnames are not 3 groups of Flintstones character names, but (say) two groups of Star Trek captains' and first officers' surnames - edit multx accordingly.

The required changes are obvious, and must be made in 4 places. In the Flintstones to Star Trek example:

1. In the HOST_GROUP environment variable definition section:
   1. Replace 'HOST_GROUP_F=...' with 'HOST_GROUP_C="kirk picard sisko janeway archer" # Captains'
   2. Replace 'HOST_GROUP_R=...' with 'HOST_GROUP_F="spock riker nerys chakotay tpol worf" # First Officers'
   3. Delete 'HOST_GROUP_K=...'
   4. Update 'HOST_GROUP_a="$HOST_GROUP_C $HOST_GROUP_F"'
2. Change 'OPTIONS=FRK...' to 'OPTIONS='CF...'
3. In the getopts case statement:
   1. Replace 'F) ... $HOST_GROUP_F' with 'C) ... $HOST_GROUP_C'
   2. Replace 'R) ... $HOST_GROUP_R' with 'F) ... $HOST_GROUP_F'
   3. Delete 'K) ... $HOST_GROUP_K' line
4. In the usage message:
   1. Replace '-F ... $HOST_GROUP_F' with '-C ... $HOST_GROUP_C'
   2. Replace '-R ... $HOST_GROUP_R' with '-F ... $HOST_GROUP_F'
   3. Delete the '-K ... $HOST_GROUP_K' line

Using multx with -f hostnames_file

If you can't be bothered changing multx, it's possible to use it with the '-f file' option (here specified twice). These must be files containing the relevant hostnames.

This example collects the packages installed on all the hosts in DC_1 and DC_2 (assuming these two files exist and have valid contents, of course):

Notes

1. multx takes your commands and runs them on multiple hosts. If your commands do bad things, multx just causes those bad things to happen on multiple systems. I take no responsibility whatsoever for any damage you cause when running your commands using multx!!!
2. As with any parallel tool, be careful. If you make a mistake, you make it N times!
3. Everything on this web page assumes one is running bash, sh, or ksh. If you're still running csh after all these years, there's no hope for you!
4. '$MT' is equivalent to'$MULTXTARGET' when using the -o or -i options. If curly braces would be required by a shell, multx also requires them. E.g., '$MULTXTARGET.info' does not require them, but '${MULTXTARGET}_info' does.
5. I've never had more than 20 hosts to test them all logging in to the "original initiator" simultaneously. I recall that I had to increase MaxStartups above the (then) default of 10 in /etc/ssh/sshd_config.
6. To permit root to login using ssh, set PermitRoot to yes in /etc/ssh/sshd_config.
7. Other tools such as pssh or clusterssh may also be useful. multx has always done everything I needed to do.
8. In background (asynchronous) mode - where commands are running on lots of hosts at the same time - and the commands are generating output, it's possible that the output from some host(s) could become interspersed with the output from other(s). The only time this has ever been observed in the wild is when multiple output-producing commands are issued in the same multx call. For example, multx -abq 'command_1; command_2'. If this is a small problem, remove the '-b' option. If it's a big problem, let me know and I'll investigate.
9. multx (& co) also runs on Cygwin. It has not been tested on *BSD.