multx (MULTiple eXecution) is a shell script that invokes the same command(s) on multiple "target" hosts. multx can execute these command(s) either sequentially (foreground) or in parallel (background). multx has only a few command line arguments, so ∫knowledge dt is tiny.

It's useful when you must execute commands on multiple *NIX hosts and you don't have access to a working configuration management tool that you know how to use.

In order to use multx without having to (interactively) type passphrases or passwords (except for the initial setup, of course!), the use of two other shell scripts is strongly recommended. The three scripts are:

• multx - The script that actually executes the commands on the remote hosts. To be most useful, use multx together with the two scripts below. multx will almost certainly need to be edited to reflect local hostnames/IP addresses. That is:

  multx is a template to be edited as required.

• start-ssh-agent - The script that sets up ssh-agent and the environment.
• stop-ssh-agent - The script that stops ssh-agent and cleans up the environment.

Download these shell_scripts from the github repository.

Note: once ssh-agent is running with the correct configuration, all other commands that are based on ssh will work without being prompted for authentication as well. For example, scp and rsync.

The Default multx Usage Message

Preparation: Configuring ssh and ssh-agent

Before one can use multx without typing passphrases or passwords every time (strongly recommended), ssh must be configured on all the target systems and ssh-agent on at least one initiating system. See manage_ssh_agent for details on how to do this.

Simple Examples of Running multx

Note that typing passwords or passphrases is not required in any of the examples below. Also, it would be just as easy to run remote commands on 100 target hosts as on a few.

1. Running hostname on set 1 of hosts (foreground):
   $ multx -1 hostname Sat Feb 13 15:37:08 AEDT 2016 : multx starting fred === <hostname> === fred wilma === <hostname> === wilma Sat Feb 13 15:37:17 AEDT 2016 : multx finished
2. Running hostname on set 1 of hosts (background):
   $ multx -1b hostname Sat Feb 13 15:38:09 AEDT 2016 : multx starting fred === <hostname> === wilma === <hostname> === wilma fred Sat Feb 13 15:38:09 AEDT 2016 : multx finished
   Note that the order of the output is non-deterministic when using the background option!
3. Creating a file with date on all hosts (background):
   $ multx -ab 'date > /tmp/junk' Sat Feb 13 15:42:46 AEDT 2016 : multx starting fred === <date > /tmp/junk> === wilma === <date > /tmp/junk> === barney === <date > /tmp/junk> === betty === <date > /tmp/junk> === pebbles === <date > /tmp/junk> === bamm-bamm === <date > /tmp/junk> === Sat Feb 13 15:42:48 AEDT 2016 : multx finished
4. Using sed to "edit" a file [create a modified new file] (background)"
   $ multx -ab 'sed s/2016/1492/ /tmp/junk > /tmp/rubbish' Sat Feb 13 15:45:41 AEDT 2016 : multx starting fred === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === wilma === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === barney === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === betty === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === pebbles === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === bamm-bamm === <sed s/2016/1492/ /tmp/junk > /tmp/rubbish> === Sat Feb 13 15:45:42 AEDT 2016 : multx finished
5. Listing the contents of a file with cat (foreground):
   $ multx -a 'cat /tmp/rubbish' Sat Feb 13 15:45:59 AEDT 2016 : multx starting fred === <cat /tmp/rubbish> === Sat Feb 13 12:42:47 AWST 1492 wilma === <cat /tmp/rubbish> === Sat Feb 13 12:42:47 AWST 1492 barney === <cat /tmp/rubbish> === Sat Feb 13 12:42:47 AWST 1492 betty === <cat /tmp/rubbish> === Sat Feb 13 12:42:47 AWST 1492 pebbles === <cat /tmp/rubbish> === Sat Feb 13 12:42:47 AWST 1492 bamm-bamm === <cat /tmp/rubbish> === Sat Feb 13 00:46:21 AWST 1492 Sat Feb 13 15:46:04 AEDT 2016 : multx finished
6. Checking if all hosts have the same version of a file (background, quiet mode):
   $ multx -abq 'echo $(cksum /etc/sysconfig/postfix) $(hostname) $(uname)' | sort 409984396 9662 /etc/sysconfig/postfix barney Linux 409984396 9662 /etc/sysconfig/postfix betty Linux 409984396 9662 /etc/sysconfig/postfix fred Linux 409984396 9662 /etc/sysconfig/postfix wilma Linux 452371448 14776 /etc/sysconfig/postfix bamm-bamm Linux 452371448 14776 /etc/sysconfig/postfix pebbles Linux Sat Feb 13 15:43:06 AEDT 2016 : multx starting Sat Feb 13 15:43:08 AEDT 2016 : multx finished
7. Checking if all hosts have the correct date (background, quiet mode):
   $ multx -abq 'echo $(date) $(hostname)' | sort Sat Feb 13 17:00:16 AWST 2016 bamm-bamm Sun Feb 14 04:56:41 AWST 2016 barney Sun Feb 14 04:56:41 AWST 2016 betty Sun Feb 14 04:56:41 AWST 2016 fred Sun Feb 14 04:56:41 AWST 2016 pebbles Sun Feb 14 04:56:41 AWST 2016 wilma Sun Feb 14 07:56:40 AEDT 2016 : multx starting Sun Feb 14 07:56:42 AEDT 2016 : multx finished
8. Displaying a particular line of interest from a particular file (background, quiet mode):
   $ multx -abq 'echo $(hostname) : $(grep ^users: /etc/group)' | sort bamm-bamm : users:x:100:mux barney : users:x:100:lux,mux betty : users:x:100:mux fred : users:x:100:tux,lux,mux pebbles : users:x:100:lux,mux wilma : users:x:100:lux,mux Sun Feb 14 07:55:06 AEDT 2016 : multx starting Sun Feb 14 07:55:08 AEDT 2016 : multx finished
9. Running commands as root (with sudo) on the target host(s). In this example, 'hostname' is run as your user ID and calling sudo runs 'id' as root (background, quiet mode). Double quotes are particularly useful here:
   $ multx -1bq 'printf "%-10s: %s\n" $(hostname) "$(sudo id)"' | sort fred : uid=0(root) gid=0(root) groups=0(root),2000(nfsuser) wilma : uid=0(root) gid=0(root) groups=0(root),2000(nfsuser),2001(foouser) Sun Feb 14 21:35:18 AEDT 2016 : multx starting Sun Feb 14 21:35:19 AEDT 2016 : multx finished

   If sudo on the target(s) fails with a sudo tty error, for example:

   $ multx -1q 'sudo id' Tue Apr 4 10:46:05 AEST 2017 : multx starting sudo: sorry, you must have a tty to run sudo sudo: sorry, you must have a tty to run sudo Tue Apr 4 10:46:07 AEST 2017 : multx finished
   Tell multx to pass ssh the option(s) it needs to allow this access using '-s option(s)'. Typically, "-s -t" or "-s -tt"):
   $ multx -s -t -1q 'sudo id' Tue Apr 4 10:46:16 AEST 2017 : multx starting uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 Connection to fred closed. uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 Connection to wilma closed. Tue Apr 4 10:46:18 AEST 2017 : multx finished

   Of course one can do much more interesting things than just 'id'. One can do literally anything. For example, stop/start services, or even stop/reboot the remote host(s)! Specifying the multx background option ("-b") may require "-s -tt":

   $ multx -s -tt -1bq sudo systemctl stop dse.service Tue Apr 4 10:46:16 AEST 2017 : multx starting Connection to fred closed. Connection to wilma closed. Tue Apr 4 10:46:18 AEST 2017 : multx finished
10. Adding and/or deleting particular host(s):
    $ multx -12bq -h pebbles -x barney -x wilma hostname - or, equivalently: $ multx -12bq -h pebbles -x 'barney|wilma' hostname Mon Mar 7 08:04:25 AEDT 2016 : multx starting fred betty pebbles Mon Mar 7 08:04:25 AEDT 2016 : multx starting

Hopefully, the examples above convey the basic idea. Of course, multx can do many more things than the simple examples above!

Rolling Out a New File

1. Copy the new file (newfile) to all "target" hosts. This can be done in two ways:
   
   
   1. If all the hosts are configured to be both "initiators" and "targets", (recommended, if practical) use this method. Call multx on the "original initiator" to execute scp on all the "target" hosts. This will copy newfile from the "original initiator" to newfile.tmp on all the target hosts (including to itself):
      $ multx -abq '. .ssh/ssh-agent; scp -p initiator:newfile newfile.tmp
   2. If there is only one "initiator", one would have to copy to each "target" host in turn:
      $ scp -p newfile target_1:newfile.tmp
      ...
      $ scp -p newfile target_N:newfile.tmp

      One quick and easy way to do this would be with a script like this:

      INITIATORNAME=$1; TARGETNAME=$2; shift; shift for HOST in $*; do scp -p $INITIATORNAME $HOST:$TARGETNAME & # Launch in background for asynchronous copies done wait
      And then execute that script:
      $ script newfile newfile.tmp target_1 target_2 ... target_N
      Launching each scp in the background means all the copies happen asynchronously. And, of course, no passwords or passphrases would be required.
2. Make a backup of the original file (optional):
   $ multx -abq 'cp -p /usr/local/bin/file /usr/local/bin/file.save'
3. Install the new file:
   $ multx -abq 'cp newfile.tmp /usr/local/bin/file'
4. After a suitable period of time... Cleanup (optional):
   $ multx -abq 'rm newfile.tmp /usr/local/bin/file.save'

Change the Same File on Multiple Hosts

1. Create a new version of the file in every host:

   On set 1 of hosts:
   $ multx -1bq 'sed "/^ *- seeds: *".*"$/s/\".*\"/\"1.1.1.1,2.2.2.2,3.3.3.3\"/" /etc/dse/cassandra/cassandra.yaml > /tmp/new.yaml'

   On set 2 of hosts:
   $ multx -2bq 'sed "/^ *- seeds: *".*"$/s/\".*\"/\"4.4.4.4,5.5.5.5,6.6.6.6\"/" /etc/dse/cassandra/cassandra.yaml > /tmp/new.yaml'

2. Check the new version on both sets hosts (optional, but strongly recommended):
   $ multx -12 'diff /etc/dse/cassandra/cassandra.yaml /tmp/new.yaml'
   or, alternately:
   $ multx -12qb 'echo $(diff /etc/dse/cassandra/cassandra.yaml /tmp/new.yaml) $(hostname)' | sort
3. Update the configuration file with the modified version on both sets of hosts:
   $ multx -12bq 'cp /tmp/new.yaml /etc/dse/cassandra/cassandra.yaml'

The only limit is your imagination!

Store Some Info from Each Target Host in a Named File on the Initiating Host

There are (as of version 0.1.0) at least two ways to do this. First, the easy way, using the '-o outfilepat' method. Note that one must prevent the initiating system's shell from attempting to evaluate ${HOSTNAME} by 'escaping' the $. This can be done by preceding the $ with a \ (\${HOSTNAME}) or by surrounding it with single quotes ('${HOSTNAME}').

1. $ multx -1qb -o '${HOSTNAME}.info' 'rpm -qa | sort'
2. If all the hosts are configured to be both "initiators" and "targets" and ssh-agent is running on all systems with a valid passphrase (see above), one could do the same thing 'old school' (the hard way):

   $ multx -1bq 'rpm -qa | sort > tmp/junk; . .ssh/ssh-agent.env; scp /tmp/junk fred:$(hostname).info; rm /tmp/junk'

Both of these commands should produce files like the ones below on the "initiating" host (which is fred in the second example) - each of which contains the information requested. In this example, the information requested is a sorted list of all the packages installed on each of the target hosts.

Customizing multx to Your Environment(s)

Remember: multx is a template to be edited as required.

If, on the off chance, your hostnames are not Flintstones character names, but (say) Star Trek, edit multx accordingly.

The required changes are obvious, and must be made in 4 places. In the Flintstones to Star Trek example:

1. In the HOST_GROUP section:
   1. Change '-1' to '-C' (for Captains).
   2. Change 'HOST_GROUP_1="..."' to 'HOST_GROUP_C="kirk picard sisko janeway archer"'.
   3. Change '-2' to '-F' (for First officers).
   4. Change 'HOST_GROUP_2="..."' to 'HOST_GROUP_F="spock riker nerys chakotay tpol worf"'.
   5. Delete '-3' if it's not required.
2. Change 'OPTIONS=123abch:f:o:qx:' to 'OPTIONS='CFabch:f:o:qx:'
3. In the getopts case statement, change '1)' to 'C)', '2)' to 'F)' and delete the line with '3)'
4. In the usage message, change '-1' to '-C', '-2' to '-C' and delete the '-3' line.

Using multx with -f hostnames_file

If you can't be bothered changing multx, it's possible to use it with the '-f file' option (here specified twice). These must be files containing the relevant hostnames.

This example collects the packages installed on all the hosts in DC_1 and DC_2 (assuming these two files exist and have valid contents, of course):

$ multx -f DC_1_hostnames -f DC_2_hostnames -bq -o '$HOSTNAME.packages' 'rpm -qa | sort'

Notes

1. multx takes your commands and runs them on multiple hosts. If your commands do bad things, multx just causes those bad things to happen on multiple systems. I take no responsibility whatsoever for any damage you cause when running your commands using multx!!!
2. As with any parallel tool, be careful. If you make a mistake, you make it N times!
3. Everything on this web page assumes one is running bash, sh, or ksh. If you're still running csh after all these years, there's no hope for you!
4. With the '-o outfilepat' option, '$H' works exactly like '$HOSTNAME'. If curly braces would be required by a shell, multx also requires them. E.g., '$HOSTNAME.info' does not require them, but '${HOSTNAME}_info' does.
5. I've never had more than 20 hosts to test them all logging in to the "original initiator" simultaneously. I recall that I had to increase MaxStartups above the (then) default of 10 in /etc/ssh/sshd_config.
6. To permit root to login using ssh, set PermitRoot to yes in /etc/ssh/sshd_config.
7. Other tools such as pssh or clusterssh may also be useful. multx has always done everything I needed to do.
8. In background (asynchronous) mode - where commands are running on lots of hosts at the same time - and the commands are generating output, it's possible that the output from some host(s) could become interspersed with the output from other(s). The only time this has ever been observed in the wild is when multiple output-producing commands are issued in the same multx call. For example, multx -abq 'command_1; command_2'. If this is a small problem, remove the '-b' option. If it's a big problem, let me know and I'll investigate.
9. multx (& co) also runs on Cygwin. It has not been tested on *BSD.