On — run commands on multiple hosts

on is program to run a command on a number of target hosts via ssh in parallel. There are other such programs out there, but I wanted to write my own and shape it as I like it.

Main features are parallel execution and a flexible means to define and use target host groups in a configuration file.

The output of each command run is shown as it comes in, by default, or optionally sorted by host.

Usage

on is called like this:

on [options] <hostlist> <command ...>

The parameter consists of one or more host names and/or host group names (prefixed with @) separated by commas; is the command as to be run on the hosts.

A - instead of a host name lets on read one or more host names (separated by whitespace) from stdin. This does not work in group definitions or with group names.

See "Configuration" below for how to define host groups.

Options are parsed according to POSIX, meaning all options must be named before any other arguments and -- ends option processing, plus long options. See "Options" below for available options.

Examples

1. Host list. This runs the command needrestart -p on the hosts holme, naibel, and randers:

   on holme,naibel,randers needrestart -p
   

   We see that the first argument is the host list, host names separated by commas, followed by the command to be executed. The output has a header per host, with a lead string of ============= to be recognised easily, the host name, and the exit status of the program, followed by the command output on the next line:

   ============= randers: 0
   OK - Kernel: 6.12.63+deb13-amd64, Microcode: CURRENT, Services: none, Containers: none, Sessions: none|Kernel=0;0;;0;2 Microcode=0;0;;0;1 Services=0;;0;0 Containers=0;;0;0 Sessions=0;0;;0
   ============= holme: 0
   OK - Kernel: 6.8.0-90-generic, Microcode: CURRENT, Services: none, Containers: none, Sessions: none|Kernel=0;0;;0;2 Microcode=0;0;;0;1 Services=0;;0;0 Containers=0;;0;0 Sessions=0;0;;0
   ============= naibel: 0
   OK - Kernel: 6.12.63+deb13-amd64, Microcode: CURRENT, Services: none, Containers: none, Sessions: none|Kernel=0;0;;0;2 Microcode=0;0;;0;1 Services=0;;0;0 Containers=0;;0;0 Sessions=0;0;;0
   

   The output of each host is shown immediately as it arrives. This makes the order of the hosts in the total output unpredictable. If you need a predictable order, use the -s option to sort the output by host name. This implies that on waits for the output of all hosts to arrive. {I may want to change that.}

2. Host groups. In addition to single host names, you can use the names of defined groups, mixed into the host list, prefixed with @ to distiguish them from host names:

   on holme,naibel,@extern do_something
   

   Of course, like a host name, a group name can stand on its own:

   on @w21 touch /tmp/kilroy_was_here
   

Configuration

The configuration file, default path /opt/w21/etc/on-groups, defines the host groups that can be used. It has a simple syntax using clauses of the form group GROUPNAME { MEMBERS ... }, where the members can be host names or the members of another group referenced by @GROUPNAME.

Example:

# Comments are introduced with `#`.

group infra {                 # infrastructure servers
    holme
    naibel
    randers
}

group extern {
    nybo
    sundby
    tarup
}

# a group definition can reference other groups
group intern {
    @infra
    mellum
}

The ideas:

• whitespace is insignificant except as a separator
• group name defines a group name
• the group body is a list of hosts and/or group references in curly braces
• a group reference @name includes the members of group name
• group definitions may be recursive, but not cyclic
• it is an error to reference a non-existent group
• there is an implicit all group, consisting of all hosts in all defined groups
• group all may not be referenced in any group definition
• an include "pathname" statement includes group definitions from a file pathname

Options

These command options can be used in the usual way:

-0, --zero:              show 0 exit status, too
-?, --help:              show help on options and usage
-e, --ssh ARG:           ssh command to use (str arg, default '/usr/bin/ssh')
-f, --groups-file PATH:  config file(s) with host group definitions
                           (str arg, default '/opt/w21/etc/on-groups')
-g, --groups:            list defined host group names and exit
-G, --groupdefs PATTERN: list expanded group definitions and exit
                           (str arg, default None)
-h, --help:              show help on options and usage
-H, --print-hosts:       print hosts to run on and exit
-q, --quiet:             be quiet (no output except error messages)
-s, --sort:              sort output by host (waits for all to finish)
-S, --search NAME:       search host in groups, print where found, and exit
                           (str arg, default None)
-t, --threads ARG:       maximum number of threads running in parallel
                           (int arg, default 20)
-T, --no-title:          don't print per-host or per-group title
-v, --verbose:           increase verbosity (up to three times)

For everything else, see figure 1.