Invoking Orchestration Actions
A newer version is available; see the version menu above for details.
About This Page
Puppet Enterprise (PE) has two ways to invoke orchestration actions:
- The live management page of the PE console
- The Linux command line on the puppet master server
This page covers only the command line. See the Navigating Live Management page of this manual for instructions on using live management to invoke actions.
Note: Although you will be running these commands on the Linux command line, they can invoke orchestration actions on both *nix and Windows machines.
Puppet Enterprise’s orchestration engine, MCollective, has its own section of the documentation site, which includes more complete details and examples for command line orchestration usage.
This page covers basic CLI usage and all PE-specific information; for more details, see the following pages from the MCollective docs:
Logging In as
To run orchestration commands, you must log in to the puppet master server as the special
peadmin user account, which is created during installation.
Note: Puppet Enterprise 3.0 does not support adding more orchestration user accounts.
This means that, while it is possible (albeit complex) to allow other accounts on other machines to invoke orchestration actions, upgrading to a future version of PE may disable access for these extra accounts, requiring you to re-enable them manually. We do not provide instructions for enabling extra orchestration accounts.
By default, the
peadmin account cannot log in with a password. We recommend two ways to log in:
Anyone able to log into the puppet master server as an admin user with full root
sudo privileges can become the
peadmin user by running:
$ sudo -i -u peadmin
This is the default way to log in as the
peadmin user. It means that orchestration commands can only be issued by the group of users who can fully control the puppet master.
Adding SSH Keys
If you wish to allow other users to run orchestration commands without giving them full control over the puppet master, you can add their public SSH keys to
peadmin’s authorized keys file.
You can use Puppet’s
ssh_authorized_key resource type to do this, or add keys manually to the
All orchestration actions are invoked with the
mco executable. The
mco command always requires a subcommand to invoke actions.
Note: For security, the
mcocommand relies on a config file (
/var/lib/peadmin/.mcollective) which is only readable by the
peadminuser. PE automatically configures this file; it usually shouldn’t be modified by users.
mco command has several subcommands, and it’s possible to add more — run
mco help for a list of all available subcommands. The default subcommands in Puppet Enterprise 3.0 are:
This is the general purpose orchestration client, which can invoke actions from any MCollective agent plugin.
These subcommands only invoke certain kinds of actions, but have some extra UI enhancements to make them easier to use than the equivalent
mco rpc command.
Help and support subcommands:
These subcommands can display information about the available agent plugins and subcommands.
help— displays help for subcommands.
mco plugin doccommand can display help for agent plugins.
completion— a helper for shell completion systems.
Inventory and reporting subcommands:
These subcommands can retrieve and summarize information from Puppet Enterprise agent nodes.
ping— pings all matching nodes and reports on response times
facts— displays a summary of values for a single fact across all systems
inventory— general reporting tool for nodes, collectives and subcollectives
find— like ping, but doesn’t report response times
Getting Help on the Command Line
You can get information about subcommands, actions, and other plugins on the command line.
Use one of the following commands to get help for a specific subcommand:
$ mco help <SUBCOMMAND> $ mco <SUBCOMMAND> --help
List of Plugins
To get a list of the available plugins, which includes MCollective agent plugins, data query plugins, discovery methods, and validator plugins, run
mco plugin doc.
Agent Plugin Help
Related orchestration actions are bundled together in MCollective agent plugins. (Puppet-related actions are all in the
puppet plugin, etc.)
To get detailed info on a given plugin’s actions and their required inputs, run:
$ mco plugin doc <PLUGIN>
If there is also a data plugin with the same name, you may need to prepend
agent/ to the plugin name to disambiguate:
$ mco plugin doc agent/<PLUGIN>
Orchestration actions are invoked with either the general purpose
rpc subcommand or one of the special-purpose subcommands. Note that unless you specify a filter, orchestration commands will be run on every server in your Puppet Enterprise deployment; make sure you know what will happen before confirming any potentially disruptive commands. For more info on filters, see “Filtering Actions” below.
The most useful subcommand is
mco rpc. This is the general purpose orchestration client, which can invoke actions from any MCollective agent plugin. See “List of Built-In Actions” for more information about agent plugins.
$ mco rpc service restart service=httpd
The general form of an
mco rpc command is:
$ mco rpc <AGENT PLUGIN> <ACTION> <INPUT>=<VALUE>
mco rpc can invoke any action, sometimes a special-purpose application can provide a more convenient interface.
$ mco puppet runall 5
runallaction is able to run many nodes without exceeding a certain load of concurrent runs. It does this by repeatedly invoking the
statusaction, and only sending a
runonceaction to the next node if there’s enough room in the concurrency limit.
This uses the same actions that the
mco rpccommand can invoke, but since
rpcdoesn’t know that the output of the
statusaction is relevant to the timing of the
runonceaction, it can’t provide that improved UI.
Each special-purpose subcommand (
package) has its own CLI syntax. For example,
mco service puts the name of the service before the action, to mimic the format of the more common platform-specific service commands:
$ mco service httpd status
mco help <SUBCOMMAND> to get specific help for each subcommand.
By default, orchestration actions affect all PE nodes. You can limit any action to a smaller set of nodes by specifying a filter.
$ mco service pe-httpd status --with-fact fact_is_puppetconsole=true
Note: For more details about filters, see the following pages from the MCollective docs:
All command line orchestration actions can accept the same filter options, which are listed under the “Host Filters” section of any
mco help <SUBCOMMAND> text:
Host Filters -W, --with FILTER Combined classes and facts filter -S, --select FILTER Compound filter combining facts and classes -F, --wf, --with-fact fact=val Match hosts with a certain fact -C, --wc, --with-class CLASS Match hosts with a certain config management class -A, --wa, --with-agent AGENT Match hosts with a certain agent -I, --wi, --with-identity IDENT Match hosts with a certain configured identity
Each type of filter lets you specify a type of metadata and a desired value. The orchestration action will only run on nodes where that data has that desired value.
Any number of fact, class, and agent filters can also be combined in a single command; this will make it so nodes must match every filter to run the action.
Matching Strings and Regular Expressions
Filter values are usually simple strings. These must match exactly and are case-sensitive.
Most filters can also accept regular expressions as their values; these are surrounded by forward slashes, and are interpreted as standard Ruby regular expressions. (You can even turn on various options for a subpattern, such as case insensitivity —
-F "osfamily=/(?i:redhat)/".) Unlike plain strings, they accept partial matches.
Filtering by Identity
A node’s “identity” is the same as its Puppet certname, as specified during installation. Identities will almost always be unique per node.
$ mco puppet runonce -I web3balancer.example.com
- You can use the
--with-identityoption multiple times to create a filter that matches multiple specific nodes.
- You cannot combine the identity filter with other filter types.
- The identity filter accepts regular expressions.
Filtering by Fact, Class, and Agent
- Facts are the standard Puppet Enterprise facts, which are available in your Puppet manifests and can be viewed as inventory information in the PE console. A list of the core facts is available here. Use the
--with-factoption with a
fact=valuepair to filter on facts.
- Classes are the Puppet classes that are assigned to a node. This includes classes assigned in the console, assigned via Hiera, declared in
site.pp, or declared indirectly by another class. Use the
--with-classoption with a class name to filter on classes.
- Agents are MCollective agent plugins. Puppet Enterprise’s default plugins are available on every node, so filtering by agent makes more sense if you are distributing custom plugins to only a subset of your nodes. For example, if you made an emergency change to a custom plugin that you distribute with Puppet, you could filter by agent to trigger an immediate Puppet run on all affected systems. (
mco puppet runall 5 -A my_agent) Use the
--with-agentoption to filter on agents.
Since mixing classes and facts is so common, you can also use the
--with option to supply a mixture of class names and
Compound “Select” Filters
--select option accepts arbitrarily complex filters. Like
-W, it can accept a mixture of class names and
fact=value pairs, but it has two extra tricks:
-W filter always combines facts and classes with “and” logic — nodes must match all of the criteria to match the filter.
-S filter lets you combine values with nested Boolean “and”/”or”/”not” logic:
$ mco service httpd restart -S "((customer=acme and osfamily=RedHat) or domain=acme.com) and /apache/"
In addition, the
-S filter lets you use data plugin queries as an additional kind of metadata.
Data plugins can be tricky, but are very powerful. To use them effectively, you must:
- Check the list of data plugins with
mco plugin doc.
- Read the help for the data plugin you want to use, with
mco plugin doc data/<NAME>. Note any required input and the available outputs.
get_dataaction on a single node to check the format of the output you’re interested in. This action requires
source(the plugin name) and
query(the input) arguments:
$ mco rpc rpcutil get_data source="fstat" query="/etc/hosts" -I web01
This will show all of the outputs for that plugin and input on that node.
- Construct a query fragment of the format
<PLUGIN>('<INPUT>').<OUTPUT>=<VALUE>— note the parentheses, the fact that the input must be in quotes, the
.outputnotation, and the equals sign. Make sure the value you’re searching for matches the expected format, which you saw when you did your test query.
Use that fragment as part of a
$ mco find -S "fstat('/etc/hosts').md5=/baa3772104/ and osfamily=RedHat"
You can specify multiple data plugin query fragments per
Testing Filters With
Before invoking any potentially disruptive action, like a service restart, you should test the filter with
mco find or
mco ping, to make sure your command will act on the nodes you expect.
Batching and Limiting Actions
By default, orchestration actions run simultaneously on all of the targeted nodes. This is fast and powerful, but is sometimes not what you want:
- Sometimes you want the option to cancel out of an action with control-C before all nodes have run it.
- Sometimes, like when retrieving inventory data, you want to run a command on just a sample of nodes and don’t need to see the results from everything that matches the filter.
- Certain actions may consume limited capacity on a shared resource (such as the puppet master server), and invoking them on a “thundering herd” of nodes can disrupt that resource.
In these cases, you can batch actions, to run all of the matching nodes in a controlled series, or limit them, to run only a subset of the matching nodes.
- Use the
--batch <SIZE>option to invoke an orchestration action on only
<SIZE>nodes at once. PE will invoke it on the first
<SIZE>nodes, wait briefly, invoke it on the next batch, and so on.
- Use the
--batch-sleep <SECONDS>option to control how long PE should sleep between batches.
- Use the
--limit <COUNT>option to invoke an action on only
<COUNT>can be an absolute number or a percentage. The nodes will be chosen randomly.
- Use the
--oneoption to invoke an action on just one matching node, chosen randomly.