High availability overview

Enabling high availability for Puppet Enterprise ensures that your system remains operational even if certain infrastructure components become unreachable.

High availability configuration creates a replica of your primary master. The primary master is your Puppet master or, if your installation includes compile masters, your master of masters. The replica node is called the primary master replica. You may only have one provisioned replica at a time.

Note: In this release, we don’t support high availability for split installations – where the master, console, and PuppetDB components are installed on separate machines. Only monolithic installations - with or without compile masters and a master of masters - is supported.

There are two main advantages to enabling high availability:

  • If your primary master fails, the replica takes over, continuing to perform critical operations. For details, see What happens during failovers.
  • If your primary master can’t be repaired, you can promote the replica to master. Promotion establishes the replica as the new, permanent master.

High availability architecture

The primary master replica is not an exact copy of the primary master. Rather, the replica duplicates specific infrastructure components and services. Hiera data and other custom configurations aren’t replicated.

Replication may be read-write, meaning that data can be written to the service or component on either the master or the replica, and the data is synced to both nodes. Alternatively, replication may be read-only, where data is written only to the master and synced to the replica. Some components and services, like Puppet server and the Console services UI, aren’t replicated because they contain no native data.

Some components and services are activated immediately when you enable a replica; others aren’t active until you promote a replica.

Component or service Type of replication Activated when replica is…
Puppet server none enabled
File sync client read-only enabled
PuppetDB read-write enabled
Certificate authority read-only promoted
RBAC service read-only enabled
Node classifier service read-only enabled
Activity service read-only enabled
Orchestration service read-only promoted
Console service UI none promoted

For details about PE components and services, see the PE architecture overview.

Monolithic HA configuration with a single master

Monolithic HA configuration with a single master

In a monolithic installation, when Puppet runs fail over, agents communicate with the replica instead of the master.

Monolithic HA configuration with compile masters

Monolithic HA configuration with compile masters

In a monolithic installation with compile masters, agents communicate with load balancers or compile masters, which in turn communicate with the master or replica.

Regardless of your load balancer configuration, all Puppet agents use the master and replica directly for PCP connections, which ensures that you can control agents using orchestrator even if the master fails.

What happens during failovers

Failover occurs when the replica takes over services usually performed by the master.

Failover is automatic, meaning that you don’t have to take action to activate the replica. With high availability enabled, Puppet runs are directed first to the master. If the primary master is either fully or partially unreachable, Puppet runs are directed to the replica.

In partial failovers, Puppet runs may use the server, node classifier, or PuppetDB on the replica if those services aren’t reachable on the master. For example, if the master’s node classifier fails, but its Puppet server is still running, agent runs use the Puppet server on the master but fail over to the replica’s node classifier.

What works during failovers:

  • Scheduled Puppet runs
  • Catalog compilation
  • Viewing classification data using the node classifier API
  • Reporting and queries based on PuppetDB data

What doesn’t work during failovers:

  • Deploying new Puppet code
  • Editing node classifier data
  • Using the console
  • Certificate functionality, including provisioning new agents, revoking certificates, or running the puppet certificate command
  • Most CLI tools
  • Application Orchestration

System and software requirements

Your Puppet infrastructure must meet specific requirements in order to configure high availability.

Component Requirement
Operating system
  • RedHat 7
  • CentOS 7
  • Ubuntu 16.04
Software
  • You must use Code Manager so that code is deployed to both the master and the replica after you enable a replica.
  • Orchestrator must be enabled so that agents are updated when you provision or enable a replica. Orchestrator is enabled by default.
Replica
  • Must be a Puppet agent node that doesn’t have a specific function already. For example, you can’t provision an existing master or compile master as a replica.
  • Must have the same hardware specifications and capabilities as your primary master node.
Firewall Both the master and the replica must comply with these port requirements:
Node names
  • You must use resolvable domain names when specifying node name for the master and replica.
RBAC tokens
  • You must have an admin RBAC token when running puppet infrastructure commands, including provision, enable, and forget. You can generate a token using the puppet-access command.
  • Note: You don’t need an RBAC token to promote a replica.

Classification changes in high availability installations

When you provision and enable a replica, the system makes a number of classification changes in order to manage high availability without affecting existing configuration.

These preconfigured node groups are added in high availability installations:

Node group Matching nodes Inherits from
PE HA Master primary master PE master node group
PE HA Replica primary master replicas PE infrastructure node group

These parameters are used to configure high availability installations:

Parameter Purpose Node group Classes Notes
agent-server-urls Specifies the list of servers that Puppet agents contact, in order.

PE Agent

PE Infrastructure Agent

puppet_enterprise::
profile::agent

In monolithic installations with compile masters, agents must be configured to communicate with the load balancers or compile masters.

Important: Setting agents to communicate directly with the replica in order to use the replica as a compile master is not a supported configuration.

replication_mode Sets replication type and direction on masters and replicas.

PE master (none)

HA master (source)

HA replica (replica)

puppet_enterprise::
profile::master

puppet_enterprise::
profile::database

puppet_enterprise::
profile::console

System parameter. Don’t modify.
ha_enabled_replicas Tracks replica nodes that are failover ready. PE Infrastructure puppet_enterprise

System parameter. Don't modify.

Updated when you enable a replica.

Load balancer timeout in high availability installations

High availability configuration uses timeouts to determine when to fail over to the replica. If the load balancer timeout is shorter than the server and agent timeout, connections from agents might be terminated during failover.

To avoid timeouts, set the timeout option for load balancers to four minutes or longer. This duration allows compile masters enough time for required queries to PuppetDB and the node classifier service. You can set the load balancer timeout option using parameters in the haproxy or f5 modules.


↑ Back to top