Regenerating Certs and Security Credentials in Monolithic Puppet Enterprise Deployments

A newer version is available; see the version menu above for details.

Important: The procedures in this document apply to 3.2.x versions of Puppet Enterprise. Any issues in PE due to CVE-2014-0160, a.k.a. “Heartbleed,” have been fixed in PE 3.7.0. PE 3.7.0 users should not perform these steps as certificate functionality has changed between versions.

Note: If you’re visiting this page to remediate your Puppet Enterprise deployment due to CVE-2014-0160, a.k.a. “Heartbleed,” please see this announcement for additional information and links to more resources before using this guide. Before applying these instructions, please bear in mind that this is a non-trivial operation that contains some manual steps and will require you to replace certificates on every agent node managed by your puppet master.

Note: This page explains how to regenerate all certificates in a monolithic PE deployment — that is, where the puppet master, PuppetDB, and PE console roles are all installed on the same server. See this page for instructions on regenerating certificates in a split PE deployment.

Overview

In some cases, you may find that you need to regenerate the certificates and security credentials (private and public keys) generated by PE’s built-in certificate authority (CA). For example, you may have a puppet master that you need to move to a different network in your infrastructure, or you may find that you need to regenerate all the certificates and security credentials in your infrastructure due to an unforeseen security vulnerability.

Regardless of your situation, regenerating your certificatess involves the following four steps (complete procedures follow below):

  1. On your master, you’ll clear the certs and security credentials, regenerate the CA, and then regenerate the certs and security credentials.
  2. Next, you’ll clear and regenerate certs and security credentials for PuppetDB.
  3. Then, you’ll clear and regenerate certs and security credentials for the PE console.
  4. Lastly, you’ll clear and regenerate certs and security credentials for all agent nodes.

Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the puppet cert clean command on your puppet master and then follow step four for any agent certs that need to be replaced.

Step 1: Clear and Regenerate Certificates on Your Puppet Master

On your monolithic puppet master:

  1. Back up the /etc/puppetlabs/puppet/ssl/, /etc/puppetlabs/puppetdb/ssl/, and /opt/puppet/share/puppet-dashboard/certs directories. If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet Labs support as soon as you restore service so we can help you secure your site.
  2. Stop the puppet agent service with sudo puppet resource service pe-puppet ensure=stopped.
  3. Stop the orchestration service with sudo puppet resource service pe-mcollective ensure=stopped.
  4. Stop the puppet master service with sudo puppet resource service pe-httpd ensure=stopped.
  5. Clear all certs from your master with sudo rm -r /etc/puppetlabs/puppet/ssl/*.
  6. Regenerate the CA by running sudo puppet cert list -a. You should see this message: Notice: Signed certificate request for ca.
  7. Generate the puppet master’s new certs with sudo puppet master --no-daemonize --verbose.
  8. When you see Notice: Starting Puppet master <your Puppet and PE versions>, type CTRL + C.
  9. Start the puppet master service with sudo puppet resource service pe-httpd ensure=running.
  10. Start the puppet agent service with sudo puppet resource service pe-puppet ensure=running.

At this point:

  • You have a brand new CA certificate and key.
  • Your puppet master has a certificate from the new CA, and it can once again field new certificate requests.
  • The puppet master will reject any requests for configuration catalogs from nodes that haven’t replaced their certificates (which, at this point, will be all of them except the master).
  • The puppet master can’t serve catalogs even to agents that do have new certificates, since it can’t communicate with the console and PuppetDB.
  • Orchestration and live management are down.

Step 2: Clear and Regenerate Certs for PuppetDB

On your monolithic puppet master:

  1. Stop the PuppetDB service with sudo puppet resource service pe-puppetdb ensure=stopped.
  2. Clear the certs and security credentials from the PuppetDB SSL directory with sudo rm -r /etc/puppetlabs/puppetdb/ssl/*.
  3. Regenerate the certs and security credentials for PuppetDB with sudo /opt/puppet/sbin/puppetdb-ssl-setup -f.
  4. Start the PuppetDB service with sudo puppet resource service pe-puppetdb ensure=running.

At this point:

  • The puppet master can talk to PuppetDB again.
  • The puppet master can’t serve catalogs to agents yet, since it still won’t trust the console service.
  • Orchestration and live management are still down.

Step 3: Clear and Regenerate Certs for the PE Console

On your monolithic puppet master:

  1. Navigate to the console certs directory with sudo cd /opt/puppet/share/puppet-dashboard/certs. Stay in this directory for the following steps.
  2. Remove all the credentials in this directory with sudo rm -r /opt/puppet/share/puppet-dashboard/certs/*.
  3. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:create_key_pair.
  4. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:request. The cert will be generated, and a CSR submitted.
  5. Use puppet cert sign pe-internal-dashboard to sign the console certificate request.
  6. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:retrieve.
  7. Ensure the console can access the new credentials with sudo chown -R puppet-dashboard:puppet-dashboard /opt/puppet/share/puppet-dashboard/certs.
  8. Restart the console service with sudo service pe-httpd restart.

At this point:

  • The puppet master can talk to the console again, and vice versa.
  • The puppet master can now serve catalogs to agents.

    However, it will only trust agents that have replaced their certificates. The only agent that has replaced its certificate at this point is the monolithic puppet master.

  • The console is usable, but because its SSL certificate has been replaced, your web browser may notice the change, assume it results from a malicious attack, and refuse to allow you access. If this happens, you may need to delete the old cert from your browser’s collection of cached certificates. Details of this process are beyond the scope of this guide and will vary by browser and platform. (You can delay having to figure this out by temporarily using a different browser.)
  • Orchestration and live management may not immediately work, but they will start working again as soon as both the puppet master server and the console node complete a puppet agent run. (The certificates used by MCollective and the ActiveMQ service are completely managed by Puppet and don’t have to be manually regenerated.)

On the monolithic puppet master, you can now start a successful agent run with sudo puppet agent -t.

Step 4: Clear and Regenerate Certs for PE Agents

To replace the certs on agents, you’ll need to log into each agent node and do the following:

  1. Stop the puppet agent service. On *nix nodes, run sudo puppet resource service pe-puppet ensure=stopped. On Windows nodes, run the same command (minus sudo) with Administrator privileges.
  2. Stop the orchestration service. On *nix nodes, run sudo puppet resource service pe-mcollective ensure=stopped. On Windows nodes, run the same command (minus sudo) with Administrator privileges.
  3. Delete the agent’s SSL directory. On *nix nodes, run sudo rm -r /etc/puppetlabs/puppet/ssl/*. On Windows nodes, delete the $confdir\ssl directory, using the Administrator confdir. See here for more information on locating the confdir.
  4. Re-start the puppet agent service. On *nix nodes, run sudo puppet resource service pe-puppet ensure=running. On Windows nodes, run the same command (minus sudo) with Administrator privileges.

    Once puppet agent starts, it will automatically generate keys and request a new certificate from the CA puppet master.

  5. If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this with the PE console’s request manager, or by logging into the CA puppet master server, running sudo puppet cert list to see pending requests, and running sudo puppet cert sign <NAME> to sign requests.

Once an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run. After a node has fetched its new certificate and completed a full Puppet run, it will once again appear in orchestration and live management. If, after waiting for a short time, you don’t see the agent node in live management, use NTP to make sure time is in sync aross your PE deployment. On Windows nodes, you may need to log into the node and check the status of the Marionette Collective service — sometimes it can hang while attempting to stop or restart.

Once you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.

↑ Back to top