SSL: Regenerating all Certificates in a Puppet deployment
Included in Puppet Enterprise 2017.2.
Note: If you’re visiting this page to remediate your Puppet Enterprise deployment due to CVE-2014-0160, a.k.a. “Heartbleed,” please see this announcement for additional information and links to more resources before using this guide. Before applying these instructions, please bear in mind that this is a non-trivial operation that contains some manual steps and will require you to replace certificates on every agent node managed by your Puppet master.
IMPORTANT: This page describes the steps for regenerating certs in an open source Puppet deployment. If you use Puppet Enterprise you should NOT use this page as doing so will leave you with an incomplete replacement and non-functional deployment. Instead, PE customers must refer to one of the following pages:
In some cases, you might need to regenerate the certificates and security credentials (private and public keys) that are generated by Puppet’s built-in certificate authority (CA). For example, you might have a Puppet master you need to move to a different network in your infrastructure, or you might have experienced an unforeseen security vulnerability that makes existing credentials untrustworthy.
Regardless of your situation, regenerating your certs involves the following three steps (complete procedures follow below):
- On your master, you’ll clear the certs and security credentials, regenerate the CA, and then regenerate the certs and security credentials.
- Next, you’ll clear and regenerate certs and security credentials for any extensions.
- Lastly, you’ll clear and regenerate certs and security credentials for all agent nodes.
Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the
puppet cert clean command on your Puppet master and then follow step 3 for any agents that need to be replaced.
Step 1: Clear and regenerate certs on your Puppet master
On the Puppet master hosting the CA:
- Back up the SSL directory, which should be
/etc/puppetlabs/puppet/ssl/. If something goes wrong, you might need to restore this directory so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should get some assistance as soon as possible so you can keep your site secure.
- Stop the Puppet agent service with
sudo puppet resource service puppet ensure=stopped.
- Stop the Puppet master service. For Puppet Server, use
sudo puppet resource service puppetserver ensure=stopped. For a Rack-based master, stop whatever web server you’re using.
- Delete the SSL directory with
sudo rm -r /etc/puppetlabs/puppet/ssl.
Regenerate the CA by running
sudo puppet cert list -a. You should see this message:
Notice: Signed certificate request for ca.
- Generate the Puppet master’s new certs with
sudo puppet master --no-daemonize --verbose.
- When you see
Notice: Starting Puppet master <your Puppet version>, type CTRL + C.
- Start the Puppet master service by reversing whatever you did in step 3. (E.g.,
sudo puppet resource service puppetserver ensure=running.)
- Start the Puppet agent service with
sudo puppet resource service puppet ensure=running.
At this point:
- You have a brand new CA certificate and key.
- Your Puppet master has a certificate from the new CA, and it can once again field new certificate requests.
- The Puppet master will reject any requests for configuration catalogs from nodes that haven’t replaced their certificates (which, at this point, will be all of them except itself).
- If you are using any extensions that rely on Puppet certificates, like PuppetDB or MCollective, the Puppet master won’t be able to communicate with them. Consequently, it might not be able to serve catalogs, even to agents that do have new certificates.
Step 2: Clear and regenerate certs for any extensions
You might be using an extension, like PuppetDB or MCollective, to enhance Puppet. These extensions probably use certificates from Puppet’s CA in order to communicate securely with the Puppet master.
For each extension like this, you’ll need to regenerate the certificate(s) it uses. Many tools have scripts or documentation to help you set up SSL, and you can often just re-run the setup instructions.
- PuppetDB users should first follow the instructions below on regenerating agent certificates, since PuppetDB re-uses Puppet agents’ certificates. After that, restart the PuppetDB service.
- MCollective often uses SSL certificates from Puppet’s CA. If you are replacing your Puppet CA and are using the same certs for MCollective, you should go through the standard deployment guide and re-do any steps involving security credentials. You’ll generally need to replace client certificates, your server keypair, and the ActiveMQ server’s keystore and truststore.
Step 3: Clear and regenerate certs for Puppet agents
To replace the certs on agents, you’ll need to log into each agent node and do the following:
- Stop the Puppet agent service. On *nix nodes, run
sudo puppet resource service puppet ensure=stopped. On Windows nodes, run the same command (minus
sudo) with Administrator privileges.
- Locate Puppet’s SSL directory and delete everything in it. The SSL directory should be at
Re-start the Puppet agent service. On *nix nodes, run
sudo puppet resource service puppet ensure=running. On Windows nodes, run the same command (minus
sudo) with Administrator privileges.
Once the Puppet agent starts, it will automatically generate keys and request a new certificate from the CA Puppet master.
- If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this by logging into the CA Puppet master server, running
sudo puppet cert listto see pending requests, and running
sudo puppet cert sign <NAME>to sign requests.
Once an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run.
Once you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.