SSL: Regenerating All Certificates in a Puppet Deployment
Included in Puppet Enterprise 3.7. A newer version is available; see the version menu above for details.
Note: If you’re visiting this page to remediate your Puppet Enterprise deployment due to CVE-2014-0160, a.k.a. “Heartbleed,” please see this announcement for additional information and links to more resources before using this guide. Before applying these instructions, please bear in mind that this is a non-trivial operation that contains some manual steps and will require you to replace certificates on every agent node managed by your Puppet master.
IMPORTANT: This page describes the steps for regenerating certs in an open source Puppet deployment. If you use Puppet Enterprise you should NOT use this page since doing so will leave you with an incomplete replacement and non-functional deployment. Instead, PE customers must refer to one of the following pages:
In some cases, you may find that you need to regenerate the certificates and security credentials (private and public keys) that are generated by Puppet’s built-in certificate authority (CA). For example, you may have a Puppet master you need to move to a different network in your infrastructure, or you may find that you need to regenerate all the certificates and security credentials in your infrastructure due to an unforeseen security vulnerability.
Regardless of your situation, regenerating your certs involves the following three steps (complete procedures follow below):
- On your master, you’ll clear the certs and security credentials, regenerate the CA, and then regenerate the certs and security credentials.
- Next, you’ll clear and regenerate certs and security credentials for any extensions.
- Lastly, you’ll clear and regenerate certs and security credentials for all agent nodes.
Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the
puppet cert clean command on your Puppet master and then follow step 3 for any agents that need to be replaced.
Step 1: Clear and Regenerate Certs on Your Puppet Master
On the Puppet master hosting the CA:
- Back up the
/etc/puppetlabs/puppet/ssl/directory. If something goes wrong, you may need to restore this directory so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should get some assistance as soon as possible so you can keep your site secure.
- Stop the Puppet agent service with
sudo puppet resource service puppet ensure=stopped.
- Stop the Puppet master service. Your Puppet master is probably managed by a Rack-supporting web server, such as Apache with Passenger; you’ll need to stop that web server. (For example,
sudo puppet resource service apache2 ensure=stopped.)
- Locate Puppet’s ssldir by [reading the setting from Puppet][config_print]; run either
sudo puppet config print ssldir --section masteror
sudo puppet master --configprint ssldir.
- Delete all files in the ssldir with
sudo rm -r <PATH TO SSLDIR>.
Regenerate the CA by running
sudo puppet cert list -a. You should see this message:
Notice: Signed certificate request for ca.
- Generate the Puppet master’s new certs with
sudo puppet master --no-daemonize --verbose.
- When you see
Notice: Starting Puppet master <your Puppet version>, type CTRL + C.
- Start the Puppet master service by reversing whatever you did in step 3. (E.g.,
sudo puppet resource service apache2 ensure=running.)
- Start the Puppet agent service with
sudo puppet resource service puppet ensure=running.
At this point:
- You have a brand new CA certificate and key.
- Your Puppet master has a certificate from the new CA, and it can once again field new certificate requests.
- The Puppet master will reject any requests for configuration catalogs from nodes that haven’t replaced their certificates (which, at this point, will be all of them except itself).
- If you are using any extensions that rely on Puppet certificates, like PuppetDB or Puppet Dashboard, the Puppet master won’t be able to communicate with them. Consequently, it may not be able to serve catalogs, even to agents that do have new certificates.
Step 2: Clear and Regenerate Certs for any Extensions
You may be using extensions like PuppetDB or Puppet Dashboard to enhance Puppet. These extensions probably use certificates from Puppet’s CA in order to communicate securely with the Puppet master.
For each extension like this, you’ll need to regenerate the certificate(s) it uses. Many tools have scripts or documentation to help you set up SSL, and you can often just re-run the setup instructions.
- PuppetDB users should first follow the instructions below on regenerating agent certificates, since PuppetDB re-uses Puppet agents’ certificates. After that, run PuppetDB’s ssl configuration script to regenerate its SSL configuration, and restart the PuppetDB service.
- Puppet Dashboard can use certificates for its inventory service. Repeat the SSL config instructions.
- MCollective often uses SSL certificates from Puppet’s CA. If you are replacing your Puppet CA and are using the same certs for MCollective, you should go through the standard deployment guide and re-do any steps involving security credentials. You’ll generally need to replace client certificates, your server keypair, and the ActiveMQ server’s keystore and truststore.
Step 3: Clear and Regenerate Certs for Puppet Agents
To replace the certs on agents, you’ll need to log into each agent node and do the following:
- Stop the Puppet agent service. On *nix nodes, run
sudo puppet resource service puppet ensure=stopped. On Windows nodes, run the same command (minus
sudo) with Administrator privileges.
- Locate Puppet’s ssldir and delete everything in it. The ssldir can be located by running
puppet config print ssldir --section agentor
puppet agent --configprint ssldir— be sure to run that command with
sudoon *nix nodes and with Administrator privileges on Windows nodes.
Re-start the Puppet agent service. On *nix nodes, run
sudo puppet resource service pe-puppet ensure=running. On Windows nodes, run the same command (minus
sudo) with Administrator privileges.
Once the Puppet agent starts, it will automatically generate keys and request a new certificate from the CA Puppet master.
- If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this by logging into the CA Puppet master server, running
sudo puppet cert listto see pending requests, and running
sudo puppet cert sign <NAME>to sign requests.
Once an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run.
Once you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.