Regenerating Certs and Security Credentials in Split Puppet Enterprise Deployments

A newer version is available; see the version menu above for details.

Important: The procedures in this document apply to 3.2.x versions of Puppet Enterprise. Any issues in PE due to CVE-2014-0160, a.k.a. “Heartbleed,” have been fixed in PE 3.7.0. PE 3.7.0 users should not perform these steps as certificate functionality has changed between versions.

Note: If you’re visiting this page to remediate your Puppet Enterprise deployment due to CVE-2014-0160, a.k.a. “Heartbleed,” please see this announcement for additional information and links to more resources before using this guide. Before applying these instructions, please bear in mind that this is a non-trivial operation that contains some manual steps and will require you to replace certificates on every agent node managed by your puppet master.

Note: This page explains how to regenerate all certificates in a split PE deployment — that is, where the puppet master, PuppetDB, and PE console roles are all installed on separate servers. See this page for instructions on regenerating certificates in a monolithic PE deployment.

Overview

In some cases, you may find that you need to regenerate the SSL certificates and security credentials (private and public keys) that are generated by PE’s built-in certificate authority (CA). For example, you may have a puppet master you need to move to a different network in your infrastructure, or you may find you need to regenerate all the certificates and security credentials in your infrastructure due to an unforeseen security vulnerability.

Regardless of your situation, regenerating your certs involves the following four steps (complete procedures follow below):

  1. On your master, you’ll clear the certs and security credentials, regenerate the CA, and then regenerate the certs and security credentials.
  2. Next, you’ll clear and regenerate certs and security credentials for PuppetDB.
  3. Then, you’ll clear and regenerate certs and security credentials for the PE console
  4. Lastly, you’ll clear and regenerate certs and security credentials for all agent nodes.

Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the puppet cert clean command on your puppet master and then follow step four for any agents that need to be replaced.

Step 1: Clear and Regenerate Certs on Your Puppet Master

On your puppet master:

  1. Back up the /etc/puppetlabs/puppet/ssl/ directory. If something goes wrong, you may need to restore this directory so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet Labs support as soon as you restore service, so we can help you secure your site.
  2. Stop the puppet agent service with sudo puppet resource service pe-puppet ensure=stopped.
  3. Stop the orchestration service with sudo puppet resource service pe-mcollective ensure=stopped.
  4. Stop the puppet master service with sudo puppet resource service pe-httpd ensure=stopped.
  5. Clear all certs from your master with sudo rm -r /etc/puppetlabs/puppet/ssl/*.
  6. Regenerate the CA by running sudo puppet cert list -a. You should see this message: Notice: Signed certificate request for ca.
  7. Generate the puppet master’s new certs with sudo puppet master --no-daemonize --verbose.
  8. When you see Notice: Starting Puppet master <your Puppet and PE versions>, type CTRL + C.
  9. Start the puppet master service with sudo puppet resource service pe-httpd ensure=running.
  10. Start the puppet agent service with sudo puppet resource service pe-puppet ensure=running.

At this point:

  • You have a brand new CA certificate and key.
  • Your puppet master has a certificate from the new CA, and it can once again field new certificate requests.
  • The puppet master will reject any requests for configuration catalogs from nodes that haven’t replaced their certificates (which, at this point, will be all of them except the master).
  • The puppet master can’t serve catalogs even to agents that do have new certificates, since it can’t communicate with the console and PuppetDB.
  • Orchestration and live management are down.

Step 2: Clear and Regenerate Certs for PuppetDB

On your PuppetDB server:

  1. Back up the /etc/puppetlabs/puppet/ssl/ and /etc/puppetlabs/puppetdb/ssl/ directories. If something goes wrong, you may need to reinstate these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet Labs support as soon as you restore service, so we can help you secure your site.
  2. Stop the puppet agent service with sudo puppet resource service pe-puppet ensure=stopped.
  3. Stop the orchestration service with sudo puppet resource service pe-mcollective ensure=stopped.
  4. Stop the PuppetDB service with sudo puppet resource service pe-puppetdb ensure=stopped.
  5. Delete puppet agent’s SSL credentials with sudo rm -r /etc/puppetlabs/puppet/ssl/*.
  6. Delete the SSL credentials from the PuppetDB SSL directory with sudo rm -r /etc/puppetlabs/puppetdb/ssl/*.
  7. Request an agent certificate from the CA puppet master with sudo puppet agent -t. The puppet master will autosign the request, and the puppet agent will fetch the certificate.

    The Puppet agent will also try to do a full Puppet run, which will fail. This is as expected, so don’t worry about it.

    If the master doesn’t autosign the certificate in this step, you may have changed its autosign configuration. You’ll need to manually sign the certificate (see below).

  8. Regenerate the certs and security credentials for PuppetDB with sudo /opt/puppet/sbin/puppetdb-ssl-setup -f.
  9. Start the PuppetDB service with sudo puppet resource service pe-puppetdb ensure=running.
  10. Re-start the puppet agent service with sudo puppet resource service pe-puppet ensure=running.

Note: If you are using your own PostgreSQL database on a different server and have encrypted communications with it using SSL, or if you have encrypted communication between database instances for replication, you should consider regenerating your certificates and keys. For more information on doing this, see the PostgreSQL documentation.

At this point:

  • The PuppetDB server is now completely taken care of.
  • The puppet master can talk to PuppetDB again.
  • The puppet master can’t serve catalogs to agents yet, since it still won’t trust the console server.
  • Orchestration and live management are still down.

Step 3: Clear and Regenerate Certs for the PE Console

On your console server:

  1. Back up the /etc/puppetlabs/puppet/ssl/ and /opt/puppet/share/puppet-dashboard/certs directories. If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet Labs support as soon as you restore service, so we can help you secure your site.
  2. Stop the puppet agent service with sudo puppet resource service pe-puppet ensure=stopped.
  3. Stop the orchestration service with sudo puppet resource service pe-mcollective ensure=stopped.
  4. Stop the console service with sudo puppet resource service pe-httpd ensure=stopped.
  5. Delete puppet agent’s SSL credentials with sudo rm -r /etc/puppetlabs/puppet/ssl/*.
  6. Request an agent certificate from the CA puppet master with sudo puppet agent -t. The puppet master will autosign the request, and the puppet agent will fetch the certificate.

    The puppet agent will also try to do a full Puppet run, which will fail. This is as expected, so don’t worry about it.

    If the master doesn’t autosign the certificate in this step, you may have changed its autosign configuration. You’ll need to manually sign the certificate (see below).

  7. Navigate to the console certs directory with sudo cd /opt/puppet/share/puppet-dashboard/certs. Stay in this directory for the following steps.
  8. Remove all the credentials in this directory with sudo rm -r /opt/puppet/share/puppet-dashboard/certs/*.
  9. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:create_key_pair.
  10. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:request. The puppet master will autosign the request, and the script will fetch the certificate.

If the master doesn’t autosign the certificate in this step, you may have changed its autosign configuration. You’ll need to manually sign the certificate (see below).

  1. Run sudo /opt/puppet/bin/rake RAILS_ENV=production cert:retrieve.
  2. Ensure the console can access the new credentials with sudo chown -R puppet-dashboard:puppet-dashboard /opt/puppet/share/puppet-dashboard/certs.
  3. Re-start the console service with sudo puppet resource service pe-httpd ensure=running.
  4. Re-start the puppet agent service with sudo puppet resource service pe-puppet ensure=running.

At this point:

  • The console server is now completely taken care of.
  • The puppet master can talk to the console again, and vice versa.
  • The puppet master can now serve catalogs to agents.

    However, it will only trust agents that have replaced their certificates. The only agents that have replaced their certificates at this point are the puppet master node, the PuppetDB node, and the console node.

  • The console is usable, but because its SSL certificate has been replaced, your web browser may notice the change, assume it results from a malicious attack, and refuse to allow you access. If this happens, you may need to go into your browser’s collection of cached certificates and delete the old cert. Details of this process are beyond the scope of this guide and will vary by browser and platform. (You can delay having to figure this out by temporarily using a different browser.)
  • Orchestration and live management may not immediately work, but they will start working again within about 30 minutes, as soon as both the puppet master server and the console node complete a puppet agent run. (The certificates used by MCollective and the ActiveMQ service are completely managed by Puppet, and don’t have to be manually regenerated.)

On any of the nodes that are completely taken care of, you can start a successful agent run with sudo puppet agent -t. Try it on your console and PuppetDB nodes to ensure it works as expected.

Step 4: Clear and Regenerate Certs for PE Agents

To replace the certs on agents, you’ll need to log into each agent node and do the following:

  1. Stop the puppet agent service. On *nix nodes, run sudo puppet resource service pe-puppet ensure=stopped. On Windows nodes, run the same command (minus sudo) with Administrator privileges.
  2. Stop the orchestration service. On *nix nodes, run sudo puppet resource service pe-mcollective ensure=stopped. On Windows nodes, run the same command (minus sudo) with Administrator privileges.
  3. Delete the agent’s SSL directory. On *nix nodes, run sudo rm -r /etc/puppetlabs/puppet/ssl/*. On Windows nodes, delete the $confdir\ssl directory, using the Administrator confdir. See here for more information on locating the confdir.
  4. Re-start the puppet agent service. On *nix nodes, run sudo puppet resource service pe-puppet ensure=running. On Windows nodes, run the same command (minus sudo) with Administrator privileges.

    Once the puppet agent starts, it will automatically generate keys and request a new certificate from the CA puppet master.

  5. If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this with the PE console’s request manager, or by logging into the CA puppet master server, running sudo puppet cert list to see pending requests, and running sudo puppet cert sign <NAME> to sign requests.

Once an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run. After a node has fetched its new certificate and completed a full Puppet run, it will once again appear in orchestration and live management. If, after waiting for a short time, you don’t see the agent node in live management, use NTP to make sure time is in sync aross your PE deployment. On Windows nodes, you may need to log into the node and check the status of the Marionette Collective service — sometimes it can hang while attempting to stop or restart.

Once you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.

↑ Back to top