Regenerating certificates in monolithic Puppet Enterprise deployments

Note: This page explains how to regenerate all certificates in a monolithic PE deployment — that is, where the Puppet master, PuppetDB, and PE console components are all installed on the same server. See this page for instructions on regenerating certificates in a split PE deployment.

Overview

In some cases, you may find that you need to regenerate the certificates and security credentials (private and public keys) generated by PE’s built-in certificate authority (CA). For example, you may have a Puppet master that you need to move to a different network in your infrastructure, or you may find that you need to regenerate all the certificates and security credentials in your infrastructure due to an unforeseen security vulnerability.

Regardless of your situation, regenerating your certificates involves the following major steps (complete procedures follow below):

  1. Shut down all PE-related services.
  2. Delete and recreate the CA and generate the certificates and security credentials needed for PE to resume basic operation.
  3. Clear and regenerate certs and security credentials for PuppetDB.
  4. Clear and regenerate certs for orchestration-services.
  5. Clear and replace certs and security credentials for the PE console.
  6. Restart all PE-related services.
  7. Run Puppet on the monolithic server.
  8. Clear and replace certs and security credentials for all agent nodes.

Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the puppet cert clean command on your Puppet master and then follow step five for any agent certs that need to be replaced.

Important notes and warnings

  • In the following instructions, when <CERTNAME> is used, it refers to the Puppet agent’s certname. To find this value, run puppet config print certname before starting.

  • You must be logged in as a root, (or in the case of Windows agents, as an account with Administrator Privileges) to make these changes.

  • If you encounter any errors during steps that involve service stop/start, rm, cp, or chmod commands, you should diagnose these before continuing, as the success each step is very important to the success of the next step.

  • Regenerating your certificates will invalidate all existing authentication tokens. Once the regeneration process is complete, all PE users must generate new authentication tokens.

 puppet resource service puppet ensure=stopped
 puppet resource service pe-puppetserver ensure=stopped
 puppet resource service pe-activemq ensure=stopped
 puppet resource service mcollective ensure=stopped
 puppet resource service pe-puppetdb ensure=stopped
 puppet resource service pe-postgresql ensure=stopped
 puppet resource service pe-console-services ensure=stopped
 puppet resource service pe-nginx ensure=stopped
 puppet resource service pe-orchestration-services ensure=stopped
 puppet resource service pxp-agent ensure=stopped

Step 2: Delete and recreate the CA

On your monolithic Puppet master server:

  1. Back up the following directories.

    • /etc/puppetlabs/puppet/ssl/

    • /etc/puppetlabs/puppetdb/ssl/

    • /opt/puppetlabs/server/data/console-services/certs/

    • /opt/puppetlabs/server/data/postgresql/9.4/data/certs/

    • /etc/puppetlabs/orchestration-services/ssl

    If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet support as soon as you restore service so we can help you secure your site.

  2. Delete the CA and clear all certs from your master.

     rm -rf /etc/puppetlabs/puppet/ssl/*
    
  3. Remove the cached catalog.

     rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json
    
  4. From the ssl directory, regenerate the CA.

     puppet cert list -a
    

    You should see this message: Notice: Signed certificate request for ca

  5. Generate the Puppet master’s new certs.

     puppet cert generate <CERTNAME> --dns_alt_names=<DNS_ALT_NAMES>
    

    Note: Be sure to specify any dns alt names you have. You can find the list of your current dns alt names with puppet cert list <CERTNAME>. By default, PE uses puppet and puppet.domain.

  6. Create the new MCollective certificates.

     puppet cert generate pe-internal-mcollective-servers
     puppet cert generate pe-internal-peadmin-mcollective-client
     puppet cert generate pe-internal-puppet-console-mcollective-client
    
  7. Copy the CA’s certificate revocation list (CRL) into place.

     cp /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem /etc/puppetlabs/puppet/ssl/crl.pem
     chown -R pe-puppet:pe-puppet /etc/puppetlabs/puppet/ssl
    

Step 3: Clear and regenerate certs for PuppetDB

On your monolithic Puppet master server:

  1. Clear the certs and security credentials from the PuppetDB SSL directory.

     rm -rf /etc/puppetlabs/puppetdb/ssl/*
    
  2. Copy the certs and security credentials generated in step 2.5 for the Puppet master node to the PuppetDB SSL directory.

    Note that the Puppet master, PuppetDB, and PE console share the same agent cert and security credentials.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pem
    
  3. Create PuppetDB’s .pk8 cert.

     cd /etc/puppetlabs/puppetdb/ssl
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pem -out /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-puppetdb:pe-puppetdb /etc/puppetlabs/puppetdb/ssl
    
  4. Clear the certs and security credentials from the PostgreSQL certs directory.

     rm -rf /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
    
  5. Copy the certs and security credentials generated in step 2.5 for the Puppet master to the PostgreSQL certs directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.private_key.pem
     chmod 400 /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
     chown pe-postgres:pe-postgres /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
    

Step 4: Clear and regenerate certs for orchestration-services

On your monolithic Puppet master server:

  1. Remove all credentials in the orchestration-services certificate directory.

     rm -rf /etc/puppetlabs/orchestration-services/ssl/*
    
  2. Copy the PE agent cert and security credentials to the orchestration-services cert directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pem
    
  3. Create the orchestration-services .pk8 cert.

     cd /etc/puppetlabs/orchestration-services/ssl
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pem -out /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-orchestration-services:pe-orchestration-services /etc/puppetlabs/orchestration-services/ssl/
    

Step 5: Clear and regenerate certs for the PE console

On your monolithic Puppet master server:

  1. Remove all credentials in the console-services certificate directory.

     rm -rf /opt/puppetlabs/server/data/console-services/certs/*
    
  2. Copy the PE agent cert and security credentials to the console-services cert directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pem
    
  3. Create the console-services .pk8 cert.

     cd /opt/puppetlabs/server/data/console-services/certs
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pem -out /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs/
    
  4. Ensure the console can access the new credentials.

     chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs
    

Step 6: Restart PE services

Now that all the certs are in place, restart the PE services you stopped.

  puppet resource service pe-puppetserver ensure=running
  puppet resource service pe-postgresql ensure=running
  puppet resource service pe-puppetdb ensure=running
  puppet resource service pe-console-services ensure=running
  puppet resource service pe-nginx ensure=running
  puppet resource service pe-activemq ensure=running
  puppet resource service mcollective ensure=running
  puppet resource service puppet ensure=running
  puppet resource service pe-orchestration-services ensure=running
  puppet resource service pxp-agent ensure=running

Step 7: Run Puppet on the monolithic server

Kick off a Puppet run. A successful Puppet run on the master is necessary to ensure that the MCollective service and ActiveMQ is configured properly.

Note: If you want to regenerate the DH param files, see Configuring the PE console to use a custom Diffie-Hellman file, which contains instructions on regenerating files. You will need to delete any DH param files that are in place (at /etc/puppetlabs/nginx/<PROXY-CUSTOM-dhparam>.pem) before regenerating them.

Step 8: Clear and regenerate certs for Puppet agents

Unless otherwise indicated, the following steps should be performed on your agent nodes.

  1. Back up the /etc/puppetlabs/puppet/ssl/ directory.

    If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet support as soon as you restore service so we can help you secure your site.

  2. Stop the Puppet agent and MCollective services.

    puppet resource service pe-puppet ensure=stopped
    puppet resource service pe-mcollective ensure=stopped
    
  3. Delete the agent’s SSL directory. On *nix nodes, run rm -rf /etc/puppetlabs/puppet/ssl.

    On Windows nodes, delete the $confdir\ssl directory, using the Administrator confdir.

  4. Remove the cached catalog. on *nix nodes, run rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json.

    On Windows nodes, delete the $client_datadir\catalog\<CERTNAME>.json file, using the Administrator confdir.

  5. Re-start the Puppet agent and MCollective services.

    puppet resource service pe-puppet ensure=running
    puppet resource service pe-mcollective ensure=running
    

    After Puppet agent starts, it will automatically generate keys and request a new certificate from the CA Puppet master.

  6. If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this with the PE console’s request manager, or by logging into the CA Puppet master server, running puppet cert list to see pending requests, and running puppet cert sign <NAME> to sign requests.

After an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run. After a node has fetched its new certificate and completed a full Puppet run, it will once again appear under MCollective control. You might need to use NTP to make sure time is in sync across your PE deployment. On Windows nodes, you may need to log into the node and check the status of the Marionette Collective service — sometimes it can hang while attempting to stop or restart.

After you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.

↑ Back to top