Regenerating certificates in split Puppet Enterprise deployments

Note: This page explains how to regenerate all certificates in a split PE deployment — that is, where the Puppet master, PuppetDB, and PE console components are all installed on separate servers. See this page for instructions on regenerating certificates in a monolithic PE deployment.

If you don’t need to regenerate certs for all components in your deployment, you can choose instructions for just the Puppet master, the PE console, or PuppetDB.

Overview

In some cases, you may find that you need to regenerate the SSL certificates and security credentials (private and public keys) that are generated by PE’s built-in certificate authority (CA). For example, you may have a Puppet master you need to move to a different network in your infrastructure, or you may find you need to regenerate all the certificates and security credentials in your infrastructure due to an unforeseen security vulnerability.

Regardless of your situation, regenerating your certificates involves the following major steps (complete procedures follow below):

  1. Shut down all PE-related services.
  2. Delete and recreate the CA and generate the certificates and security credentials needed for PE to resume basic operation.
  3. Clear and regenerate certs and security credentials for PuppetDB.
  4. Clear and regenerate certs for orchestration-services.
  5. Clear and regenerate certs and security credentials for the PE console.
  6. Restart all PE-related services.
  7. Run Puppet on the Puppet Master, PuppetDB, and Console Nodes
  8. Clear and replace certs and security credentials for all agent nodes.

Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can use the puppet cert clean command on your Puppet master and then follow step 8 for any agents that need to be replaced.

Important notes and warnings

  • You must be logged in as a root, (or in the case of Windows agents, as an account with Administrator Privileges) to make these changes.

  • If you encounter any errors during steps that involve service stop/start, rm, cp, or chmod commands, you should diagnose these before continuing, as the success each step is very important to the success of the next step.

  • Regenerating your certificates will invalidate all existing authentication tokens. Once the regeneration process is complete, all PE users must generate new authentication tokens.

  1. On your Puppet master node, run the following commands:

     puppet resource service puppet ensure=stopped
     puppet resource service pe-puppetserver ensure=stopped
     puppet resource service pe-activemq ensure=stopped
     puppet resource service mcollective ensure=stopped
     puppet resource service pe-orchestration-services ensure=stopped
     puppet resource service pxp-agent ensure=stopped
    
  2. On your PuppetDB node, run the following commands:

     puppet resource service puppet ensure=stopped
     puppet resource service pe-puppetdb ensure=stopped
     puppet resource service pe-postgresql ensure=stopped
     puppet resource service mcollective ensure=stopped
    
  3. On your PE console node, run the following commands:

     puppet resource service puppet ensure=stopped
     puppet resource service pe-console-services ensure=stopped
     puppet resource service pe-nginx ensure=stopped
     puppet resource service mcollective ensure=stopped
    

Step 2: Delete and recreate the CA

Notes: In the following instructions, when <CERTNAME> is used, it refers to the Puppet agent’s certname. To find this value, run puppet config print certname before starting.

On your Puppet master:

  1. Back up the /etc/puppetlabs/puppet/ssl/ and /etc/puppetlabs/orchestration-services/ssl directories.

    If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet support as soon as you restore service so we can help you secure your site.

  2. Clear all certs from your Puppet master.

     rm -rf /etc/puppetlabs/puppet/ssl/*
    
  3. Remove the cached catalog.

     rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json
    
  4. From the ssl directory, regenerate the CA.

     puppet cert list -a
    

    You should see this message: Notice: Signed certificate request for ca.

  5. Generate the Puppet master’s new certs.

     puppet cert generate <CERTNAME> --dns_alt_names=<DNS_ALT_NAMES>
    

    Note: Be sure to specify any dns alt names you have. You can find the list of your current dns alt names with puppet cert list <CERTNAME>. By default, PE uses puppet and puppet.domain.

  6. Copy over the CA certificate revocation list (CRL).

      cp /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem /etc/puppetlabs/puppet/ssl/crl.pem
      chown -R pe-puppet:pe-puppet /etc/puppetlabs/puppet/ssl
    
  7. Generate new MCollective certs.

      puppet cert generate pe-internal-mcollective-servers
      puppet cert generate pe-internal-peadmin-mcollective-client
      puppet cert generate pe-internal-puppet-console-mcollective-client
    
  8. Restart the pe-puppetserver service.

      puppet resource service pe-puppetserver ensure=running
    

Step 3: Clear and regenerate certs for PuppetDB

On your PuppetDB server:

  1. Back up the following directories:

    • /etc/puppetlabs/puppet/ssl/

    • /etc/puppetlabs/puppetdb/ssl/

    • /opt/puppetlabs/server/data/postgresql/9.4/data/certs/

  2. Delete Puppet agent’s SSL cert and security credentials.

      rm -rf /etc/puppetlabs/puppet/ssl/*
    
  3. Remove the cached catalog.

     rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json
    
  4. Generate keys and request a new certificate from the CA Puppet master. These certs will end up in /etc/puppetlabs/puppet/ssl.

      puppet agent --test --no-daemonize --noop
    

    Note: This agent run will not complete successfully, but it is necessary to set up the agent certificate for the node. You will see some errors about node definition and the inability to submit facts due to PuppetDB being offline. You can ignore these.

  5. Delete puppetDB’s SSL cert and security credentials.

       rm -rf /etc/puppetlabs/puppetdb/ssl/*
    
  6. Copy the agent certs and security credentials to the PuppetDB SSL directory.

      cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.cert.pem
      cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.public_key.pem
      cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pem
    

    Note: If you’ve set up a load-balanced PuppetDB, you will need to follow the instructions in PuppetDB behind a load balancer causes Puppet server errors to ensure PuppetDB uses a shared certificate instead of the agent cert.

  7. Create PuppetDB’s .pk8 cert.

     cd /etc/puppetlabs/puppetdb/ssl
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pem -out /etc/puppetlabs/puppetdb/ssl/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-puppetdb:pe-puppetdb /etc/puppetlabs/puppetdb/ssl
    
  8. Clear the certs and security credentials from the PostgreSQL certs directory.

      rm -rf /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
    
  9. Copy the agent certs and security credentials to the PostgreSQL certs directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/postgresql/9.4/data/certs/<CERTNAME>.private_key.pem
     chmod 400 /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
     chown pe-postgres:pe-postgres /opt/puppetlabs/server/data/postgresql/9.4/data/certs/*
    

Step 4: Clear and regenerate certs for orchestration-services

On your Puppet master server:

  1. Remove all credentials in the orchestration-services certificate directory.

     rm -rf /etc/puppetlabs/orchestration-services/ssl/*
    
  2. Copy the PE agent cert and security credentials to the orchestration-services cert directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pem
    
  3. Create the orchestration-services .pk8 cert.

     cd /etc/puppetlabs/orchestration-services/ssl
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pem -out /etc/puppetlabs/orchestration-services/ssl/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-orchestration-services:pe-orchestration-services /etc/puppetlabs/orchestration-services/ssl/       
    

Step 5: Clear and regenerate certs for the PE console

On the PE console server:

  1. Back up the following directories:

    • /etc/puppetlabs/puppet/ssl/

    • /opt/puppetlabs/server/data/console-services/certs

  2. Delete Puppet agent’s SSL cert and security credentials.

     rm -rf /etc/puppetlabs/puppet/ssl/*
    
  3. Remove the cached catalog.

     rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json
    
  4. Generate keys and request a new certificate from the CA Puppet master. These certs will end up in /etc/puppetlabs/puppet/ssl.

     puppet agent --test --no-daemonize --noop
    

    Note: This agent run will not complete successfully, but it is necessary to set up the agent certificate for the node. You will see some errors about node definition and the inability to submit facts due to PuppetDB being offline. You can ignore these.

  5. Purge the console-services directory.

     rm -rf /opt/puppetlabs/server/data/console-services/certs/*
    
  6. Copy the agent cert and security credentials to the console-services cert directory.

     cp /etc/puppetlabs/puppet/ssl/certs/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.cert.pem
     cp /etc/puppetlabs/puppet/ssl/public_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.public_key.pem
     cp /etc/puppetlabs/puppet/ssl/private_keys/<CERTNAME>.pem /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pem
    
  7. Create the console-services .pk8 cert.

     cd /opt/puppetlabs/server/data/console-services/certs
     openssl pkcs8 -topk8 -inform PEM -outform DER -in /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pem -out /opt/puppetlabs/server/data/console-services/certs/<CERTNAME>.private_key.pk8 -nocrypt
     chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs/
    
  8. Ensure the console can access the new credentials.

     chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs
    

Step 6: Restart PE services

Now that all the certs are in place, restart the PE services you stopped.

  1. On your Puppet master node, run the following commands:

     puppet resource service puppet ensure=running
     puppet resource service pe-puppetserver ensure=running
     puppet resource service pe-activemq ensure=running
     puppet resource service mcollective ensure=running
     puppet resource service pe-orchestration-services ensure=running
     puppet resource service pxp-agent ensure=running
    
  2. On your PuppetDB node, run the following commands:

     puppet resource service puppet ensure=running
     puppet resource service pe-puppetdb ensure=running
     puppet resource service pe-postgresql ensure=running
     puppet resource service mcollective ensure=running
    
  3. On your PE console node, run the following commands:

     puppet resource service puppet ensure=running
     puppet resource service pe-nginx ensure=running
     puppet resource service pe-console-services ensure=running
     puppet resource service mcollective ensure=running
    

Step 7: Run Puppet on the Puppet master, PuppetDB, and console nodes

Kick off a Puppet run on the Puppet master, PuppetDB, and console. A successful Puppet run on the master, PuppetDB and PE console is necessary to ensure that the MCollective service and ActiveMQ is configured properly.

Note: If you want to regenerate the DH param files, see Configuring the PE console to use a custom Diffie-Hellman file, which contains instructions on regenerating files. You will need to delete any DH param files that are in place (at /etc/puppetlabs/nginx/<PROXY-CUSTOM-dhparam>.pem) before regenerating them.

Step 8: Clear and regenerate certs for Puppet agents

Unless otherwise indicated, the following steps should be performed on your agent nodes.

  1. Back up the /etc/puppetlabs/puppet/ssl/ directory.

    If something goes wrong, you may need to restore these directories so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, you should contact Puppet support as soon as you restore service so we can help you secure your site.

  2. Stop the Puppet agent and MCollective services.

    puppet resource service pe-puppet ensure=stopped
    puppet resource service pe-mcollective ensure=stopped
    
  3. Delete the agent’s SSL directory. On *nix nodes, run rm -rf /etc/puppetlabs/puppet/ssl.

    On Windows nodes, delete the $confdir\ssl directory, using the Administrator confdir.

  4. Remove the cached catalog. on *nix nodes, run rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<CERTNAME>.json.

    On Windows nodes, delete the $client_datadir\catalog\<CERTNAME>.json file, using the Administrator confdir.

  5. Re-start the Puppet agent and MCollective services.

    puppet resource service pe-puppet ensure=running
    puppet resource service pe-mcollective ensure=running
    

    When Puppet agent starts, it will automatically generate keys and request a new certificate from the CA Puppet master.

  6. If you are not using autosigning, you will need to sign each agent node’s certificate request. You can do this with the PE console’s request manager, or by logging into the CA Puppet master server, running puppet cert list to see pending requests, and running puppet cert sign <NAME> to sign requests.

After an agent node’s new certificate is signed, it will fetch it automatically within a few minutes and begin a Puppet run. After a node has fetched its new certificate and completed a full Puppet run, it will once again appear under MCollective control. You might need to use NTP to make sure time is in sync across your PE deployment. On Windows nodes, you may need to log into the node and check the status of the Marionette Collective service — sometimes it can hang while attempting to stop or restart.

After you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.

↑ Back to top