How to rotate the Firefox Release signing GPG Key#

This is a rough guide, and is likely going to be out of date every time we have to rotate the keys. It is likely you will have to take different or additional steps than described here. It’s always a good idea to look at the most recent rotation bug (like https://bugzilla.mozilla.org/show_bug.cgi?id=1703397) before getting started.

When#

You should start this process at least a month prior to the current key expiring. Because rotation involves access to the offline master keychain, there’s usually a bit of lead time involved.

Process#

At a high level, the process is as follows:

  1. Generate a new signing subkey

  2. Publish the new public key

  3. Import the new private key into autograph

  4. Update the public key in a few places

  5. Start signing with the new private key

Here’s slightly more detail on each step

Generate a new signing subkey#

This is documented on https://mana.mozilla.org/wiki/display/SECURITY/Releng+Product+Signing%3A+GnuPG+key+generation+and+handling.

The most important end result of this process will be a new private signing subkey that has been deployed to Autograph, a new public signing subkey, and access to the private key through a new set of credentials that you will be provided.

Deployment Part 1: Testing the new signing subkey#

The new signing subkey will be deployed to the production instance of Autograph, which means it must be tested through a production worker and repository. The best and safest way to do this is against the Adhoc Signing repository - which will not impact any users or developers if something goes wrong. You can do this as follows:

  1. Update the GPG_PUBKEY_PATH entry for firefoxci-adhoc-3 in the scriptworker-scripts repository.

    1. Create a new gpg keyring and import the current key:

      mkdir new-keyring
gpg --homedir new-keyring --import current.key

    2. Add the new public key to the keyring:

      gpg --homedir new-keyring --import new-public.key

    3. Export all the keys to a new armored file:

      gpg --homedir new-keyring --export --armor > new.key

    4. Add the header from the existing KEY file to new.key, with the new sub key information (generated with e.g. gpg –show-key –list-options show-unusable-subkeys < new.key). For example, in the 2023 rotations we added the following (the last line is the new subkey information):

      This file contains the public PGP key that is used to sign builds and
artifacts of Mozilla projects (such as Firefox and Thunderbird).

Please realize that this file itself or the public key servers may be
compromised.  You are encouraged to validate the authenticity of these keys in
an out-of-band manner.

Mozilla users: pgp < KEY

pub   rsa4096 2015-07-17 [SC]
      14F26682D0916CDD81E37B6D61B7B526D98F0353
uid           [  full  ] Mozilla Software Releases <release@mozilla.com>
sub   rsa4096 2015-07-17 [S] [expired: 2017-07-16]
sub   rsa4096 2017-06-22 [S] [expired: 2019-06-22]
sub   rsa4096 2019-05-30 [S] [expires: 2021-05-29]
sub   rsa4096 2021-05-17 [S] [expires: 2023-05-17]
sub   rsa4096 2023-05-05 [S] [expires: 2025-05-04]

    5. Add new.key to signingscript and point the firefoxci-adhoc-3 pool’s GPG_PUBKEY_PATH variable at it.

    6. Open a Pull Request with the changes; wait for it to get merged.

  2. Update the GPG username and password for firefoxci-adhoc-3 in the relengworker SOPS repository.

  3. Deploy production scriptworkers to pick up the changes you made above.

  4. Open a PR with a new signing manifest in the Adhoc Signing repository that will sign with GPG (note: when it does a test signing as part of opening a PR, it will not be using the new keys, because PRs use dep certs).

  5. Get your PR reviewed and merged.

  6. Use the Promote an Adhoc Signature action to kick off signing for your new manifest. This should sign successfully with the new subkey.

  7. Verify the KEY file and detached signature that the task publishes. This will look something like:

    # Download the file that was signed, the detached signature, and the `KEY` to your local machine
# Create a new keyring and import the published KEY file
mkdir new
gpg --homedir new --import KEY
# Verify the detached signature
gpg --homedir new --verify *.asc
# You should see output like:
# gpg: Good signature from "Mozilla Software Releases <release@mozilla.com>"

It can be helpful to see how this was done in the past before doing this yourself. You are encouraged to look through the cloudops-infra and relengworker SOPS repositories to find the appropriate changesets.

You can also find an example of the adhoc signing manifest in this PR. If that signing manifest still exists in the repository, you can even skip steps 4 and 5, and promote that manifest in step 6 instead.

Deployment Part 2: Everything else#

Now that you’ve verified that the autograph credentials work, and that the gpg signatures produced are correct, you can roll it out to the remaining signingscript pools. This will look nearly identical to the steps above – and in fact, you can usually just copy and paste the credentials you already put in the SOPS repo, and the GPG_PUBKEY_PATH variable in signingscript. This must be done for each signing pool that uses our primary GPG key. At the time of writing this is the following (not including the adhoc one you just updated):

firefoxci-gecko-3 prod
firefoxci-comm-3 prod
firefoxci-mobile-3 prod
firefoxci-app-services-3 prod
firefoxci-glean-3 prod
firefoxci-adhoc-3 prod

Do not take this list as complete. Things have likely changed since these instruction were written. You should inspect both files and make a complete list of everything using the previous subkeys.

Once you’ve updated both files again (and had the cloudops-infra PR merged) you’re ready to deploy. Take caution to avoid doing this if there are releases in flight. If you do, files from the same release will get signed with different keys. This doesn’t break anything, but it does cause confusion.

Publish the new public key#

The new key needs to be published on keys.openpgp.org. Be sure to “verify” the key after publishing, by having them send an e-mail link to click on, to make sure users will see an identity associated with it (otherwise it’s useless).

A blog post like https://blog.mozilla.org/security/2019/06/13/updated-firefox-gpg-key/ should also be made.

Import the new private key into autograph#

You will need to send the new private key and its passphrase to an autograph team member. They will handle importing it into autograph, and creating new credentials (if necessary).

Update the public key#

We publish our public key in a couple of places, and store it in others to verify some of our own signatures. Specifically, at least the following will need to be updated:

Start signing with the new private key#

This will probably involve changing the signingscript secrets to use new autograph credentials that are associated with the new key, and then deploying signingscript.