Signing and Notarization#

Certificates#

For certificate rotation, see the signing page in RelEng Confluence space.

Provisioning Profiles#

Since we started adding restricted entitlements to Firefox, Apple forces us to include a Provisioning Profile in the bundle. Mozilla VPN also uses profiles.

For a summary of what a Provisioning Profile is, see this article. At a minimum these are the things in each profile:

  1. One App ID (aka Bundle ID). i.e.: org.mozilla.firefox

  2. One Certificate reference (this cert has to be provided by Apple)

  3. The entitlements that are enabled for this “Identifier” (Bundle ID)

  4. Devices (Only present in Development Provisioning Profiles!)

Given profiles have 1-to-1 relation to certificates, they need to be rotated at the same time.

At this point, profiles are stored in the workers under the provisionprofiles directory, and there is no automation around deploying them.

Notarization#

We currently notarize Firefox using rcodesign (aka apple-codesign). Using this rust tool allows us to notarize apps in Linux, more specifically in signingscript.

Machines and maintenance#

The machine list is here.

We also keep quarantine and current status of each machine there.

Some deployment automation has been added to ronin-puppet. We want to be able to fully automate rollout, from imaging to rollout. We also want to be able to bump dependency versions in ronin-puppet and have it Just Work. The todo list for those is here. As of November 2023, SRE is investigating an alternative method to image and maintain the Mac fleet. RelEng might use some of these tools to maintain the Mac signers.

Some useful wiki links:

General notarization workflow (DEPRECATED)#

This behavior is deprecated, but still available, since ESR still uses it. After ESR115 goes away, we should either delete this, or update to reflect the current workflow

In the mac_notarize behavior, iScript will:

  • extract the files from a dmg

  • sign widevine and omnija (autograph signing)

  • sign mac, without the mac signing servers

  • create .pkg installers and sign them

  • create a zipfile of the .app and .pkg files

  • send that zipfile to Apple for notarization

  • poll Apple for notarization status

  • on success, “staple” the notarization to the app

  • create tarballs of the .app files

However, we would often have issues in the polling step or otherwise end up wasting expensive signing worker cycles just sitting there idle.

To remedy this, we split notarization into three behaviors: mac_notarize_part_1, notarization_poller, and mac_notarize_part_3.

In the mac_notarize_part_1 behavior, iScript will:

  • extract the files from a dmg

  • sign widevine and omnija (autograph signing)

  • sign mac, without the mac signign servers

  • create a zipfile of the .app and .pkg files

  • send that zipfile to Apple for notarization

  • create tarballs of the .app files, unstapled

  • upload the tarballs and .pkg files as artifacts, as well as a uuids json file.

In the notarization_poller task, notarization_poller will download the uuids json file from the part_1 task, and poll Apple. If they all return complete, the task goes green. Otherwise the task will fail or throw an exception.

In the mac_notarize_part_3 behavior, iScript will:

  • download the tarballs and .pkg files from part_1, staple the notarization, and upload

Debugging#

The code used is here.

An error like iscript.exceptions.TimeoutError: Timed out polling for uuid aa2dc2bc-9059-426e-a292-0bfb575a337b! means that Apple has taken too long to notarize. We may want to bump the `notarization_poll_timeout <mozilla-releng/scriptworker-scripts>`__ everywhere. Generally a rerun has fixed this issue.

Escalation#

@hneiva knows notarization the best, and can help debug.