Releng Changelog#

Every time a new releaseduty cycle begins, new RelEng people are (re)ramping up. To ease the transition, we’re keeping this CHANGELOG that should trace all the tools/productivity/infrastructure changes. This includes high-level changes and should come into compliance with the rest of the documentation.

This page best serves the people that have previously been into the releaseduty cycle. Starting from the baseline, people coming back after N cycles can ramp up incrementally with the latest changes.

As releaseduty squirrels are the ones with the best context when it comes to releases, they are the ones to edit this page and amend it accordingly. Keep in mind that changes should be in compliance with the other pieces of documentation.

During 115.0 >= 2023-05-09#

During 114.0 >= 2023-04-11#

During 113.0 >= 2023-03-14#

  • langpack submission switched to v5 of the AMO API, so manual action is no longer needed when a new language is added

During 112.0 >= 2023-02-14#

  • new langpacks caused bustage, bug 1817344, https://moz-releng-docs.readthedocs.io/en/latest/addons/langpacks.html?highlight=addon#how-to-handle-new-languages-for-release

During 111.0 >= 2023-01-16#

  • [Jan 2023 rotation] Change all the things

  • tooltool.py synchronization and consolidation

  • Widevine native aarch64, bug 1812343

During 110.0 >= 2022-12-12#

  • Focus moved to the new mono-repo, https://github.com/mozilla-mobile/firefox-android/ (bug 1797704)

  • scriptworker 46.0.0 released which removes 2021 CoT public key

  • Balrog v3.21 deployed to production

During 109.0 >= 2022-11-14#

During 108.0 >= 2022-10-17#

  • Android components moved to the new mono-repo, https://github.com/mozilla-mobile/firefox-android/ (bug 1797702)

During 107.0 >= 2022-08-20#

During 106.0 >= 2022-08-22#

  • Adding new base_rev and base_ref parameters in taskgraph (bug 1784232)

    • Deployed new build-decision image (D155195)

    • List to be expanded once new changes are deployed.

  • Balrog 3.20 deployed

During 105.0 >= 2022-07-25#

  • Deployed new iscript + scriptworker version to mac signers

  • FirefoxCI cluster upgraded to Taskcluster 44.18.0

    • relduty no longer announces minor upgrades on dev-platform

During 104.0 >= 2022-06-28#

  • update-verify sends requests with “Cache-Control: max-stale=0” to avoid getting out of date responses from the CDN fronting balrog (https://bugzilla.mozilla.org/show_bug.cgi?id=1778728)

  • Android:

    • Android-Components gets released off of https://shipit.mozilla-releng.net/ instead of Github Releases (RELENG-11)

    • Fenix/Focus are not immediately shipped anymore. We now have a 2-step process where builds are promoted and sent to our testing population before shipping them to Github or archive.mozilla.org (bug 1776830 and bug 1778056)

    • Version number now follow the gecko scheme. This means 104.0.0-beta.1 will instead be called 104.0b1 (bug 1777255)

    • L10n string uplift are automatically approved by mergify (bug 1779797)

    • Android-Components bumps are also automatically approved (bug 1780740)

During 103.0 >= 2022-05-31#

  • FirefoxCI cluster upgraded to Taskcluster 44.14.0

  • https://github.com/mozilla/delivery-dashboard/ deployed (first time since 2018!)

  • Deployed balrog v3.18 to production (https://bugzilla.mozilla.org/show_bug.cgi?id=1772799)

  • (ahal) Copied new scriptworker CoT private key to mac signers

During 102.0 >= 2022-05-02#

  • (ahal) Deployed scriptworker pools to pick up new CoT private key

  • (ahal) Updated dependencies in image-task docker image

  • Treestatus updated for esr-102 and deployed

  • signingscript update for authenticode timestamping

  • scriptworker-scripts and shipit updated for esr-102 and deployed

  • new docker-worker pools to fix ssl certs

  • Widevine 4.10.2449.0 final rollout

  • aus5.mozilla.org (balrog prod client domain) certificate renewed (https://bugzilla.mozilla.org/show_bug.cgi?id=1768253

  • FirefoxCI cluster upgraded to Taskcluster 44.13.7

During 101.0 >= 2022-04-05#

  • Updated Mac certificates on all prod mac hosts. + Nightly builds now use the same keychain file as release.

  • FirefoxCI cluster upgraded from Taskcluster 44.8.4 to 44.11.1; Postgres DB instance doubled in RAM.

  • Updated all iOS apps in Bitrise to use new cert and provisioning profiles

  • https://pypi.org/project/mozilla-version/ updated to 1.0.1 -> beetmover, bouncer, pushapk, treescript deps updated

  • Experimental change to timestamp server for signing windows firefox, https://github.com/mozilla-releng/scriptworker-scripts/pull/501/

  • Allowed System Addons to be served from archive.m.o, https://github.com/mozilla-releng/balrog/pull/2520

During 100.0 >= 2022-03-08#

  • Deployed Balrog v3.16 to production

  • FirefoxCI cluster upgraded from 43.2.0 -> 44.8.1

  • beetmover-geckoview jobs (copying geckoview artifacts to maven) now run as part of the promote_firefox graphs on release and beta instead of push_firefox

  • Switched SENTRY_DSN for Balrog -> sentry.io

  • SHA-256 signing of Windows Installers, bug 1745467

  • Widevine update, bug 1758423 (nightlytest, nightly, and beta channels)

  • Treescript merge day CLOBBER datestring

During 99.0 >= 2022-02-07#

Changed#

  • Internal pypi uploads for numpy/scipy/pillow, bug 1757792

  • Deployed pollbot v1.4.7 to production

  • Deployed balrog v3.14 to production

  • Deployed balrog v3.13 to production

  • Minor refactor to release_artifacts in gecko_taskgraph

  • Re-imaged mac-v1-signing3 and mac-v1-signing8. Quarantined mac-v1-signing19 for MozillaVPN work.

  • Signingscript retries for winsign (timestamp.digicert.com), bug 1590599

During 98.0 >= 2022-01-10#

Changed#

  • RELENG-659 - Py3 support on xpi/addons and updated the taskcluster version used

  • Change ShipIt to point to xpi-manifest “main” branch

  • Deployed new version of iscript to mac signers

  • Deployed balrog v3.12 to production

  • Deployed signingscript to production

  • Modified mac-signing releng ssh allowlist

During 97.0 >= 2021-12-06#

Changed#

  • l10n-cross-channel-quarantine now nukes comm/ at the end of the run

  • Updated puppet dependencies for mac signers

  • fixed treescript merge-automation to be idempotent

  • refactored the system addons balrog rules

During 96.0 >= 2021-11-01#

Changed#

  • removed pushsnap scriptworker pools

During 95.0 >= 2021-10-04#

Changed#

  • focus-android releases switched to taskgraph

  • firefox desktop update-verify watershed

  • new firefox desktop msix packages

  • update-verify async download PR

During 82.0 >= 2020-09-21#

Changed#

  • partner attribution configs landed

  • balrog v2 submission

  • ESR68 EOL

  • central-to-beta merges should automatically l10n-bump

During 81.0 >= 2020-08-24#

Changed#

  • L10n bumper will look at the hg pushlog for the default revision, rather than Elmo. If we need to pin a previous revision due to bustage, land a change updating the revision and changing pin to true for that locale.

  • ESR68 l10n bumper disabled.

  • partner builds moved from b8 to b5 to adjust for shorter release cycles

  • partner attributions landed

During 80.0 >= 2020-07-27#

Changed#

  • Lots of accounts disabled; watch for fallout!

During 75.0 >= 2020-04-06#

Changed#

  • Changelog moved to rtd

  • Mac Notarization is now single-zip instead of multi-account due to bug 1620697. This may slow down the success case by a few minutes, but reduce or remove the multi-hour manual-intervention failure case.

Fixed#

  • Scriptworker now catches a number of asyncio.TimeoutErrors, and should be more robust.

During 74.0 >= 2020-03-10#

Added#

  • added support for shipping Firefox Flatpaks to Flathub

During 73.0 >= 2020-02-10#

Changed#

  • py38 rollout for scriptworkers, tooltool, balrog, and other release services

  • aws -> gcp, then back again, for tier 1 workers

  • mac 3-tier notarization

During 71.0 >= 2019-10-14#

Changed#

  • (still-in-progress) signing servers are being shut down. we’ve already stopped using them, however

  • mergeduty processed is back to two stages like it used to be before Quantum: beta -> release in the first iteration while central -> beta and bumping central in the second week

  • l10n bumper for central is moving to taskcluster cron shortly; will uplift to beta and esr68 later

  • switched to GCP scriptworkers in all gecko trees (including esr68)

  • after TCW of 9th Nov, switched to GCP scriptworkers in comm repos too, but also Github mobile projects

  • secrets are now managed in sops, no longer in hiera

  • decommissioning of the AWS infrastructure

  • treescript has been refactored

  • scriptworker-scripts docker images now mount the repo at /app instead of the script (we may revisit this to lower disk usage)

  • mergeduty process change for upcoming automated betas. tl;dr - no longer Devedition b1 and b2 during RC week. central merges to beta a week later

During 69.0 >= 2019-05-13#

Changed#

  • mac signing now happens on the mac notarization pool.

  • mac nightly and release builds are now notarized as well as signed.

  • mac signing tasks also create pkg installers

During 67.0 >= 2019-03-11#

Changed#

  • update & final verify will check SSL certificates of Balrog, archive.m.o etc

  • the release promotion action is now a hook.

  • chain of trust no longer uses gpg to create and verify signatures.

During 66.0 >= 2019-01-30#

Changed#

  • scriptworker 22 now signs and verifies ed25519 signatures; gpg signatures are deprecated.

  • partner repacks are now enabled on ESR60

  • adding list of whatsnewpage locales in-tree is now part of mergeduty

  • removed thunderbird docs. thunderbird is now managed by thunderbird team and they have support for shipit v2

During 64.0 >= 2018-10-15#

Changed#

  • MARs are now signed via autograph (hashes only)

  • Mergeduty tasks are now tracked in Trello

  • Releaseduty now assists CIDuty with puppet pyup PRs once per week

During 63.0 >= 2018-09-05#

Added#

  • added 6 new mac signing servers

Changed#

  • switched to ship-it v2

  • reduced mac signing concurrency from 4 to 2 to reduce server churn

During 62.0 >= 2018-06-26#

Changed#

  • switched to Ship-it v2 for beta releases

Added#

  • releasewarrior local automation to add WNP

During 61.0 >= 2018-05-09#

Added#

  • added partner off-cycle support + docs

  • widevine docs

Changed#

  • update verify only blocks on required platforms to start (eg: win32 update verify only blocks on win32 beetmover and balrog)

  • python 2.7.15 rollout to a portion of our infra

  • python 3.6.5 rollout completed to our scriptworkers

  • update verify no longer fails on expected differences, but will fail on any files found only in source or target; bug 1461490

During 60.0 >= 2018-03-13#

Added#

  • releaseduty now responsible for periodic file update approvals in phabricator. Change this line to update those responsible.

  • added esr52-latest* bouncer aliases

  • added firefox-esr-next-* bouncer aliases

  • added esr60 support

  • rerun action task in Treeherder - hopefully will help avoid duplicate reruns

  • KEY file to releases directory in bug 1446816 for >=60.0b11

  • added addonscript for langpack pushing to amo

Changed#

  • re-added source tarball for >=60.0b6

  • updated scriptworkers to share more client code

  • update verify now fails on any differences, and creates a summary log for >=60.0b8. additional documentation on how to interpret failures are available.

  • split snap repackage and push task in bug 1447263

  • updated rr3 add partner configs for fx60+

  • tc partner repacks!

  • split esr52 and esr60 templates

Fixed#

  • fixed beta+fennec bouncer aliases

  • push-apk-breakpoint removed in bug 1432817

During 59.0 >= 2018-01-24#

Added#

  • we track all major changes in this newly added CHANGELOG

  • separate RC docs for both Fennec and Firefox

  • new-style WNP setup, as documented.

  • we now have support for snaps in release channel

Changed#

  • old release promotion based on releasetasks is dead. Everything is in-tree scheduled now. Changes are riding the trains.

  • all releaseduty related documentation has been moved out of the wiki under warrior’s docs subfolder.

  • use release graphid to enter graphids, rather than editing the .json manually.

  • in-tree release notifications going away in favor of native taskcluster notifications. pulse-notify is to be retired whenever we kill esr52

  • Balrog toplevel submission and release scheduling now happens in balrog scriptworker for Fx60+.

During 58.0 >= 2017-11-15#

Added#

Changed#

  • old how-tos are now gathered under one roof in releasewarrior-2.0/old-howtosreleasewarrior-2.0/old-how-tos but they are gradually being migrated to the wiki which becomes the single source of truth

  • authentication method for taskcluster-cli changed. If you are upgrading from an older version of taskcluster-cli, you may have to remove the ${HOME}/.config/taskcluster.yml file to work with the new authentication method.

Removed#

  • dropped support for old releasewarrior

  • dropped the need for QE signoffs in Balrog

During 57.0 >= 2017-09-27#

Added#

  • a merge day instance has been added to ease the handover during mergedays but also for speed and reliability. More on this here. When connecting to the merge day instance, keep in mind to ensure its environment is clean and ready for use

  • simplified documentation for mid-betas and release checklists. If not already, duplicate an existing sheet in this google docs checklist and clear out the status that was carried over from the previous release.

Changed#

  • mergeduty dates shifted. We now merge beta to release and cetral to beta in the same day, keeping a relbranch for beta and hold central from version bump until before the release.

  • because of 57 and the aforementioned mergeduty change, the exception becomes rule so that for each new beta cycle X, we have X.0b1 and X.b2 just for Devedition, while Firefox starts at X.b3

Removed#

During 56.0 >= 2017-08-09#

Added#

  • releaseduty now checks on status of central nightlies when not undertaking other duties

  • releaseduty now handles signing of arbitrary .apk’s as requested by MoCo teams (Klar/Focus, Rocket)

Changed#

  • release-drivers mailing list is no longer used for signoffs/gtb’s, please subscribe to release-signoffs@

  • QE will now send an explicit GO for DevEdition Beta’s to be pushed live, usually in the same e-mail as the GO for Beta

Removed#

  • Fennec single locales no longer shipped