Releng Changelog#
Every time a new releaseduty cycle begins, new RelEng people are (re)ramping up. To ease the transition, we’re keeping this CHANGELOG that should trace all the tools/productivity/infrastructure changes. This includes high-level changes and should come into compliance with the rest of the documentation.
This page best serves the people that have previously been into the releaseduty cycle. Starting from the baseline, people coming back after N
cycles can ramp up incrementally with the latest changes.
As releaseduty squirrels are the ones with the best context when it comes to releases, they are the ones to edit this page and amend it accordingly. Keep in mind that changes should be in compliance with the other pieces of documentation.
During 119.0 >= 2023-08-28#
During 118.0 >= 2023-07-31#
During 117.0 >= 2023-07-04#
During 116.0 >= 2023-06-05#
During 115.0 >= 2023-05-09#
During 114.0 >= 2023-04-11#
During 113.0 >= 2023-03-14#
langpack submission switched to v5 of the AMO API, so manual action is no longer needed when a new language is added
During 112.0 >= 2023-02-14#
new langpacks caused bustage, bug 1817344, https://moz-releng-docs.readthedocs.io/en/latest/addons/langpacks.html?highlight=addon#how-to-handle-new-languages-for-release
During 111.0 >= 2023-01-16#
[Jan 2023 rotation] Change all the things
tooltool.py synchronization and consolidation
Widevine native aarch64, bug 1812343
During 110.0 >= 2022-12-12#
Focus moved to the new mono-repo, https://github.com/mozilla-mobile/firefox-android/ (bug 1797704)
scriptworker 46.0.0 released which removes 2021 CoT public key
Balrog v3.21 deployed to production
During 109.0 >= 2022-11-14#
Added repackage mac .pkg (bug 1798589)
Widevine update, (bug 1795749)
During 108.0 >= 2022-10-17#
Android components moved to the new mono-repo, https://github.com/mozilla-mobile/firefox-android/ (bug 1797702)
During 107.0 >= 2022-08-20#
During 106.0 >= 2022-08-22#
Adding new
base_rev
andbase_ref
parameters in taskgraph (bug 1784232)Deployed new
build-decision
image (D155195)List to be expanded once new changes are deployed.
Balrog 3.20 deployed
During 105.0 >= 2022-07-25#
Deployed new iscript + scriptworker version to mac signers
FirefoxCI cluster upgraded to Taskcluster 44.18.0
relduty no longer announces minor upgrades on dev-platform
During 104.0 >= 2022-06-28#
update-verify sends requests with “Cache-Control: max-stale=0” to avoid getting out of date responses from the CDN fronting balrog (https://bugzilla.mozilla.org/show_bug.cgi?id=1778728)
Android:
Android-Components gets released off of https://shipit.mozilla-releng.net/ instead of Github Releases (RELENG-11)
Fenix/Focus are not immediately shipped anymore. We now have a 2-step process where builds are promoted and sent to our testing population before shipping them to Github or archive.mozilla.org (bug 1776830 and bug 1778056)
Version number now follow the gecko scheme. This means 104.0.0-beta.1 will instead be called 104.0b1 (bug 1777255)
L10n string uplift are automatically approved by mergify (bug 1779797)
Android-Components bumps are also automatically approved (bug 1780740)
During 103.0 >= 2022-05-31#
FirefoxCI cluster upgraded to Taskcluster 44.14.0
https://github.com/mozilla/delivery-dashboard/ deployed (first time since 2018!)
Deployed balrog v3.18 to production (https://bugzilla.mozilla.org/show_bug.cgi?id=1772799)
(ahal) Copied new scriptworker CoT private key to mac signers
During 102.0 >= 2022-05-02#
(ahal) Deployed scriptworker pools to pick up new CoT private key
(ahal) Updated dependencies in image-task docker image
Treestatus updated for esr-102 and deployed
signingscript update for authenticode timestamping
scriptworker-scripts and shipit updated for esr-102 and deployed
new docker-worker pools to fix ssl certs
Widevine 4.10.2449.0 final rollout
aus5.mozilla.org (balrog prod client domain) certificate renewed (https://bugzilla.mozilla.org/show_bug.cgi?id=1768253
FirefoxCI cluster upgraded to Taskcluster 44.13.7
During 101.0 >= 2022-04-05#
Updated Mac certificates on all prod mac hosts. + Nightly builds now use the same keychain file as release.
FirefoxCI cluster upgraded from Taskcluster 44.8.4 to 44.11.1; Postgres DB instance doubled in RAM.
Updated all iOS apps in Bitrise to use new cert and provisioning profiles
https://pypi.org/project/mozilla-version/ updated to 1.0.1 -> beetmover, bouncer, pushapk, treescript deps updated
Experimental change to timestamp server for signing windows firefox, https://github.com/mozilla-releng/scriptworker-scripts/pull/501/
Allowed System Addons to be served from archive.m.o, https://github.com/mozilla-releng/balrog/pull/2520
During 100.0 >= 2022-03-08#
Deployed Balrog v3.16 to production
FirefoxCI cluster upgraded from 43.2.0 -> 44.8.1
beetmover-geckoview
jobs (copying geckoview artifacts to maven) now run as part of thepromote_firefox
graphs on release and beta instead ofpush_firefox
Switched SENTRY_DSN for Balrog -> sentry.io
SHA-256 signing of Windows Installers, bug 1745467
Widevine update, bug 1758423 (nightlytest, nightly, and beta channels)
Treescript merge day CLOBBER datestring
During 99.0 >= 2022-02-07#
Changed#
Internal pypi uploads for numpy/scipy/pillow, bug 1757792
Deployed pollbot v1.4.7 to production
Deployed balrog v3.14 to production
Deployed balrog v3.13 to production
Minor refactor to release_artifacts in
gecko_taskgraph
Re-imaged mac-v1-signing3 and mac-v1-signing8. Quarantined mac-v1-signing19 for MozillaVPN work.
Signingscript retries for winsign (timestamp.digicert.com), bug 1590599
During 98.0 >= 2022-01-10#
Changed#
RELENG-659 - Py3 support on xpi/addons and updated the taskcluster version used
Change ShipIt to point to xpi-manifest “main” branch
Deployed new version of iscript to mac signers
Deployed balrog v3.12 to production
Deployed signingscript to production
Modified mac-signing releng ssh allowlist
During 97.0 >= 2021-12-06#
Changed#
l10n-cross-channel-quarantine now nukes comm/ at the end of the run
Updated puppet dependencies for mac signers
fixed treescript merge-automation to be idempotent
refactored the system addons balrog rules
During 96.0 >= 2021-11-01#
Changed#
removed pushsnap scriptworker pools
During 95.0 >= 2021-10-04#
Changed#
focus-android releases switched to taskgraph
firefox desktop update-verify watershed
new firefox desktop msix packages
update-verify async download PR
During 82.0 >= 2020-09-21#
Changed#
partner attribution configs landed
balrog v2 submission
ESR68 EOL
central-to-beta merges should automatically l10n-bump
During 81.0 >= 2020-08-24#
Changed#
L10n bumper will look at the hg pushlog for the default revision, rather than Elmo. If we need to pin a previous revision due to bustage, land a change updating the revision and changing
pin
totrue
for that locale.ESR68 l10n bumper disabled.
partner builds moved from b8 to b5 to adjust for shorter release cycles
partner attributions landed
During 80.0 >= 2020-07-27#
Changed#
Lots of accounts disabled; watch for fallout!
During 75.0 >= 2020-04-06#
Changed#
Changelog moved to rtd
Mac Notarization is now single-zip instead of multi-account due to bug 1620697. This may slow down the success case by a few minutes, but reduce or remove the multi-hour manual-intervention failure case.
Fixed#
Scriptworker now catches a number of
asyncio.TimeoutError
s, and should be more robust.
During 74.0 >= 2020-03-10#
Added#
added support for shipping Firefox Flatpaks to Flathub
During 73.0 >= 2020-02-10#
Changed#
py38 rollout for scriptworkers, tooltool, balrog, and other release services
aws -> gcp, then back again, for tier 1 workers
mac 3-tier notarization
During 71.0 >= 2019-10-14#
Changed#
(still-in-progress) signing servers are being shut down. we’ve already stopped using them, however
mergeduty processed is back to two stages like it used to be before Quantum: beta -> release in the first iteration while central -> beta and bumping central in the second week
l10n bumper for central is moving to taskcluster cron shortly; will uplift to beta and esr68 later
switched to GCP scriptworkers in all gecko trees (including esr68)
after TCW of 9th Nov, switched to GCP scriptworkers in
comm
repos too, but also Github mobile projectssecrets are now managed in
sops
, no longer inhiera
decommissioning of the AWS infrastructure
treescript has been refactored
scriptworker-scripts docker images now mount the repo at
/app
instead of the script (we may revisit this to lower disk usage)mergeduty process change for upcoming automated betas. tl;dr - no longer Devedition b1 and b2 during RC week. central merges to beta a week later
During 69.0 >= 2019-05-13#
Changed#
mac signing now happens on the mac notarization pool.
mac nightly and release builds are now notarized as well as signed.
mac signing tasks also create pkg installers
During 67.0 >= 2019-03-11#
Changed#
update & final verify will check SSL certificates of Balrog, archive.m.o etc
the release promotion action is now a hook.
chain of trust no longer uses gpg to create and verify signatures.
During 66.0 >= 2019-01-30#
Changed#
scriptworker 22 now signs and verifies ed25519 signatures; gpg signatures are deprecated.
partner repacks are now enabled on ESR60
adding list of whatsnewpage locales in-tree is now part of mergeduty
removed thunderbird docs. thunderbird is now managed by thunderbird team and they have support for shipit v2
During 64.0 >= 2018-10-15#
Changed#
MARs are now signed via autograph (hashes only)
Mergeduty tasks are now tracked in Trello
Releaseduty now assists CIDuty with puppet pyup PRs once per week
During 63.0 >= 2018-09-05#
Added#
added 6 new mac signing servers
Changed#
switched to ship-it v2
reduced mac signing concurrency from 4 to 2 to reduce server churn
During 62.0 >= 2018-06-26#
Changed#
switched to Ship-it v2 for beta releases
Added#
releasewarrior local automation to add WNP
During 61.0 >= 2018-05-09#
Added#
added partner off-cycle support + docs
widevine docs
Changed#
update verify only blocks on required platforms to start (eg: win32 update verify only blocks on win32 beetmover and balrog)
python 2.7.15 rollout to a portion of our infra
python 3.6.5 rollout completed to our scriptworkers
update verify no longer fails on expected differences, but will fail on any files found only in source or target; bug 1461490
During 60.0 >= 2018-03-13#
Added#
releaseduty now responsible for periodic file update approvals in phabricator. Change this line to update those responsible.
added
esr52-latest*
bouncer aliasesadded
firefox-esr-next-*
bouncer aliasesadded esr60 support
rerun action task in Treeherder - hopefully will help avoid duplicate reruns
KEY file to releases directory in bug 1446816 for >=60.0b11
added addonscript for langpack pushing to amo
Changed#
re-added source tarball for >=60.0b6
updated scriptworkers to share more client code
update verify now fails on any differences, and creates a summary log for >=60.0b8. additional documentation on how to interpret failures are available.
split snap repackage and push task in bug 1447263
updated rr3 add partner configs for fx60+
tc partner repacks!
split esr52 and esr60 templates
Fixed#
fixed beta+fennec bouncer aliases
push-apk-breakpoint removed in bug 1432817
During 59.0 >= 2018-01-24#
Added#
we track all major changes in this newly added CHANGELOG
separate RC docs for both Fennec and Firefox
new-style WNP setup, as documented.
we now have support for snaps in
release
channel
Changed#
old release promotion based on releasetasks is dead. Everything is in-tree scheduled now. Changes are riding the trains.
all releaseduty related documentation has been moved out of the wiki under warrior’s
docs
subfolder.use
release graphid
to enter graphids, rather than editing the .json manually.in-tree release notifications going away in favor of native taskcluster notifications. pulse-notify is to be retired whenever we kill esr52
Balrog toplevel submission and release scheduling now happens in balrog scriptworker for Fx60+.
During 58.0 >= 2017-11-15#
Added#
the use of the new releasewarrior
Changed#
old how-tos are now gathered under one roof in releasewarrior-2.0/old-howtosreleasewarrior-2.0/old-how-tos but they are gradually being migrated to the wiki which becomes the single source of truth
authentication method for taskcluster-cli changed. If you are upgrading from an older version of taskcluster-cli, you may have to remove the
${HOME}/.config/taskcluster.yml
file to work with the new authentication method.
Removed#
dropped support for old releasewarrior
dropped the need for QE signoffs in Balrog
During 57.0 >= 2017-09-27#
Added#
a merge day instance has been added to ease the handover during mergedays but also for speed and reliability. More on this here. When connecting to the merge day instance, keep in mind to ensure its environment is clean and ready for use
simplified documentation for mid-betas and release checklists. If not already, duplicate an existing sheet in this google docs checklist and clear out the status that was carried over from the previous release.
Changed#
mergeduty dates shifted. We now merge
beta
torelease
andcetral
tobeta
in the same day, keeping a relbranch forbeta
and holdcentral
from version bump until before the release.because of 57 and the aforementioned mergeduty change, the exception becomes rule so that for each new beta cycle X, we have X.0b1 and X.b2 just for
Devedition
, whileFirefox
starts at X.b3
Removed#
dropped support for tctalker and solely rely on taskcluster-cli. More on its installation here
During 56.0 >= 2017-08-09#
Added#
releaseduty now checks on status of central nightlies when not undertaking other duties
releaseduty now handles signing of arbitrary .apk’s as requested by MoCo teams (Klar/Focus, Rocket)
Changed#
release-drivers mailing list is no longer used for signoffs/gtb’s, please subscribe to release-signoffs@
QE will now send an explicit GO for DevEdition Beta’s to be pushed live, usually in the same e-mail as the GO for Beta
Removed#
Fennec single locales no longer shipped