Merge Duty

All code changes to Firefox land in the mozilla-central repository while the Fennec counterpart land to mozilla-esr68.

  • The nightly releases are built from that repo twice a day.

  • DevEdition and Beta releases are built from the beta repository

  • Extended Support Releases follow-up from the relevant ESR repo, such as mozilla-esr68

  • Release and Release Candidates are built from mozilla-release repository

How are those repositories kept in sync? That’s MergeDuty and is part of the releaseduty responsibility.

Overview of Procedure

MergeDuty consists of multiple separate days of work. Each day you must perform several sequential tasks. The days are spread out over nearly three weeks, with three major days of activity:

Historical context of this procedure:

Originally, the m-c -> m-b was done a week after m-b -> m-r. Starting at Firefox 57, Release Management wanted to ship DevEdition b1 week before the planned mozilla-beta merge day. This meant Releng had to merge both repos at the same time. With 71.0, we’re back to the initial workflow with merging m-b -> m-r in the first week and then m-c -> m-b in the follow-up week.

Do the prep work a week before the merge

Assign to yourself the migration bug

A tracking migration bug should’ve already been filed - please assign that to yourself. If there isn’t one (e.g. bug 1694412), please contact the last releaseduty owner and file one. You can find more of this in Release owners.

Do migration no-op trial runs

Doing a no-op trial run of each migration has one major benefit these days: you ensure that the migrations themselves work prior to Merge day.

General steps

  1. Go to Treeherder.

  2. Select the repo depending on the merge you want to perform (central, beta or the ESR one).

  3. On the latest push, click on the down arrow at the top right corner.

  4. Select “Custom push action…”

  5. Choose merge-automation

  6. In Treeherder, you’ll see a new push show up in Treeherder in the repo you will be merging to. It can take a few minutes for the push and task to appear.

  7. Click on the merge or bump tasks (not the Gecko decision task). A job details panel will pop up and from there you’ll find a link to the diff file in the artifacts tab.

mozilla-beta->mozilla-release migration no-op trial run

  1. Follow the general steps hopping on beta

  2. Insert the following payload and click submit.

force-dry-run: true
behavior: beta-to-release
push: true

mozilla-central->mozilla-beta migration no-op trial run

  1. Follow the general steps hopping on central

  2. Insert the following payload and click submit.

force-dry-run: true
behavior: central-to-beta
push: true

mozilla-esr bump no-op trial run

  1. Follow the general steps hopping on esr

  2. Insert the following payload and click submit.

force-dry-run: true
behavior: bump-esr
push: true

Diff should be similar to this esr68 one or this esr78 one.

Sanity check no blocking migration bugs

Make sure the bug that tracks the migration has no blocking items.

Land whatsnewpage list of locales

As of 84.0 this is no longer the case. Please double-check with RelMan whether this still needs attention - the l10n drivers provide the final list of locales to receive the WNP on the Tuesday prior to the ship date.

  1. For each release, there should already be a bug flying around named Setup WNP for users coming from < X and receiving the X release. Find it for the current release. e.g. Bug 1523699. We should always aim to chain this bug to our main mergeduty tracking bug. That is, block the WNP bug against the tracking XXX migration day. If not already, please do so. This way, it’s easier to find deps and navigate via bugs.

  2. By the Friday prior to merge day, the l10n (most likely Peiying Mo [:CocoMo]) team will have posted the final list of locales for whatsnewpage. Double-check with them again to make sure that is the final list. The list of locales comes in two forms: attachment in bug directly to be hg imported, but also as a comment. Make sure to double-check they match as that’s generated automatically and sometimes there could be fallout resulting in mismatches.

  3. Update the in-tree whatsnewpage list of locales on central and request an uplift of that to beta. Similar to this patch. It will uplift to release when the merge happens on Monday

    1. On development machine, update browser/config/whats_new_page.yml with the list of locales from the bug

    2. Commit the change and create Phabricator patch request as usual

    3. Once the patch request is approved, land the patch via lando

    4. In Bugzilla edit the phabricator attachment and add a approval-mozilla-beta? flag similar to this

    5. ensure someone from sheriffs or relman uplift this to Beta before Monday’s merge and RC go-to-build

Release Merge Day - part I

When: Wait for go from relman to release-signoff@mozilla.com. Relman might want to do the migration in two steps. Read the email to understand which migration you are suppose to do, and then wait for second email. For date, see Release Scheduling calendar or check with relman

Merge beta to release

  1. Close mozilla-beta. Check “Remember this change to undo later”. Please enter a good message as the reason for the closure, such as “Mergeduty - closing beta for $VERSION RC week”.

  2. Run the m-b -> m-r no-op trial run one more time, and show the diff to another person on releaseduty.

  3. The diff for release should be fairly similar to this, with updated the version change.

  4. Submit a new task with force-dry-run set to false:

force-dry-run: false
behavior: beta-to-release
push: true
warning

It’s not unlikely for the push to take between 10-20 minutes to complete.

warning

If an issue comes up during this phase, you may not be able to run this command (or the no-op one) correctly. You may need to publicly backout some tags/changesets to get back in a known state.

  1. Upon successful run, mozilla-release should get a version bump and branding changes consisting of a commit like this and a tag like this

  2. In the same time mozilla-beta should get a tag like this

  3. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

warning

The decision task of the resulting pushlog in the mozilla-release might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.

Reply to relman migrations are complete

Reply to the migration request with the template:

This is now complete:
* mozilla-beta is merged to mozilla-release, new version is XX.Y
* beta will stay closed until next week

Release Merge Day - part II - a week after Merge day

When: Wait for go from relman to release-signoff@mozilla.com. For date, see Release Scheduling calendar or check with relman

Merge central to beta

  1. Run the m-c -> m-b no-op trial run one more time, and show the diff to another person on releaseduty.

  2. The diff generated by the task should be fairly similar to this.

  3. Submit a new task with force-dry-run set to false:

force-dry-run: false
behavior: central-to-beta
push: true
warning

It’s not unlikely for the push to take between 10-20 minutes to complete.

  1. Upon a successful run, mozilla-beta should get a version bump and branding changes consisting of a commit like this and a tag like this. Click the first HG revision link (left side under date and timestamp) for the merge push to verify this.

  2. Verify that browser/locales/l10n-changesets.json has revisions, not default, and/or verify that the merge task has l10n-bump in the logs. You’ll need to click on the second HG revision link (commit message will be something like “no bug - Bumping Firefox |10n…”) to verify this. The diff should look like this

  3. In the same time mozilla-central should get a tag like this

  4. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

warning

The decision task of the resulting pushlog in the mozilla-beta might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.

Re-opening the tree(s)

Restore mozilla-beta tree to its previous state (approval-required) so that l10n bumper can run.

Tag central and bump versions

What happens: A new tag is needed to specify the end of the nightly cycle. Then clobber and bump versions in mozilla-central as instructions depict.

  1. Follow the general steps

  2. Insert the following payload and click submit.

force-dry-run: false
push: true
behavior: bump-central
  1. Upon successful run, mozilla-central should get a version bump consisting of a commit like this and a tag like this

  2. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

Bump ESR version

Note: You could have one ESR to bump, or two. If you are not sure, ask.

Run the bump-esr no-op trial run one more time, and show the diff to another person on releaseduty.

Diff should be similar to this one.

Push your changes generated by the no-op trial run:

  1. Follow the general steps - (As of 2020/04 this action hasn’t yet been uplifted to release or esr68, consider using mozilla-central‘s action, as the payload controls where the effects land)

  2. Insert the following payload and click submit.

force-dry-run: false
push: true
behavior: bump-esr

Note This is currently set to esr78, the defaults can be overridden in-tree in taskcluster/ci/config.yml or specified here as using an action payload such as:

force-dry-run: false
push: true
behavior: bump-esr
to-branch: esr88
to-repo: https://hg.mozilla.org/releases/mozilla-esr88
  1. Upon successful run, mozilla-esr${VERSION} should get a commit like this.

  2. Verify new changesets popped on https://hg.mozilla.org/releases/mozilla-esr78/pushloghtml

Reply to relman central bump completed

Reply to the migration request with the template:

This is now complete:
* mozilla-central is merged to mozilla-beta, new version is XX.Y
* mozilla-central has been tagged and version bumped
* mozilla-esr has been version bumped
* newly triggered nightlies will pick the version change on cron-based schedule

Update wiki versions

  1. Edit the new values manually:

Bump Nightly version and release dates in ShipIt

ShipIt currently hard-codes the version of Nightly that’s being released, as well as the release dates.

It doesn’t get automatically updated because it would need to know when a new nightly was available, not just when the version had been updated in-tree. Everything up to merging this pull request can be done early, but the PR must not be merged before the first nightly has been built and published with the new version.

  1. git clone git@github.com:mozilla-releng/shipit.git

  2. git checkout -b nightly_version_bump_${version}

  3. Edit FIREFOX_NIGHTLY’s major version in https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L49

  4. Edit the LAST and NEXT known dates (all 6 of them) at https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L54-L59

  5. Commit, and submit a pull request

  6. Merge the pull request after a new nightly version has been pushed to CDNs

Close migration bug, file one for the next release

Once release is out of the door on Tuesday, close the existing bug tracking this release, from initial step and clone that bug into a similar one, tracking the next release. Please CC all the RelEng team. One can find the next release date in Release owners.