Release Duty

Contents:

Releaseduty is a role within the Release Engineering team. It is conducted on a rolling rotation matching one person against one release cycle (currently 4 weeks). Below you will find a description of the expectations and resources needed to do the role.

Expectations

If you’re reading this, it means you’re ramping up as an official releaseduty squirrel within Mozilla RelEng, so please allow us to give you a warm welcome!

The role mainly involves handling all the coordination and communication with other teams as well as doing all the operational tasks to make sure the release workflow is as smooth as possible.

While this role can get quite disruptive, we prefer this approach of assigning the responsibility to a small set of people who will own all the tasks, while we shield the others in Release Engineering from interruptions.

Being ReleaseDuty means a couple of things: - Communication and coordination with other teams - Handle all incoming releases - Fix and debug any potential errors in the automation - Develop and improve the Release Automation process and tools

Communication

Slack Channels

Join Mozilla’s Slack network using information from the wiki

You ought to be present and pay attention to conversations happening in:

  • #releaseduty-mobile (where mobile and other Github releng-backedup automation projects’s teams raise issues)

  • #releng-notifications (where automation notifications are going, useful to know the state of our automation infrastructure state)

  • #taskcluster-cloudops (where CloudOps double-checks with RelEng whether it’s safe to deploy changes or not on their side)

Matrix Channels

Join Mozilla’s Matrix network using information from the wiki

You ought to be present and pay attention to conversations happening in:

  • #sheriffs:mozilla.org (where CIDuty team helps with various hiccups that infra might encounter))

  • #releaseduty:mozilla.org (main RelEng dedicated communication channel for releaseduty)

  • #firefox-ci:mozilla.org

Email

As ReleaseDuty you need to subscribe to certain mailing lists:

These other mailing lists will likely have useful discussions and information: * RelEng internal mailing list * Thunderbird release drivers mailing list * release-automation-notifications-thunderbird mailing list * (optional) release-automation-notifications-dev mailing list

Meetings and Calendars

Regular meetings are a vital part of making sure all the teams are kept informed and consulted during the release process. To view those meetings in your calendar you need to subscribe/be added to the following calendars:

If you join a calendar and it’s blank, you may need to delete it and get a calendar invitation from an existing subscriber

Permissions

Several tools for managing releases are protected or private. In order to do your job, you need to be granted access to a bare minimum:

How to get VPN access

First you need to be connected to the Mozilla VPN. See the mana page <https://mana.mozilla.org/wiki/display/SD/VPN> for how to get set up and started with connecting to the VPN.

How to get Ship-it access

You need to be added to the vpn_cloudops_shipit LDAP group, as well as be added to the allowedlist in the shipit config. Both of these can be done by filing a ticket requesting access.

File an Release Engineering bug under the Applications: Shipit (backend) <https://bugzilla.mozilla.org/enter_bug.cgi?format=__default__&cloned_bug_id=1626312&product=Release%20Engineering&component=Applications%3A%20Shipit%20%28backend%29> component requesting to be granted shipit access.

Have someone in Release Engineering to vouch for you in the bug.

How to get Balrog access

Similar to Shipit, you need to be added to the balrog and vpn_balrog LDAP groups and have someone add you as a user and attach you to the releng role through the Balrog admin UI.

For VPN group access, file a ticket with IT similar to this one. Have someone in Release Engineering to vouch for you in the bug.

Next, file a Github issue within the Balrog repo. Ask to be added to the Balrog admin user list and attached to the releng role. Also have someone vouch for you.

Tooling for debugging and rerunning tasks

taskcluster

Release tasks are usually run through Taskcluster, which has a useful Command-line interface

  • Download an appropriate binary from https://github.com/taskcluster/taskcluster-cli#installation

  • Copy the binary somewhere useful, such as somewhere in your `$PATH <http://www.linfo.org/path_env_var.html>`__

  • Make it executable, if using Mac or Linux: chmod a+x /path/to/taskcluster

  • taskcluster signin

  • this will open a browser window and allow you to get temporary client credentials. By default this is valid for 24 hours. The command will display two ``export`` commands you must copy/paste into your shell

  • Familiarize yourself with the subcommands, starting with taskcluster help

Note many of the taskcluster CLI commands can be conducted through Treeherder such as rerunning a task.

Firefox bookmarks

These bookmarklets should help you view tasks and taskgroups in Firefox.

  • Go to Bookmarks -> Show All Bookmarks

  • Gear symbol -> New Bookmark

  • Name: task inspector Location: https://tools.taskcluster.net/tasks/%s ; Keyword: task

  • Name: taskgroup inspector Location: https://tools.taskcluster.net/groups/%s ; Keyword: taskgroup

  • Name: stop Location: javascript:stop(); * This can be used to stop further loading in the Task Group Inspector. It shouldn’t be used when actively monitoring (i.e.: watching for failures), but it can greatly speed things up if you’re using it for other reasons. Be sure to wait for the initial tasks to load before you use it.

Now if you go to your URL bar, you can type task TASKID or taskgroup TASKGROUPID and you’ll go to that task or taskgroup in the inspector.

After ReleaseDuty

After your tour of releaseduty, it’s customary to spend 1-2 weeks fixing release automation issues. Check the Release Automation Improvements trello board trello board for ideas of what to work on and to add new items as you discover them.

Ensure the next duty cycle have signed up to any phabricator reviews, such as the periodic file updates reviews.

Miscellaneous

Teams

Hand Off

If a scheduled release has not completed its graphs prior releaseduty signing off, an explicit hand-off describing describing release state should be sent to individual folks in releng that are scheduled to come online next or will be around for a while after you. #mozbuild in Slack is best. A release@m.c email would be useful too.

Escalation

If a release is blocked. The normal flow is to:

  1. confirm issue

  2. determine what service, task kind, infrastructure, or external dependency is involved

  3. file a ticket

  4. determine which team(s) and person(s) should be escalated.

    1. Searching phonebook is useful for org and ownership charts.

    2. bugzilla and github history

    3. source code history

  5. escalate in the appropriate Slack and Matrix channel(s). At a minimum, #releaseduy@matrix.

  6. determine who is available to help based on above. What hours they work, who is their manager, etc

  7. ask for help if you can’t determine the above.

Good resources within releng:

  • general release configuration (taskgraph): tomprince/callek

  • scopes / ciadmin: tomprince/mtabara

  • chainoftrust (cot): aki

  • scriptworker (general): aki/jlorenzo

  • beetmoverscript / bouncer / artifact related: mtabara

  • signing / signingscript / autograph: aki/catlee

  • balrog / balrogscript / updates related: bhearsum/nthomas

  • l10n / treescript / addonscript: callek

  • pushapkscript / mozapkpublisher: jlorenzo

  • shipit / shipitscript: rail

Other useful resources

Glossary

  • WNP - The “What’s New Page” can be set to appear after an upgrade, to tell end-users of any changes in the browser they should be aware of.

  • FF - Firefox

  • TB - Thunderbird

  • b1, b2, etc - beta release 1, beta release 2, etc

FAQ —

  1. What does release-promotion refer to?

‘Release promotion’ is simply the idea that we take an already existing CI build from (e.g. beta) and promote that to being the build we release/ship to users. Prior to this approach, we had always rebuilt Firefox at the start of each new release. Long story short, release promotion entails taking an existing set of builds that have already been triggered and passed QA and “promoting” them to be used as a release candidate. More on promotion can be found on our docs here

  1. What is the train model?

Since 2012 Mozilla moved to a fixed-schedule release model, otherwise known as the Train Model, in which we released Firefox every six weeks to get features and updates to users faster and move at the speed of the Web. Hence, every six weeks the following merges take place: mozilla-beta => mozilla-release mozilla-central => mozilla-beta

We used to have an intermediate branch named ‘aurora’ in between central and beta but that was brought to end-of-life during April-May 2017. Instead, early beta releases are branded as ‘DevEdition’.

  1. What is a partner repack change for FF?

Partner repacks refer to 3rd party customized branded versions of Firefox that Mozilla is taking care of for some of its clients. With some exceptions, most of the partner reconfigs lie under private repositories. Mostly, the partner repacks don’t need too much of RelEng interference as all bits are held under private git repos and are directly handled by the partnering companies

  1. Is there calendar-based release scheduled for Thunderbird as for Firefox?

No. It’s irregular. Conversations happen on #tbdrivers and TB mailing list and they trigger their release in Ship-it.

  1. Why don’t I see update_verify_beta for dot releases?

From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from a normal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to the release channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release and update_verify_beta steps. Upon a successful testing phase we ship the RC on the beta channel and then on the release channel, following which we merge the code for the next release cycle so that the beta release bumps its version. In the lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time after the official release. For that reason, a dot release can’t be tested in beta channel as the at-that-moment beta version is greater than the dot release version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify for dot releases (update_verify_release == update_verify in this case).

  1. Is there explicit signoff from RelMan for DevEdition builds?

No, after b1, there isn’t signoff from RelMan on DevEdition builds. QA only verifies the DevEdition builds every two weeks. With the exception of b1, and assuming all the tasks complete as expected, the DevEdition builds should be shipped at the same time as we receive signoff for the corresponding desktop builds.

  1. How should I inform the ReleaseDuty team of recent changes in automation that may impact an upcoming release?

You can mention it to the current ReleaseDuty folks in the #releaseduty channel. Please also add it to the upcoming release in the ../releases/FUTURE/ dir. See future release support for more details.

  1. How do I coordinate with marketing on release day?

Join the #release-coordination channel on Mozilla Slack

  1. What is cdntest and localtest?

-cdntest and -localtest channels serve releases from the releases directory (mirrors or CDN) or candidates directory depending on the release and channel. They are testing channels used before we serve from the real update channel, but they use the actual files that will be served once a release is published.

  1. What’s the difference between Firefox and DevEdition?

In the beta cycle, Firefox and Devedition are different products built based on the same in-tree revision. Their functionality is the same but branding options differ.

  1. What do the terms releases directory, mirrors and CDN mean?

releases directory, mirrors and CDN are different terms for the same concept - the CDN from which shipped releases are served.