Releaseduty is a role within the Release Engineering team. It is conducted on a rolling rotation matching one person against one release cycle (currently 4 weeks). Below you will find a description of the expectations and resources needed to do the role.
If you’re reading this, it means you’re ramping up as an official releaseduty squirrel within Mozilla RelEng, so please allow us to give you a warm welcome!
The role mainly involves handling all the coordination and communication with other teams as well as doing all the operational tasks to make sure the release workflow is as smooth as possible.
While this role can get quite disruptive, we prefer this approach of assigning the responsibility to a small set of people who will own all the tasks, while we shield the others in Release Engineering from interruptions.
Being ReleaseDuty means a couple of things: - Communication and coordination with other teams - Handle all incoming releases - Fix and debug any potential errors in the automation - Develop and improve the Release Automation process and tools
Join Mozilla’s Slack network using information from the wiki
You ought to be present and pay attention to conversations happening in:
#releaseduty-mobile (where mobile and other Github releng-backedup automation projects’s teams raise issues)
#releng-notifications (where automation notifications are going, useful to know the state of our automation infrastructure state)
#taskcluster-cloudops (where CloudOps double-checks with RelEng whether it’s safe to deploy changes or not on their side)
Join Mozilla’s Matrix network using information from the wiki
You ought to be present and pay attention to conversations happening in:
#sheriffs:mozilla.org (where CIDuty team helps with various hiccups that infra might encounter))
#releaseduty:mozilla.org (main RelEng dedicated communication channel for releaseduty)
As ReleaseDuty you need to subscribe to certain mailing lists:
All types of sign-offs and approvals should go to release signoff mailing list
All releng automation emails should go to release-automation-notifications
All discussion topics should go to release drivers mailing list
These other mailing lists will likely have useful discussions and information: * RelEng internal mailing list * Thunderbird release drivers mailing list * release-automation-notifications-thunderbird mailing list * (optional) release-automation-notifications-dev mailing list
Meetings and Calendars¶
Regular meetings are a vital part of making sure all the teams are kept informed and consulted during the release process. To view those meetings in your calendar you need to subscribe/be added to the following calendars:
Public- Release Engineering (so that you attend the weekly post-mortem meeting)
Releases Scheduling (so that you can attend the Tuesday/Thursday channel meetings. You can add it following RelMan’s docs) – If their instructions don’t work, try to the “Add to Google Calendar” button at the web version of the calendar.
If you join a calendar and it’s blank, you may need to delete it and get a calendar invitation from an existing subscriber
Several tools for managing releases are protected or private. In order to do your job, you need to be granted access to a bare minimum:
How to get VPN access¶
First you need to be connected to the Mozilla VPN. See the mana page <https://mana.mozilla.org/wiki/display/SD/VPN> for how to get set up and started with connecting to the VPN.
How to get Ship-it access¶
You need to be added to the vpn_cloudops_shipit LDAP group, as well as be added to the allowedlist in the shipit config. Both of these can be done by filing a ticket requesting access.
File an Release Engineering bug under the Applications: Shipit (backend) <https://bugzilla.mozilla.org/enter_bug.cgi?format=__default__&cloned_bug_id=1626312&product=Release%20Engineering&component=Applications%3A%20Shipit%20%28backend%29> component requesting to be granted shipit access.
Have someone in Release Engineering to vouch for you in the bug.
How to get Balrog access¶
Similar to Shipit, you need to be added to the balrog and vpn_balrog LDAP groups and have someone add you as a user and attach you to the releng role through the Balrog admin UI.
Next, file a Github issue within the Balrog repo. Ask to be added to the Balrog admin user list and attached to the releng role. Also have someone vouch for you.
Tooling for debugging and rerunning tasks¶
Note many of the task-related operations can be conducted through Treeherder such as rerunning/retriggering a task. There is also a command line tool that can be used instead of the UI.
Download an appropriate binary from here
Copy the binary somewhere useful, such as somewhere in your $PATH
Make it executable, if using Mac or Linux:
chmod a+x /path/to/taskcluster
taskcluster signinto open a browser window and allow you to get temporary client credentials. By default this is valid for 24 hours. The command will display two ``export`` commands you must copy/paste into your shell
Familiarize yourself with the subcommands, starting with
in-bulk taskcluster operations¶
Sometimes operations need to be performed against a bulk of tasks. In
order to do that, RelEng has developed a handful of scripts to ease the
operations. They lie around in the
hg clone https://hg.mozilla.org/build/braindump/ cd braindump/taskcluster mkvirtualenv workspace pip install taskcluster
Follow this resource to download the
taskcluster-cli tool or read
the previous section.
taskcluster-cli is installed, ensure right env vars are set,
login and operate the tasks.
export TASKCLUSTER_ROOT_URL='https://firefox-ci-tc.services.mozilla.com/' eval taskcluster signin python tc-filter.py --graph-id <group-task-id> --state failed --action rerun
To speed things up even more, one can add this to
zshrc for an alias
to reduce the scopes and time-limit of the signin:
tc-relduty=$'eval `TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com/ taskcluster signin --expires 1h -s "queue:rerun-task:*\nqueue:cancel-task:*"`'
Inspecting a task group from the UI can sometimes be heavy for the browser. Instead, one can use a script to watch the graph progress locally via poll & sleep. In the same braindump aforementioned directory, there is a graph-progress script that can be run like:
bash graph-progress.sh <TASK-GROUP-ID>
These bookmarklets should help you view tasks and taskgroups in Firefox.
Go to Bookmarks -> Show All Bookmarks
Gear symbol -> New Bookmark
task inspectorLocation: https://tools.taskcluster.net/tasks/%s ; Keyword:
taskgroup inspectorLocation: https://tools.taskcluster.net/groups/%s ; Keyword:
Now if you go to your URL bar, you can type
task TASKID or
taskgroup TASKGROUPID and you’ll go to that task or
taskgroup in the inspector.
After your tour of releaseduty, it’s customary to spend 1-2 weeks fixing release automation issues. Check the Release Automation Improvements trello board trello board for ideas of what to work on and to add new items as you discover them.
Ensure the next duty cycle have signed up to any phabricator reviews, such as the periodic file updates reviews.
If a scheduled release has not completed its graphs prior releaseduty signing off, an explicit hand-off describing describing release state should be sent to individual folks in releng that are scheduled to come online next or will be around for a while after you. #mozbuild in Slack is best. A email@example.com email would be useful too.
If a release is blocked. The normal flow is to:
determine what service, task kind, infrastructure, or external dependency is involved
file a ticket
determine which team(s) and person(s) should be escalated.
Searching phonebook is useful for org and ownership charts.
bugzilla and github history
source code history
escalate in the appropriate Slack and Matrix channel(s). At a minimum, #releaseduy@matrix.
determine who is available to help based on above. What hours they work, who is their manager, etc
ask for help if you can’t determine the above.
Good resources within releng:
general release configuration (taskgraph): aki/jlorenzo
scopes / ciadmin: jlorenzo/mtabara
chainoftrust (cot): aki
scriptworker (general): aki/jlorenzo
beetmoverscript / bouncer / artifact related: mtabara/aki
signing / signingscript / autograph: aki/catlee
balrog / balrogscript / updates related: bhearsum/mtabara
l10n / treescript / addonscript: aki/mtabara
pushapkscript / mozapkpublisher: jlorenzo/mtabara
shipit / shipitscript: bhearsum/jlorenzo
WNP - The “What’s New Page” can be set to appear after an upgrade, to tell end-users of any changes in the browser they should be aware of.
FF - Firefox
TB - Thunderbird
b1, b2, etc - beta release 1, beta release 2, etc
What does release-promotion refer to?
‘Release promotion’ is simply the idea that we take an already existing CI build from (e.g. beta) and promote that to being the build we release/ship to users. Prior to this approach, we had always rebuilt Firefox at the start of each new release. Long story short, release promotion entails taking an existing set of builds that have already been triggered and passed QA and “promoting” them to be used as a release candidate. More on promotion can be found on our docs here
What is the train model?
Since 2012 Mozilla moved to a fixed-schedule release model, otherwise known as the Train Model, in which we released Firefox every six weeks to get features and updates to users faster and move at the speed of the Web. Hence, every six weeks the following merges take place: mozilla-beta => mozilla-release mozilla-central => mozilla-beta
We used to have an intermediate branch named ‘aurora’ in between central and beta but that was brought to end-of-life during April-May 2017. Instead, early beta releases are branded as ‘DevEdition’.
What is a partner repack change for FF?
Partner repacks refer to 3rd party customized branded versions of Firefox that Mozilla is taking care of for some of its clients. With some exceptions, most of the partner reconfigs lie under private repositories. Mostly, the partner repacks don’t need too much of RelEng interference as all bits are held under private git repos and are directly handled by the partnering companies
Is there calendar-based release scheduled for Thunderbird as for Firefox?
No. It’s irregular. Conversations happen on #tbdrivers and TB mailing list and they trigger their release in Ship-it.
Why don’t I see update_verify_beta for dot releases?
From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from a normal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to the release channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release and update_verify_beta steps. Upon a successful testing phase we ship the RC on the beta channel and then on the release channel, following which we merge the code for the next release cycle so that the beta release bumps its version. In the lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time after the official release. For that reason, a dot release can’t be tested in beta channel as the at-that-moment beta version is greater than the dot release version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify for dot releases (update_verify_release == update_verify in this case).
Is there explicit signoff from RelMan for DevEdition builds?
No, after b1, there isn’t signoff from RelMan on DevEdition builds. QA only verifies the DevEdition builds every two weeks. With the exception of b1, and assuming all the tasks complete as expected, the DevEdition builds should be shipped at the same time as we receive signoff for the corresponding desktop builds.
How should I inform the ReleaseDuty team of recent changes in automation that may impact an upcoming release?
You can mention it to the current ReleaseDuty folks in the #releaseduty channel. Please also add it to the upcoming release in the ../releases/FUTURE/ dir. See future release support for more details.
How do I coordinate with marketing on release day?
Join the #release-coordination channel on Mozilla Slack
What is cdntest and localtest?
-localtest channels serve releases from the releases
directory (mirrors or CDN) or candidates directory depending on the release and
channel. They are testing channels used before we serve from the real update
channel, but they use the actual files that will be served once a release is
What’s the difference between Firefox and DevEdition?
In the beta cycle,
Devedition are different products
built based on the same in-tree revision. Their functionality is the
same but branding options differ.
What do the terms
CDN are different terms for the same
concept - the CDN from which shipped releases are served.