Run staging releases#
We have a number of tasks that _only_ run on Betas, Releases, and ESRs. Because of this, these tasks are at high risk of failure when we uplift a new Gecko version to mozilla-beta. Many scriptworkers fall into this category, but certain verification jobs, like update verify, are also notably included.
When making changes to tasks like this staging releases are the most effective way to ensure that your changes are correct, and haven’t regressed anything.
Permissions#
To do a staging release, you need to be able to push to try, access to the VPN, and have staging shipit read/write access.
See the production documentation for how to get Shipit and VPN access.
How-To#
For central to beta migration
hop on
central
repositorymake sure you’re up to date with the tip of the repo
mach try release --version <future-version.0b1> --migration central-to-beta --tasks staging --disable-pgo
For beta to release migration
hop on
beta
repositorymake sure you’re up to date with the tip of the repo
mach try release --version <future-version.0> --migration beta-to-release --tasks staging --disable-pgo
Note
Get future-version
from shipit-staging. Ie.: If the version in shipit is 94.0b14
use 94.0b15
These will create try pushes that look-alike the repos once they are merged. Once the decision tasks of the newly created CI graphs are green, staging releases can be created off of them via the shipit-staging instance. For how to create a release via Shipit, refer to the production documentation. The same applies to staging, just ensure you are using the staging instance (https://shipit.staging.mozilla-releng.net).
Unless you are specifically testing something to do with updates it is common and acceptable to disable partials when submitting the release. If you are enabling updates take care to choose a version that is listed in all of: Ship It Stage, Balrog Stage, Stage archive (ftp.stage.mozaws.net). There’s a helper script in the braindump repository that can assist you with this.
Once the staging releases are being triggered, it’s highly recommended
that at least a comment is being dropped to Sheriffs team
(e.g. Aryx
) to let them know these are happening in order to:
avoid stepping on each others toes as they may run staging releases as well
make sure we’re up-to-date to recent patches that they may be aware of
- warning:
Allow yourself enough time to wait for these staging releases to be completed. Since they are running in
try
, they have the lowest priority even on the staging workers so it usually takes longer for them to complete.
Faster staging release thanks to existing try builds#
Sometimes, RelEng needs to test quick fixes that are unrelated to the builds themselves.
Thus, mach try release
adds unnecessary delays. If the fix is not in-tree then it’s
a matter of rerunning an existing task. However, when the fix is in-tree, it’s not
obvious that you can reuse and tweak the action task that scheduled release promotion.
1. Copy-paste the input payload in task.extra.action.context.input
of the action
task you can to schedule again. If you don’t have an existing release promotion task,
you can use another one and tweak the payload manually notably on previous_graph_ids
.
Submit a new try push with your in-tree fixes.
Load the Treeherder link you got from your try-push. Make sure you are logged into Treeherder. In the top-right corner of the UI for the push locate the small triangle, select
Custom Push Action...
from the dropdown list.From the
Action
dropdown selectRelease Promotion
.Paste the contents of the input payload (step 1) into the
Payload
box, and click theTrigger
button. Make sure the payload is properly indented.
The new release promotion graph will be attached to your new try-push while using tasks from your old one.
- warning:
This method by-passes Shipit which can make some release promotion tasks fail. This can result in blocked release graphs.
Staging scriptworkers#
Reusing builds from a recent release
Outside of mergeduty, during development cycles, we often need to work around a single specific scriptworker, whether
that entails changing the in-tree code or the *script
itself. While
triggering staging releases is a valid solution, it is often an
expensive one as it generates an entire graph. In order to be more
efficient, one can use the scriptworker selector which aims to run a selection of
scriptworker tasks against builds from a recent release. There are a number of
preset groups of tasks to run. The list is configured here and it get be
extended for other tasks/products. To get the list of task sets, along with the
list of tasks they will run:
mach try scriptworker list
The selector defaults to using tasks from the most recent beta.To use
tasks from a different release, pass --release-type <release-type>
:
mach try scriptworker --release-type release linux-signing
Override workertype
One can extend the aforementioned behavior by overriding the
worker type to use. This is particularly useful for staging releases
against the DEV scriptworker environment. Most of the workerType configs
we have in-tree are configured as level-{1,3}
for fake/production and level-1-dev
for dev.
But the latter is not present in-tree by default so it needs to be
amended. More information on this can be found in the
scriptworker-scripts documentation. One can either manually change
the intree kind’s config to that specific worker-type, or can simply pass an
argument to aforementioned command to make the replacement,
e.g. mach try scriptworker TASK-TYPE --release-type beta --worker-suffix <alias>=<suffix>
,
where TASK-TYPE
is chosen from one of the
mach try scriptworker list
returns and alias
comes from the
taskcluster ci config file). For example, running the beetmover jobs against the most recent beta
release, but on the DEV worker-type:
mach try scriptworker beetmover-candidates --release-type beta --worker-suffix beetmover=-dev