Welcome to RelEng Docs’s documentation!

Here is Mozilla’s Release Engineering Group’s more technical documentation. You might also be interested in our main web site.

Contents:

Other Useful Documentation

• Balrog
• Scriptworker
• Scriptworker Scripts
• Treeherder
• Accessing CloudOps' Jenkins
• CloudOps Contact Info

Overview and Procedures

• Procedures
  • Firefox Release Process
    • Requirements and permissions to conduct a release
    • Specific steps required to initiate and release Firefox
      • High level
      • Connect to the VPN
      • Initiate a new release
      • Trigger the phases of the release
      • Signing off in Balrog
      • How these taskcluster tasks are scheduled
    • Other Release Management Tasks
    • Escalating issues and communications
    • Source and under the hood
      • Taskcluster
      • Signing
      • Providing Updates
      • Shipit
  • Release Duty
    • Updating one ESR version to another
      • Overview
      • Figure out what rules are needed
      • Adjust rules on the esr-localtest and esr-cdntest channels
      • Test!
      • Adjust esr channel rules
      • The Snap case
      • Bouncer and EOL’ing the old ESR branch
    • Desktop Flatpak Releases
      • Channels used
      • Changes to Flatpak via web interface
      • ## Refresh Flathub credentials
    • Desktop Releases
      • Managing Different Types of Releases
    • Run staging releases
      • Permissions
      • How-to
      • Staging scriptworkers
    • Desktop Snap Releases
      • Channels used
      • Promote a snap to the stable channel
      • How to manually push a snap to the store, in case automation failed?
      • Refresh macaroons credentials
      • Refreshing collaborators
    • Mobile Releases
      • Managing Different Types of Releases
    • Merge Duty
      • Overview of Procedure
      • Do the prep work a week before the merge
      • Release Merge Day - part I
      • Release Merge Day - part II - a week after Merge day
    • Merge Duty for major ESR bump
      • Intro
      • Internal changes
      • External systems
      • Tasks
      • Odd problems
      • Merge
      • Release builds
      • Update this documentation
      • Ship it!
    • Expectations
    • Communication
      • Slack Channels
      • Matrix Channels
      • Email
      • Meetings and Calendars
    • Permissions
      • How to get VPN access
      • How to get Ship-it access
      • How to get Balrog access
    • Tooling for debugging and rerunning tasks
      • taskcluster
      • in-bulk taskcluster operations
      • graph inspection
      • Firefox bookmarks
    • After ReleaseDuty
    • Miscellaneous
    • Hand Off
    • Escalation
    • Other useful resources
    • Glossary
  • Testing Autograph
    • Testing Autograph Stage
      • Autograph-stage mar signing test
    • Testing Autograph Prod
      • Autograph-prod mar signing test
  • Tree Closing Window Planning
    • Planning Procedure
    • Execution of TCW
  • Tree Closing Window Execution
    • Wednesday Before
    • Day of TCW
    • When TCW Done, Before Opening Trees
    • Next Business Day
  • Transitioning RelEng Status
    • Overview
      • Password List
      • Miscellaneous Systems
  • Accounts Setup
  • How to automate nightly Google Play deployments
  • How to test release graphs in mobile
  • How to set up taskgraph for mobile
  • Access Bouncer
  • Analyze Update Verify Logs
    • When?
    • How?
    • Known differences
  • Manually Generate Partials
    • When?
    • How?
    • 1. build/tools changes
    • 2. Update the balrog blobs and generate taskgraph
    • 3. Update balrog blobs (one more time) and bouncers
  • Off-Cycle Partner Repacks and Funnelcakes
    • Background
    • When?
    • How?
    • Preparation
    • Additional steps
      • Funnelcake
      • Stub installers
    • Future
  • Purging the Partials Cache
  • Rotate hg.m.o cert
    • 2020 links
    • Steps
  • Sharing your terminal
    • Introduction
    • Screen
      • Setup
      • Usage
    • Tmux
      • Single user, multiple connections
      • Multiple users
  • Upload to internal Pypi
  • Widevine updates
    • When
    • How
      • Ensure what Firefox version is able to run the new version
      • Create the blob
      • Create the balrog rule
      • Testing
• Turning on Firefox tests for a new configuration
  • Push to Try server
  • Analyze test failures
  • Disable tests in the manifest files
  • Rerun try push, repeat as necessary
  • land changes and turn on tests
  • turn on run only failures
  • file bugs for test failures

Documentation and Articles

• Production & Staging Logs
• Releng Architecture
  • Audience
    • What to Include
• Best Practices
  • Coding Guidelines
    • Code Storage
    • Scripting Languages
    • Configuration Formats
    • Web Application Frameworks
    • Libraries & Tools
    • Futures
  • Documentation and Cross-training
    • Documentation
    • Cross-training
      • Bus factor
      • Use cross-training to test documentation
• Taskcluster Administration and Debugging
  • Uploading an image
    • image_builder
    • decision images
  • Troubleshooting Workers
    • Active workers
    • Worker Manager errors
    • Common types of error messages
    • Quarantining workers
    • Unquarantining workers
  • Taskcluster command line interface (CLI)
    • Difference between actions and CLI
    • Taskcluster CLI best practices
      • Principle of Least Privilege
      • Aliases
  • Rerun vs Retrigger
    • Intermittent tests in treeherder: retrigger
    • Broken Pull Request tasks: rerun
    • Broken release tasks: rerun
    • Testing scriptworker pools outside of the release: retrigger (force)
    • Notarization poller timeouts: rerun (force)
    • Release tasks with broken dependencies: cancel + rerun
  • Testing and Customizing Release Promotion actions
    • taskGroupIds
    • Testing Release Promotion actions
      • Replicating an existing release graph locally
      • Example promotion test
      • Advanced relpro usage
  • Scopes
    • Delimiters
    • Ci-configuration
    • Conventions
      • Groups / teams
      • Levels
    • Docker- and Generic-Worker scopes
    • Scriptworker scopes
      • Restricted scopes
• Hosts
  • hg.mozilla.org
• Software
  • Taskcluster
    • hg pushlog
    • Mozharness
  • BaseLogger
  • BaseConfig
  • BaseScript
  • Mozharness Example
    • Treeherder
    • VCS Sync tools
  • legacy
• Addons
  • Langpack Submission Pipeline
    • Terminology:
    • Background
      • Restrictions
      • Process
    • How To…
      • How to handle New Languages for Release
      • What if I don’t do the important step in the manual process above?
    • Screenshots relevant to above
      • 01
    • Troubleshooting
      • Firefox version is not known
      • Submitting a listed langpack for the first time
      • Submitting addons can be rate-limited
      • Intermittent errors
      • Refresh AMO API keys
  • XPI Signing Pipeline
• Balrog & Updates
  • Balrog and Scheduled Changes
    • Responsibilities
      • RelEng
      • RelMan
    • Example
    • Watersheds
  • Change Update to Last Known Good
    • Once a new build contains the fix
  • win32-win64 migration
    • High-level steps
    • Detailed descriptions
    • Firefox beta
    • Creating a new release in balrog
    • Adding new rule
    • Modifying release blob
    • Adding new rule
• Signing
  • Signing Cert Levels and Best Practices
    • Levels / cert types
      • Dep
      • Shipping certs
    • Caveats
      • Android signatures
  • Manually Sign Android APKS
    • Requirements
    • First Steps
    • To sign an APK
    • After all the signing
    • Troubleshooting
      • Can’t sign APKs
    • Future
  • Create a new release key to sign a new apk product
    • The example product here is “rocket”
  • How to rotate the Firefox Release signing GPG Key
    • When
    • Process
      • Generate a new signing subkey
      • Publish the new public key
      • Import the new private key into autograph
      • Update the public key
      • Start signing with the new private key
  • Notarization
    • Machine and maintenance
    • General workflow
    • Debugging
    • Escalation
    • Links
  • Adhoc signing
• Releng Changelog
  • During 96.0 >= 2021-11-01
    • Changed
  • During 95.0 >= 2021-10-04
    • Changed
  • During 82.0 >= 2020-09-21
    • Changed
  • During 81.0 >= 2020-08-24
    • Changed
  • During 80.0 >= 2020-07-27
    • Changed
  • During 75.0 >= 2020-04-06
    • Changed
    • Fixed
  • During 74.0 >= 2020-03-10
    • Added
  • During 73.0 >= 2020-02-10
    • Changed
  • During 71.0 >= 2019-10-14
    • Changed
  • During 69.0 >= 2019-05-13
    • Changed
  • During 67.0 >= 2019-03-11
    • Changed
  • During 66.0 >= 2019-01-30
    • Changed
  • During 64.0 >= 2018-10-15
    • Changed
  • During 63.0 >= 2018-09-05
    • Added
    • Changed
  • During 62.0 >= 2018-06-26
    • Changed
    • Added
  • During 61.0 >= 2018-05-09
    • Added
    • Changed
  • During 60.0 >= 2018-03-13
    • Added
    • Changed
    • Fixed
  • During 59.0 >= 2018-01-24
    • Added
    • Changed
  • During 58.0 >= 2017-11-15
    • Added
    • Changed
    • Removed
  • During 57.0 >= 2017-09-27
    • Added
    • Changed
    • Removed
  • During 56.0 >= 2017-08-09
    • Added
    • Changed
    • Removed
• Releng Future Projects
  • Ideas
    • Mac Signing on Linux
    • “Shippable” builds
    • Local depend signing
    • Partial updates
    • Canonical list of watershed releases
  • Guidelines
• Machine Users
  • Mercurial Users
    • Level 3
    • Level 1
• Troubleshooting Expirations Notifications
  • Apple common types of error messages
• Gecko Tests
  • Adding a keyword to mochitest and xpcshell manifests
    • Current behavior
    • How to add a keyword
  • Adding a keyword to reftest style manifests
    • Current behavior
    • How to add a keyword

Indices and tables

• Maintaining this Documentation

• Index

• Module Index

• Search Page

• Topics needing authors