Week 1 checklist

Week 1 checklist#

Welcome to Release Engineering! This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.

Basic#

The accounts get activated 1 day before joining so the manager of the team should drop a Slack message and an email to ensure comms are working properly. Moreover, your manager should have already scheduled a 1x1 to go over things (welcoming doc, permanent weekly 1x1 timeslot) on Monday and another one on Friday to ensure week #1 went smoothly and learn about the painpoints.

Accounts setup#

Here is a list of accounts and access needed to be successful in Release Engineering.

Bugzilla#

Almost everything at Mozilla goes through Bugzilla. Create a Bugzilla account if you have not already. You’ll need a few tweaks to your account to get access to everything releng-related:

  • Add privileges for bugzilla group releng confidential (Can be done by manager or bugzilla admin)

  • Add your comms nickname & ldap username as aliases for your account

  • log into bugzilla & follow links Preferences -> Account Information

  • append the aliases, with a leading ‘:’ and enclosed in brackets (’[]’) to the “Real Name” field

  • e.g.: John Doe [:jdoe]

The product to use is “Release Engineering.” There are multiple possible components under that product, so take your best guess or ask for guidance from the team.

:warning: Please speak with your manager to be added to the releng-security confidential group.

Misc#

  • SSO - we rely on auth0 across Mozilla for authentication and LDAP for authorization. Once given LDAP and you have created a permanent password, you can use that to login to the portal. From SSO, you should have links to various services from email, calendar, slack, mana, etc.

  • https://login.mozilla.com/ is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.

  • SSH - upload your public ssh key to Login Portal, GitHub. It is a good idea to generate a separate SSH keypair from your personal one or any other that you have created in the past and use that explicitly for Releng and upload that. Follow this SSH guidelines doc on how to generate, configure, and use your SSH key.

  • GPG - We use gpg keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair and upload your public key to OpenPGP so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you. You can use the the GPG quickstart guide on mana or you can use the The GNU Privacy Handbook for reference. Also don’t forget to add the fingerprint under https://login.mozilla.com/ too. Moreover, you’ll have to add your GPG key to Github and Git. This is usuaully optional but is a requirement in some repositories. Follow these instructions to add your key to Github and these instructions to register your key with Git for more details.

  • VPN - Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client. See the instructions on how to install and configure your VPN client and help choosing the right client for your platform. Note: macOS and Windows users should use Viscosity. This application comes with a free 30 day trial. During your trial, your manager can help you create a JIRA ticket to get a Viscosity full license.

  • MFA - More information on Multi-Factor-Authentication can be found in the Login Portal which is what we use for LDAP/auth0 based logins. Follow these instructions to download the Duo Mobile app and set it up.

Phabricator#

You will be granted access to this via SSO. Please ask your manager to be added to the following team reviewing aliases:

Gecko Development#

Some of your work will involve making changes to the Gecko Repository. This repository is hosted on GitHub, but we do not open pull requests against it. Instead, these changes flow through Phabricator for review, and Lando for merging.

You should have a read over the Firefox Contributors’ Quick Reference for additional details on how to write and submit patches to this repository.

There are 3 levels of commit access:

  • Level 1 access allows you to use the Try Server.

  • Level 2 access is required to land code in certain project repositories.

  • Level 3 access is required to land code in the Gecko Repository.

Your work will often require you to have level 3 access, so you should request it on Day 1 (see below). Other members of the team will provide the necessary vouching.

You will also need to fill out the Commit Access form and submit it via a bug on file such as this. Make sure to include (comment) your SSH public key and request 2 peers to vouch for you. This is required to grant you commit access to the Gecko repository.

At the time of writing, we are in a halfway state when it comes to migrating to GitHub. Although all commits flow through GitHub, all of our CI and Automation responds to events from read-only mirrors on https://hg.mozilla.org. This has little impact on how you interact with the repository, but it is good to be aware of.

Git & Github#

Please file a bug similar to this one and include the following teams to be added. Ask for your manager to vouch in the bug for these changes. The teams you need to be added are:

  1. orgs/mozilla

  2. orgs/mozilla-releng

  3. orgs/mozilla-mobile

  4. orgs/mozilla-services

  5. orgs/mozilla-extensions

  6. orgs/taskcluster

  7. orgs/mozilla-it

  8. orgs/mozilla-platform-ops

(You will not be able to see any these links above until you’ve been added.)

There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these.

Secrets#

We have a secrets vault that holds access to various passwords and keys. As you need access to various parts of infra, you will need to get access to the vault. Talk to your manager as this comes up. Please make sure your SSH and GPG keys are up to date in https://people.mozilla.org/ to ease the SOPS re-encryption. Talk to your manager about this.

In a nutshell, SOPS is a way to track secrets in a Git repository and securely share them with the rest of the team. You’ll need to set this up to access credentials and keys needed for various services Release Engineering uses or administers.

Steps (for MacOSX - but should be similar for Linux, modulo the installation of SOPS):

install Google Cloud SDK deps#

$ curl https://sdk.cloud.google.com | zsh
$ gcloud init

RelEng currently has two SOPS repositories for holding off secrets.#

  1. moz-fx-releng-secrets-global - this is dedicated for RelEng team secrets (3rd party accounts, certificates, etc). Basically it’s our own private space for holding off any type of secrets. Your manager should be able to grant you access to this repo.

  2. moz-fx-relengworker-prod-a67d - this is dedicated for our scriptworkers mozilla-releng/scriptworker-scripts. You’ll need a pull request similar to this one to get access. You can do this yourself after getting access to the cloudops-infra repository (see below). It’s mirrored to CloudOps infrastructure. This repo is owned by CloudOps, to get access create an issue similar to SVCSE-242

The one that’s most commonly used is the global one. The second one is needed only if a new type of scriptworker is added and/or we’re adjusting existing credentials in the release scriptworkers automation.

clone the sops repo somewhere on disk#

$ gcloud source repos clone releng-secrets-global --project=moz-fx-releng-secrets-global
$ gcloud source repos clone secrets-sops-relengworker --project=moz-fx-relengworker-prod-a67d

install sops#

$ brew install sops

acquire new user credentials to talk to the Google Cloud API#

gcloud auth login --update-adc

This command will login and update the “application-default login” at the same time.

celebrate by operating the sops credentials#

Have a look in the COOKBOOK in the global SOPS repo for more instructions on how to read encrypt/decrypt the files.

:warning: Talk to your manager to add your fingerprint in the SOPS repo and also grant you access in both repos server-side.

Disclaimer: Changes ongoing after August 2021#

The max session for SOPS credentials has been reset to 24h so the login command needs rerunning in the shell in order to unblock using SOPS.

Sharing secrets with a co-worker#

On a side-note, for sharing secrets between employees, read these docs.

Communication#

Mail#

Mozilla mail is handled by Gmail now.

Mailing lists#

  • You should be added to the release@mozilla.com google group as a new hire. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to release@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

  • You should be added to the releng@mozilla.com google group as a new hire. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to releng@m.c to verify that your address has been added/subscribed. Talk to your manager if it is not working.

:warning: release@m.c can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla’s control. Same goes for releng@mozilla.com.

Mail filtering#

With all that new email, you will want to set up some filters in Gmail to handle the higher-volume automated mail. You may eventually want to take a look at this information, but on day one hundreds of automation notifications are not going to be educational.

:warning: To mitigate the incoming barrage of automated emails download our imperfect, but useful, set of already existing filters.

  1. Download mailFilters.xml

  2. Open Gmail

  3. At the top right, click Settings and then See all settings

  4. Click Filters and Blocked Addresses

  5. At the bottom of the page, click Import filters

  6. Choose the mailFilters.xml file

  7. Click Open file

  8. Click Create filters

Calendar#

Like mail, we now use Google calendar

You should be added to these private calendars:

  • RelEng (meetings and reminders)

  • RelEng Bug triage

  • RelEng PTO

  • RelEng Todo

You’ll want to subscribe to the following public calendars:

Talk to your manager/mentor to get added to the various other private calendars as appropriate.

Matrix#

The majority of day-to-day communication in releng happens on Matrix. More details on this can be found on wiki Matrix. Please talk to your manager about the private/public channels you should be in.

Slack#

Some parts of Mozilla prefer Slack to Matrix, more info on mana. Open a bug like this to get added to the @releng slack group.

Todoist#

We use todoist as a way to track scheduled deployments and renewals. Ask your manager to be added the the RelEng team there.

Zoom#

For video communication we use Zoom. More details can be found on mana.

Wiki#

Historical pieces of Mozilla information can be found in wiki. Request an account filling out this form. You should receive an email with a password in 1-2 days. If not, file a bug in bugzilla.

Docs#

Our single-source of docs is this. Don’t be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask your team for more details.

Google Drive#

Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time. Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.

Mana#

Some internal Mozilla systems (IT, HR) are documented on mana. That is behind SSO and you will be granted access to that as soon as LDAP is activated.

Future access as you need it#

  • You will need acccess to our GCP infrastructure. Talk to your manager to be added to the team-releng@firefox.gcp.mozilla.com. Use a ticket similar to this in CloudOps.

  • CultureAmp - talk to your manager to set your goals and run the logistics in there for the 1x1 tracking. You’ll also need to be added to the RelEng team goals.

  • JIRA - access to JIRA is granted upon SSO access so please make sure to login via the main dashboard. Ask your manager to add you to the RELENG/FFXP boards.

  • Sentry logs - File a bug similar to this one to get access to debug various logs in Balrog and more.

  • CloudOps Jenkins - Once you’ve got added to the mozilla-services org (see the github section above), open a pull request similar to this one In addition, cloudops will need to add you to the Release Engineering group on jenkins, see this bug. Once it’s properly configured, you’ll be able to access it by making some tweaks documented here.

  • App Store Connect - File a bug like this one

  • Google Play - File a bug like this one

LDAP groups#

  • RelEng LDAP flags to access various infrastructure pieces. File a bug similar to this and ask your manager for vouching. The flags needed are listed below:

Balrog ACLs#

In addition to LDAP access to Balrog, you will need to be granted ACLs in Balrog itself. File a bug similar to this one to have those granted.

cn=active_scm_firefoxci,ou=groups,dc=mozilla
cn=all_scm_firefoxci,ou=groups,dc=mozilla
cn=releng,ou=groups,dc=mozilla
cn=balrog,ou=groups,dc=mozilla
cn=inventory,ou=groups,dc=mozilla
cn=inventory_build,ou=groups,dc=mozilla
cn=vpn_balrog,ou=groups,dc=mozilla
cn=vpn_cloudops_shipit,ou=groups,dc=mozilla
cn=vpn_releng,ou=groups,dc=mozilla
cn=tooltooleditor,ou=groups,dc=mozilla
cn=scm_firefoxci,ou=groups,dc=mozilla

You can view your LDAP groups by accessing the SSO info page

Good first touchpoint#

Following Releaseduty docs to better understand the release mechanics - https://moz-releng-docs.readthedocs.io/en/latest/procedures/release-duty/index.html