Day 1 checklist¶
Welcome to Release Engineering! This page is meant to get new hires, interns, or interested community members up to speed with the right software, configurations, and communication channels to contribute effectively to the release engineering pipeline.
The accounts get activated 1 day before joining so the manager of the team should drop a Slack message and an email to ensure comms are working properly. Moreover, your manager should have already scheduled a 1x1 to go over things (welcoming doc, permanent weekly 1x1 timeslot) on Monday and another one on Friday to ensure week #1 went smoothly and learn about the painpoints.
Here is a list of accounts and access needed to be successful in Release Engineering.
Almost everything at Mozilla goes through Bugzilla. Create a Bugzilla account if you have not already. You’ll need a few tweaks to your account to get access to everything releng-related:
Add privileges for bugzilla group releng confidential (Can be done by manager or bugzilla admin)
Add your comms nickname & ldap username as
aliasesfor your account
log into bugzilla & follow links
append the aliases, with a leading ‘:’ and enclosed in brackets (’’) to the “Real Name” field
John Doe [:jdoe]
The product to use is “Release Engineering.” There are multiple possible components under that product, so take your best guess or ask for guidance from the team.
TODO - to revisit once we reorg Bugzilla in November 2022.
:warning: Please speak with your manager to be added to the releng-security confidential group.
SSO - we rely on auth0 across Mozilla for authentication and LDAP for authorization. Once given LDAP and you have created a permanent password, you can use that to login to the portal. From SSO, you should have links to various services from email, calendar, slack, mana, etc.
https://login.mozilla.com/ is where you can change a number of authentication/authorization access bits that you have control over. Each todo in this section assumes you have access to this page.
SSH - upload your public ssh key to Login Portal, GitHub. It is a good idea to generate a separate SSH keypair from your personal one or any other that you have created in the past and use that explicitly for Releng and upload that. Follow this SSH guidelines doc on how to generate, configure, and use your SSH key.
GPG - We use gpg keys to share private information, secrets, and verify that the source came from someone we trust. Generate a keypair and upload your public key to OpenPGP so others can find it. It would be really good if you could have other people sign your key, adding more trust that this key really belongs to you. You can use the the GPG quickstart guide on mana or you can use the The GNU Privacy Handbook for reference. Also don’t forget to add the fingerprint under https://login.mozilla.com/ too. Moreover, you’ll have to add your GPG key to Github and Git. This is usuaully optional but is a requirement in some repositories. Follow these instructions to add your key to Github and these instructions to register your key with Git for more details.
VPN - Many of our systems are behind a private network in addition to auth0. Follow the prompts to generate and download an openVPN certificate that you can use to import to your vpn client. See the instructions on how to install and configure your VPN client and help choosing the right client for your platform. Note: macOS and Windows users should use Viscosity. This application comes with a free 30 day trial. During your trial, your manager can help you create a JIRA ticket to get a Viscosity full license.
MFA - More information on Multi-Factor-Authentication can be found in the Login Portal which is what we use for LDAP/auth0 based logins. Follow these instructions to download the Duo Mobile app and set it up.
Mercurial - Most development in releng (and at Mozilla writ-large) is stored in version control using hg. There is an excellent step-by-step guide for setting up and using hg: Mercurial for Mozillians. More information on this can be found in here and here.
There are 3 levels of commit access: * Level 1 access allows you to use the Try Server and setup user repos. As a new contributor, you should request this on day one. * Level 2 access is required to land code in the build and project repos. Once you have a proven track record of successful patches, you can ask your manager/mentor to vouch for your Level 2 access. Your manager/mentor can also land patches for you until you receive Level 2 access * Level 3 access is required to land code in mozilla-central and its derived integration & release branches. At some point in your Mozilla contribution story, you may need Level 3 access but many contributors never do. Talk to your manager/mentor if you think you need this access. You should already have Level 2 access when you request Level 3
You will be granted access to this via SSO. Please ask your manager to be added to the following team reviewing aliases:
The root webview of the Mozilla hg repositories is here: https://hg.mozilla.org/ while most releng code lives in repos under https://hg.mozilla.org/build. Please fill out the Commit Access form and submit it via a bug on file such as this. Make sure to include (comment) your SSH public key and request 2 peers to vouch for you.
Git & Github¶
Please file a bug similar to bug 1726348 and include the following teams to be added. Ask for your manager to vouch in the bug for these changes. The teams you need to be added are:
There are also a handful of git repos hosted directly by Mozilla. Your manager/mentor will let you know if you need access to one of these.
We have a secrets vault that holds access to various passwords and keys. As you need access to various parts of infra, you will need to get access to the vault. Talk to your manager as this comes up. Please make sure your SSH and GPG keys are up to date in https://people.mozilla.org/ to ease the SOPS re-encryption. Talk to your manager about this.
In a nutshell, SOPS is a way to track secrets in a Git repository and securely share them with the rest of the team. You’ll need to set this up to access credentials and keys needed for various services Release Engineering uses or administers.
Steps (for MacOSX - but should be similar for Linux, modulo the installation of SOPS):
install Google Cloud SDK deps¶
$ curl https://sdk.cloud.google.com | zsh $ gcloud init
RelEng currently has two SOPS repositories for holding off secrets.¶
moz-fx-releng-secrets-global- this is dedicated for RelEng team secrets (3rd party accounts, certificates, etc). Basically it’s our own private space for holding off any type of secrets.
moz-fx-relengworker-prod-a67d- this is dedicated for our scriptworkers (https://github.com/mozilla-releng/scriptworker-scripts). It’s mirrored to CloudOps infrastructure.
The one that’s most commonly used is the global one. The second one is needed only if a new type of scriptworker is added and/or we’re adjusting existing credentials in the release scriptworkers automation.
clone the sops repo somewhere on disk¶
$ gcloud source repos clone releng-secrets-global --project=moz-fx-releng-secrets-global $ gcloud source repos clone secrets-sops-relengworker --project=moz-fx-relengworker-prod-a67d
$ brew install sops
acquire new user credentials to talk to the Google Cloud API¶
gcloud auth application-default login
celebrate by operating the sops credentials¶
Have a look in the COOKBOOK in the global SOPS repo for more instructions on how to read encrypt/decrypt the files.
:warning: Talk to your manager to add your fingerprint in the SOPS repo and also grant you access in both repos server-side.
Disclaimer: Changes ongoing after August 2021¶
The max session for SOPS credentials has been reset to 24h so there’s some commands that need rerunning in the shell in order to unblock using SOPS.
$ gcloud auth login $ gcloud auth application-default login
Mozilla mail is handled by Gmail now.
With all that new email, you will want to set up some filters in Gmail to better handle some of the higher-volume automated mail into a folder. You may eventually want to handle this information, but on day one hundreds of automation notifications are not going to be educational.
:warning: Please ask your manager to provide an imperfect but useful set of already existing filters to help handling the load.
You should be added to the firstname.lastname@example.org google group as a new hire. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to email@example.com to verify that your address has been added/subscribed. Talk to your manager if it is not working.
You should be added to the firstname.lastname@example.org google group as a new hire. This mailing list is managed by Google groups. Owners of this group will be able to add you. Send a test message to email@example.com to verify that your address has been added/subscribed. Talk to your manager if it is not working.
:warning: firstname.lastname@example.org can contain security-sensitive information. Do not automatically forward your email to a system that is not under Mozilla’s control. Same goes for email@example.com.
Like mail, we now use Google calendar
You’ll want to subscribe to the following public calendars:
Talk to your manager/mentor to get added to the various other private calendars as appropriate.
The majority of day-to-day communication in releng happens on Matrix. More details on this can be found on wiki Matrix. Please talk to your manager about the private/public channels you should be in.
Some parts of Mozilla prefer Slack to Matrix, more info on mana.
For video communication we use Zoom. More details can be found on mana.
Our single-source of docs is this. Don’t be shy about making improvements to releng pages based on your experiences. Getting someone in releng to review your changes first is good practice. Just ask your team for more details.
Google Drive (formerly Google docs) is a preferred way to share things these days. This includes spreadsheets and documents that will change a great deal over time. Google Drive access should be enabled with your email account when you start. If you need access to a particular document, talk to the document owner or your manager/mentor.
Some internal Mozilla systems (IT, HR) are documented on mana. That is behind SSO and you will be granted access to that as soon as LDAP is activated.
Future access as you need it¶
Ship-it - follow procedure in here to get access.
CultureAmp - talk to your manager to set your goals and run the logistics in there for the 1x1 tracking. You’ll also need to be added to the RelEng team goals.
JIRA - access to JIRA is granted upon SSO access so please make sure to login via the main dashboard. Ask your manager to add you to the RELENG/FFXP boards.
Sentry logs - File a bug similar to this one to get access to debug various logs in Balrog and more.
CloudOps Jenkins - File a bug similar to Bug 1721444 and talk to your manager to get access to Janekins CloudOps to be able to debug. You should already have access to cloudops-infra repo if you’ve done the Github section above. Once it’s properly configured, you’ll be able to access it by making some tweaks documented here.
RelEng LDAP flags to access various infrastructure pieces. File a bug similar to this and ask your manager for vouching. The flags needed are listed below:
cn=active_scm_firefoxci,ou=groups,dc=mozilla cn=all_scm_firefoxci,ou=groups,dc=mozilla cn=releng,ou=groups,dc=mozilla cn=RelEngWiki,ou=groups,dc=mozilla cn=balrog,ou=groups,dc=mozilla cn=inventory,ou=groups,dc=mozilla cn=inventory_build,ou=groups,dc=mozilla cn=vpn_balrog,ou=groups,dc=mozilla cn=vpn_cloudops_shipit,ou=groups,dc=mozilla cn=vpn_releng,ou=groups,dc=mozilla cn=tooltooleditor,ou=groups,dc=mozilla cn=scm_firefoxci,ou=groups,dc=mozilla cn=vpn_releng_loan,ou=groups,dc=mozilla
You can view your LDAP groups by accessing the SSO info page
To access any of Release Engineering, Taskcluster, and Release Operations hosts directly, you will need to go through VPN -> Separate MFA -> your target host. To do that, you and your manager will need to file a ticket against Release Operations and have them send you an invite to add an MFA account on your Duo App.
Example ssh config (adjust the
TODO to match your own configs)
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to. HashKnownHosts yes # Host keys the client accepts - order here is honored by OpenSSH HostKeyAlgorithms firstname.lastname@example.org,email@example.com,ssh-ed25519,ssh-rsa,firstname.lastname@example.org,email@example.com,firstname.lastname@example.org,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256 Host hg.mozilla.org git.mozilla.org User TODO_USERNAME@mozilla.com Compression yes ServerAliveInterval 300 Host *.mozilla.com User TODO_USERNAME IdentityFile ~/.ssh/id_rsa_<TODO_name_of_ssh_private_key> Compression yes ServerAliveInterval 300 Host *.build.mozilla.org Compression yes User cltbld ServerAliveInterval 300
Good first touchpoint¶
Following Releaseduty docs to better understand the release mechanics - https://moz-releng-docs.readthedocs.io/en/latest/procedures/release-duty/index.html