Regular weekly releases

Push to production happen in weekly batches. Once a week (usually on Thursday) a release happens.

Current administrators that perform this weekly release are:

Protocal that we follow is:

  1. Prior to release a push to staging branch must happen. This will trigger a deploy of all projects to staging environments.

    $ git clone git@github.com:mozilla/release-services.git
    $ cd release-services
    $ git push -f origin origin/master:staging
    
  2. Verify that all the production projects in staging that they are functioning properly. Each project should have a list of steps that you can easily verify that a deployment was sucessful.

    .

    Example: verify releng-tooltool project

    Only proceed further once production projects work in staging environment.

  3. Announce that new deployment to production is going to happen

    • announce in #ci channel that a push to production is about to happen.

      Example message:

      I am about to release a new version of mozilla/release-services
      (*.mozilla-releng.net, *.moz.tools). Any alerts coming up soon will be
      best directed to me. I'll let you know when it's all done. Thank you!
      
    • inform MOC person on duty (in #moc channel) that new deployment of mozilla/release-services is going to be happen. The channel subject should contain on duty sysadmin: followed by the IRC nickname you need to contact.

      Example message:

      nickname: I am about to release a new version of
      mozilla/release-services (*.mozilla-releng.net, *.moz.tools). Any
      alerts coming up soon will be best directed to me. I'll let you know
      when it's all done. Thank you!
      
  4. Push to production. Create a merge commit of master branch and tag it. Don’t forget to push just created tag.

    $ git clone git@github.com/mozilla/release-services.git
    $ cd release-services
    $ git checkout -b production origin/production
    $ git merge master -m "Release: v$(git show master:VERSION)"
    $ git push origin production
    $ git tag v$(cat ./VERSION)
    $ git push origin v$(cat ./VERSION)
    
  5. Verify that all production projects are now deployed and working properly in production environment. Use the same checks as we did before when we were checking if projects are working on staging, but now use production URLs.

    Example: verify releng-tooltool project

  6. Fill in the release notes on GitHub

    New GitHub Release

    If the previous release was done on 2017/05/04 then a good starting point might be

    $ git log --oneline v$((($(cat VERSION)) - 1)).. HEAD \
        | cut -d' ' -f2- \
        | sort \
        | grep -v 'setup: bumping to'
    
  7. Bump version, but DO NOT push upstream

    $ git clone git@github.com/mozilla/release-services.git
    $ cd release-services
    $ echo "$((($(cat VERSION)) + 1))" | tee VERSION2
    $ sed -i -e "s|base-$(cat VERSION)|base-$(cat VERSION2)|" .taskcluster.yml
    $ mv VERSION2 VERSION
    
  8. Push new base image for new version

    $ ./please -vv tools base-image \
         --taskcluster-client-id="..." \
         --taskcluster-access-token="..."
    

    Docker username and password you get in staging secrets or production secrets secrets.

    It might happen that push to docker hub will fail since the resulting docker image is quite big (~1.5GB). When it fails you can only retrigger the docker push command.

    $ docker push mozillareleng/services:base-$(cat ./VERSION)
    
  9. Once base image is pushed to docker hub, commit the version bump and push it to upstream repository.

    $ git commit VERSION .taskcluster.yml -m "setup: bumping to v$(cat ./VERSION)"
    $ git push origin master
    

    Make sure that commit gets properly build before proceeding. This will ensure that docker base image created in previous steps is working.

  10. Announce that deployment to production is done.

    • announce in #ci channel that a push to production is complete.

      Example message:

      Previously annonced release of mozilla/release-services
      (*.mozilla-releng.net, *.moz.tools) to productions is now complete. If
      you see anything behaving weird please let me know. Changes ->
      <link-to-release-notes>.
      
    • inform MOC person on duty (in #moc channel) that deployment of mozilla/release-services is complete.

      Example message:

      nickname: Previously annonced release of mozilla/release-services
      (*.mozilla-releng.net, *.moz.tools) to productions is now complete.
      Changes -> <link-to-release-notes>.