Taskcluster Administration and Debugging

Specifically for the FirefoxCI cluster.

• Taskcluster docs

• Taskgraph docs

• Gecko in-tree taskcluster docs

• Scriptworker docs

• Scriptworker-scripts docs

• Rotate CoT keys

• Uploading an image
  • image_builder
  • decision images
• Troubleshooting Workers
  • Active workers
  • Worker Manager errors
  • Common types of error messages
  • Quarantining workers
  • Unquarantining workers
  • Worker-manager slowness
  • Viewing pending tasks
• Taskcluster command line interface (CLI)
  • Difference between actions and CLI
  • Taskcluster CLI best practices
• Rerun vs Retrigger
  • Intermittent tests in treeherder: retrigger
  • Broken Pull Request tasks: rerun
  • Broken release tasks: rerun
  • Testing scriptworker pools outside of the release: retrigger (force)
  • Notarization poller timeouts: rerun (force)
  • Release tasks with broken dependencies: cancel + rerun
  • Deadline-exceeded leaf node release tasks
• Testing and Customizing Release Promotion actions
  • taskGroupIds
  • Testing Release Promotion actions
• Scopes
  • Ci-configuration
  • Roles
  • Expansion of scopes in Automation
  • Conventions
  • Docker- and Generic-Worker scopes
  • Scriptworker scopes
• Known Problems
  • Actions are broken after modifying .taskcluster.yml
  • Missing Scopes after Branch Rename
  • Missing Mercurial features when Cloning the Repo
  • Workers are spinning up slowly
  • Workers Not Spawning After Image Bustage
  • push(MSIX) fails: “push to Store aborted: pending submission found”
• CI-Admin
  • Through cloudops-jenkins
  • Manually
• FirefoxCI cluster upgrades
  • Process
  • Minor version upgrade process
  • Major version upgrade process
• Taskcluster Staging
  • When to run these steps
  • Setup
  • Log into TC Staging and run ci-admin
  • Push to try
  • Run fxci to send mozilla-central tasks to the staging cluster
  • Known issues
• How Tasks are Triggered
  • [Pulse ->] hooks -> build-decision
  • Through action hooks
  • Through taskcluster-github
  • Side note: taskcluster_yml_repo