.. _how_tasks_are_triggered: How Tasks are Triggered ======================= How are tasks triggered in Taskcluster? - You can create arbitrary tasks through, e.g. `the create task API/UI `__. - Or you can encode one or more tasks in `.taskcluster.yml `__ - If you pass a certain threshold of complexity, you probably want your ``.taskcluster.yml`` to schedule a decision task, which will create a task graph via `taskcluster-taskgraph `__. ... But what generates the task(s) in ``.taskcluster.yml``? [Pulse ->] hooks -> build-decision ---------------------------------- We have a python module named `build-decision`_ that knows how to trigger decision tasks for hg.m.o pushes and github+hg.m.o cron tasks. This is bundled in the `build-decision docker image `__. Once we build that image in `automation `__, Releng chooses a given build-decision image to use and uploads it to `docker-hub `__. We then update the pinned image in the `hg push hook template `__ and the `cron hook template `__; the next time we run :ref:`ci-admin `, ci-admin updates the various on-push (`e.g. `__) and cron (`e.g. `__) hooks. hg.m.o on-push ~~~~~~~~~~~~~~ When we push a change to hg.m.o, we send a `pulse message `__. The `mozilla-central hg-push hook `__ listens to ``exchange/hgpushes/v2 with mozilla-central``, creates a task using the build-decision image, and sets the ``PULSE_MESSAGE`` env var to the contents of the pulse message. This task then creates the decision task, which in turn creates a task graph as appropriate. We can replicate this later, using `fxci `__, via the ``fxci replay-hg-push`` subcommand. Cron ~~~~ We have three types of cron hooks: Time-based cron hooks ^^^^^^^^^^^^^^^^^^^^^ When we enable either `gecko-cron `__ (firefox desktop or thunderbird only), or `taskgraph-cron `__ for a project, and when there is a `.cron.yml `__ at the top of the repository, then we create a cron task `every 15 minutes `__ to see if anything should run. This cron task runs `build-decision`_, which knows how to generate a cron decision task. For example, the `mozilla-central cron hook `__ has a schedule of ``0 0,15,30,45 * * * *``, and runs the ``cron`` command entry point in the ``build-decision`` docker image. Manually triggered cron hooks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes we want to be able to trigger some of these cron tasks outside of the regularly scheduled times. (Sometimes we even have 0 regularly scheduled times for specific cron tasks, and we only trigger them manually.) To enable this, we specify the ``project.cron.targets``, like for mozilla-central `here `__; in ``cron.targets`` we enable the desktop nightlies to be triggered off-cycle, as well as l10n-bumper and others. The ``scriptworker-canary`` cron hook accepts input (``allow-input: true``). So when we run :ref:`ci-admin `, we generate things like the `mozilla-central l10n-bumper cron hook `__, which has no schedule, and we have the ``'--force-run=l10n-bumper'`` option specified in the ``cron`` command. We can trigger these hooks manually. The `mozilla-central scriptworker-canary cron hook `__ similarly has no schedule, and has the ``'--force-run=scriptworker-canary'`` option. Because we allow for input, we set the ``HOOK_PAYLOAD`` env var to the payload that we send to the hook. .. _pulse_triggered_cron_hooks: Pulse-triggered cron hooks ^^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes we want to be able to trigger some of these cron hooks via some external, non-time-based, non-manually-triggered signal. In this case, we can add a pulse binding to a given cron hook. In the `android-l10n-tooling project `__, we have the ``update-l10n`` cron.target where we specify ``bindings`` as follows:: - exchange: exchange/taskcluster-github/v1/push routing_key_pattern: primary.mozilla-mobile.fenix - exchange: exchange/taskcluster-github/v1/push routing_key_pattern: primary.mozilla-mobile.android-components - exchange: exchange/taskcluster-github/v1/push routing_key_pattern: primary.mozilla-mobile.firefox-tv - exchange: exchange/taskcluster-github/v1/push routing_key_pattern: primary.mozilla-lockwise.lockwise-android And indeed, the `update-l10n hook `__ generated by :ref:`ci-admin ` has no schedule; it does have those pulse bindings, it includes the ``cron`` option ``'--force-run=update-l10n'``, and is also manually triggerable. The github push pulse event appears to be `sent by taskcluster-github `__, so we trigger this hook whenever there is a github push to these repos, but we can also trigger the hook manually if there's an issue with the pulse-driven pipeline. Through action hooks -------------------- As noted in :ref:`actions_tc_yml`, we embed the contents of .taskcluster.yml into action hooks, which are named by .tc.yml hash. This avoids having an intermediate task:: hook -> action task -> result rather than:: hook -> build-decision -> action task -> result Some of the rationale and debate are surfaced in `bug 1463522 `__ and `bug 1415868 comment 77 `__. We may want to revisit this debate, but until then, our action hooks directly create an action task. However, these are different than other hooks, in that action tasks are run against a previously-run decision task, and rely on the decision task's artifacts, especially ``actions.json``. Through taskcluster-github -------------------------- Taskcluster-github `listens to Github events, directly parses .taskcluster.yml, and creates decision tasks `__. We may want to revisit whether we want the app to do this, or if we want an intermediate `build-decision`_ task in between. Side note: ``taskcluster_yml_repo`` ----------------------------------- In `Releng-RFC 36`_ we are trying to enable standard build/test workflows without having to land custom code in a given repository. One feature we already support in ci-configuration is ``taskcluster_yml_repo``. Before we combined the ci-configuration and ci-admin repos, ci-configuration specified ci-admin as its `taskcluster_yml_project `__. We refer to this in the `ciadmin.generate.hg_pushes.make_hook function `__; the `build-decision hg-push cli `__ supports that; the `build_decision.hg_push.build_decision function `__ then specifies the ``taskcluster_yml_repo``'s .taskcluster.yml as the URL to use to render the decision task. This is pretty great: it's already supported; if you download the ``.taskcluster.yml`` from another repo, you can also clone it and use its ``taskcluster/`` directory; and if we didn't mind multiple template repos, we could create a ``taskcluster_yml_repo`` for every build/test workflow we want to support in a generic way. **However.** There are caveats: - ``taskcluster_yml_repo`` and ``taskcluster_yml_project`` are only supported in hg.m.o projects. If we want to support these fully across all types of projects, we need to support all of the above ways of triggering decision/action/cron tasks, or the unsupported way won't know how to find the appropriate ``.taskcluster.yml`` file and will fail. We only support the hg-push, build-decision workflow with ``taskcluster_yml_repo`` currently. - Once we get that working, we have to figure out how we refer to arbitrary repos in the shared ``.taskcluster.yml`` and possibly ``taskcluster.ci.config``. This may be simple: we were able to use ``${repoUrl}`` and ``${push.revision}`` in the `shared ci-admin .tc.yml `__, but we also listed it under our `taskcluster.ci.config.taskgraph.repositories `__. Perhaps this isn't a worry for level 1 repos: `xpi-template doesn't list each downstream repo `__, but we needed to in `xpi-manifest `__. Depending on the answers to the above, ``taskcluster_yml_repo`` could be a good stopgap solution or a stepping stone on the way to `Releng-RFC 36`_. .. _`build-decision`: https://hg.mozilla.org/ci/ci-configuration/file/tip/build-decision .. _`Releng-RFC 36`: https://github.com/mozilla-releng/releng-rfcs/pull/36