Agent Tasks

About 14 min

Agent Tasks

One of the more advanced features of Hazaar is its background task scheduler, which is handled by the Warlock Agent. It is completely transparent and lets you run code in the background, after a delay, at a specific date and time, or on a recurring schedule, all within the context of your application. The scheduler is very light but powerful, and so does not consume many resources.

This page covers:

running tasks from application code, with exec(), execDelay(), schedule() and interval();
configured schedules, which run on a CRON schedule without any application code;
catching up missed schedules that fell due while the Agent wasn't running.

Overview

The scheduler runs as a single loop within the Warlock Agent process. Tasks are executed in their own child processes so that they can't hang up or kill the Agent. The number of tasks that run in parallel is limited, so that resources aren't consumed out of control should a rogue task be doing something it shouldn't. Everything is logged, making it easy to identify any problems.

Methods

The Hazaar\Warlock\Client has four methods for running code on an Agent. In each one, $callable is a closure, a Class::method string or a [Class, method] array. $params are passed to it as positional arguments. Each returns the task ID on success, or false.

$warlock = new Hazaar\Warlock\Client();
$warlock->connect();

// Run now, in the background.
$warlock->exec($callable, $params = [], $tag = null, $overwrite = false, $timeout = null);

// Run once, after $delay seconds.
$warlock->execDelay($delay, $callable, $params = [], $tag = null, $overwrite = false, $timeout = null);

// Run once, at $when (a Hazaar\Util\DateTime).
$warlock->schedule($when, $callable, $params = [], $tag = null, $overwrite = false, $timeout = null);

// Run every $seconds seconds, starting $seconds from now, until cancelled.
$warlock->interval($seconds, $callable, $params = [], $tag = null, $overwrite = false, $timeout = null);

// Cancel any of the above by the ID it returned.
$warlock->cancelTask($taskID);

schedule() and interval() take a Class::method string, since the Agent has to be able to load the class.

$timeout is the most a single run may take, in seconds, overriding the Agent's process.timeout. 0 disables it. It includes the time the run takes to bootstrap the application. A run that times out is killed and reported to the error handler, and isn't retried.

Warning

The Agent may be configured with a different timezone from your application. When you schedule something for a specific time, include the timezone in the DateTime you pass to schedule().

Example Usage

Consider the following example code:

$warlock = new Hazaar\Warlock\Client();
$warlock->connect();

$code = function() {
    echo "APPLICATION_PATH = " . APPLICATION_PATH . "\n";
    echo "APPLICATION_ENV  = " . APPLICATION_ENV . "\n";
};

if ($warlock->execDelay(5, $code)) {
    $this->redirect($this->url());
}

throw new \Exception('Unable to execute delayed function');

The above is a very simple example of how to use the execDelay() method to run a function after a certain period of time has elapsed. We will now step through each line and outline what is happening.

1 - Instantiate the warlock client object

$warlock = new Hazaar\Warlock\Client();
$warlock->connect();

This sets up the warlock client object, which handles all communication between the application and the Warlock server.

2 - Create an 'anonymous function' (aka Closure)

In PHP, anonymous functions are implemented using the Closure class. The syntax to create an anonymous function is just the same as most other languages that allow anonymous functions.

$code = function() {
    echo "APPLICATION_PATH = " . APPLICATION_PATH . "\n";
    echo "APPLICATION_ENV  = " . APPLICATION_ENV . "\n";
};

The function that we have created simply echoes out the current APPLICATION_PATH and APPLICATION_ENV environment variables. This will prove that the function is being executed in the correct application context. This output will appear in the Agent's log.

3 - Schedule the code for execution

$warlock->execDelay(5, $code);

In the above code we send the function to the execDelay() method and tell it to run the function in 5 seconds.

To run something at a specific time, use schedule(), and to run it repeatedly, use interval():

$warlock->schedule(new Hazaar\Util\DateTime('today 5pm Australia/Sydney'), 'App\Models\Reports::daily');
$taskID = $warlock->interval(300, 'App\Models\Cache::refresh');

// Later, stop it.
$warlock->cancelTask($taskID);

Running the Agent

Tasks are run by the Agent, so it must be running. From your application's root directory:

$ vendor/bin/warlock agent run

The Agent will start and you will see its log output in your terminal. See the Warlock CLI for all of its options, including running it as a daemon.

Calling static class methods

As well as closures, you can call static class methods. A static method works almost exactly the same as a closure, except that the code can be defined anywhere in your project.

For example:

namespace App\Models;

class MyTestClass
{
    public static function doTheThing(){
        error_log('The thing is done!');
    }
}

If you need access to the service context (for example, to call trigger() or log()), use a closure or an instance method instead of a static method.

To run this method in the background you can pass the callable in one of the following ways:

$warlock->execDelay(30, ['App\Models\MyTestClass', 'doTheThing']);

is the same as

$warlock->execDelay(30, 'App\Models\MyTestClass::doTheThing');

It's dealer's choice which you use.

Passing arguments

exec() runs a callable in the background straight away, and execDelay() runs it after a delay. Both take the callable's arguments as an array, which are passed to it as positional arguments:

$warlock = new Hazaar\Warlock\Client();
$warlock->connect();

// Runs App\Models\Invoice::send(1234, '[email protected]') now...
$warlock->exec(['App\Models\Invoice', 'send'], [1234, '[email protected]']);

// ...or in 60 seconds.
$warlock->execDelay(60, ['App\Models\Invoice', 'send'], [1234, '[email protected]']);

Arguments travel to the agent as JSON, so pass scalars and arrays. An associative array arrives as a stdClass object, and objects don't keep their class.

Configured Schedules

Tasks can also be scheduled in the Agent's configuration, so that they run without being started by application code. This lets you run a static method on a schedule.

Using the above example class MyTestClass, to run doTheThing() every hour on business days between 9am and 5pm, add a schedule to the Agent configuration file (configs/agent.json by default, or the file given to warlock agent run --config):

PHP

<?php
return [
    'development' => [
        'schedule' => [
            [
                'tag' => 'do-the-thing',
                'when' => '0 9-17 * * 1-5',
                'action' => 'App\Models\MyTestClass::doTheThing',
            ],
        ],
    ],
];

JSON

{
    "development": {
        "schedule": [
            {
                "tag": "do-the-thing",
                "when": "0 9-17 * * 1-5",
                "action": "App\\Models\\MyTestClass::doTheThing"
            }
        ]
    }
}

When the scheduled time comes around, a new Runner process is started and App\Models\MyTestClass::doTheThing() is called.

Warning

In JSON, and in double-quoted PHP strings, the backslashes in a class name must be doubled (App\\Models\\MyTestClass). Otherwise the class name is parsed incorrectly and the method is not called.

Each schedule accepts these settings:

Setting	Required	Description
`action`	Yes	The callable to run: a `Class::method` string or a `[Class, method]` array.
`when`	Yes	When to run it. A CRON expression (e.g. `0 9-17 * * 1-5`) runs it on that schedule. `@reboot` runs it once, `task.boot_delay` seconds after the Agent starts. A Unix timestamp runs it once at that time.
`args`	No	An argument to pass to the callable. It is passed as a single argument, so a map arrives as one `stdClass` object (or array), not as separate arguments.
`tag`	No	A unique name for the schedule. A second schedule with the same tag is not registered, unless it sets `overwrite`. Recommended for every schedule, and needed for catch-up to keep track of it reliably.
`overwrite`	No	If `true`, this schedule replaces an earlier one with the same `tag`. Defaults to `false`.
`timeout`	No	The most a single run may take, in seconds, overriding `process.timeout`. `0` disables it.
`catchup`	No	Set to `false` to opt this schedule out of catch-up. Defaults to `true` when catch-up is enabled.
`catchup_max_age`	No	Overrides `task.catchup.max_age` for this schedule.
`catchup_buffer`	No	Overrides `task.catchup.buffer` for this schedule.
`catchup_rerun_interrupted`	No	Overrides `task.catchup.rerun_interrupted` for this schedule.

Catching Up Missed Schedules

By default, a configured schedule that falls due while the Agent isn't running is simply missed: when the Agent starts again, the schedule waits for its next occurrence. For a task that runs every few minutes that rarely matters, but if a nightly report or a twice-daily distribution is missed because the Agent was being upgraded or the server was rebooted, it won't run until the next day.

The Agent can instead track the executions of each schedule and catch up the ones it missed when it starts, much like anacron does after a system reboot. When catch-up is enabled:

The Agent records each execution of every CRON schedule in a history file.
When the Agent starts, it compares each schedule's most recent due time with the history.
A schedule that missed an execution is run once, however many occurrences were missed, shortly after the Agent starts (task.boot_delay seconds), and then carries on with its normal schedule.

What matters is whether the Agent was running. Configured schedules are run by the Agent, so restarting the Warlock Server on its own doesn't make the Agent miss them.

Tips

Turning catch-up on never runs anything straight away. The first time the Agent sees a schedule, it only records it, so catch-up starts protecting a schedule from its next occurrence onwards.

Enabling catch-up

Catch-up is disabled by default. Enable it with task.catchup.enabled in the Agent configuration. Every CRON schedule then catches up, unless it opts out with catchup: false.

PHP

<?php
return [
    'development' => [
        'task' => [
            'boot_delay' => 5,
            'catchup' => [
                'enabled' => true,
                'stagger' => 10,
            ],
        ],
        'schedule' => [
            [
                'tag' => 'distribute',
                'when' => '0 0,12 * * *',
                'action' => 'App\Models\Distribution::run',
                'catchup_max_age' => 28800,
                'catchup_buffer' => 3600,
            ],
            [
                'tag' => 'nightly-report',
                'when' => '30 2 * * *',
                'action' => 'App\Models\Reports::nightly',
            ],
            [
                'tag' => 'check-failed',
                'when' => '15 * * * *',
                'action' => 'App\Models\Distribution::checkFailed',
                'catchup' => false,
            ],
        ],
    ],
];

JSON

{
    "development": {
        "task": {
            "boot_delay": 5,
            "catchup": {
                "enabled": true,
                "stagger": 10
            }
        },
        "schedule": [
            {
                "tag": "distribute",
                "when": "0 0,12 * * *",
                "action": "App\\Models\\Distribution::run",
                "catchup_max_age": 28800,
                "catchup_buffer": 3600
            },
            {
                "tag": "nightly-report",
                "when": "30 2 * * *",
                "action": "App\\Models\\Reports::nightly"
            },
            {
                "tag": "check-failed",
                "when": "15 * * * *",
                "action": "App\\Models\\Distribution::checkFailed",
                "catchup": false
            }
        ]
    }
}

With this configuration, if the Agent is stopped at 11:00 and started again at 14:00:

distribute missed its 12:00 run. It was missed 2 hours ago, inside its 8 hour window (catchup_max_age), and its next run at 00:00 is well outside its 1 hour buffer (catchup_buffer), so it is caught up at 14:00:05. Had the Agent been down until 21:00, the catch-up would have been skipped, because by then the 12:00 run was missed more than 8 hours ago.
nightly-report didn't miss anything, as it last fell due at 02:30, which ran. Had the Agent also been down at 02:30, it would be caught up at 14:00:15, 10 seconds (stagger) after distribute.
check-failed missed its 11:15, 12:15 and 13:15 runs, but has opted out, so it just runs at 14:15 as usual.

Options

These are set under task.catchup in the Agent configuration. Most can be overridden for an individual schedule.

Setting (`task.catchup`)	Per-schedule override	Default	Description
`enabled`	`catchup`	`false`	Enables schedule tracking and catch-up. When it is enabled, every CRON schedule catches up unless it sets `"catchup": false`. A schedule can't opt in when catch-up is disabled.
`max_age`	`catchup_max_age`	`0`	The catch-up window, in seconds. A missed execution that is older than this is skipped. `0` means there is no limit.
`buffer`	`catchup_buffer`	`0`	In seconds. A missed execution is skipped if the schedule's next regular run is due within this long of the catch-up, so that the task doesn't run twice in quick succession. `0` means a catch-up is never skipped for this reason.
`stagger`		`0`	Seconds between successive catch-ups, so that a long outage doesn't start a burst of tasks at once. The first catch-up runs `task.boot_delay` seconds after startup, the second `stagger` seconds after that, and so on.
`rerun_interrupted`	`catchup_rerun_interrupted`	`false`	Run an execution again if it was interrupted because the Agent stopped while it was running. See What counts as an execution.
`file`		`schedule.db`	The history file. A relative path is in the application's runtime directory (`app.runtimePath`). An absolute path is used as it is.

task.boot_delay (default 5) also delays the first catch-up, to give the rest of the system time to start.

How a missed run is decided

When the Agent starts, it checks each CRON schedule in turn. For each one, it works out the schedule's most recent due time, the time it last fell due before now, and looks up the schedule's history. The first rule that matches decides what happens:

No history. The schedule is new, or catch-up has just been enabled. Its most recent due time is recorded as a baseline and it doesn't run.
The when expression has changed. The old history no longer describes this schedule, so a new baseline is recorded and it doesn't run.
The history is in the future. The system clock has gone backwards. A new baseline is recorded and it doesn't run.
The schedule has opted out (catchup: false). The history is moved on to the most recent due time, so that it is up to date if the schedule opts back in, and it doesn't run.
The most recent due time was already handled. It ran, or was recorded as a baseline or skipped, so there is nothing to catch up. The exception is an execution that was interrupted (the Agent stopped while it was running) when rerun_interrupted is set: that execution runs again.
The missed execution is older than max_age. Skipped.
The next regular run is due before the catch-up would start. Skipped, because the regular run is about to happen anyway.
The next regular run is due within buffer seconds of the catch-up. Skipped.
Otherwise, a catch-up is scheduled at now + boot_delay + n × stagger, where n is the number of catch-ups already scheduled.

Skipped catch-ups are written to the log, with the reason, and recorded in the history with the status skipped.

Examples

The basic case. A task runs twice a day (0 0,12 * * *). The Agent is stopped at 11:00 for an upgrade and started again at 14:00. The most recent due time is 12:00, which never ran, so the task runs at 14:00 plus boot_delay. Its next run is at 00:00, as normal.

A long outage. The same Agent is down from Monday to Thursday afternoon, missing seven runs. Only one catch-up runs, for the most recent of them (Thursday 12:00). Catch-up exists to make sure the task has run since it last fell due, not to replay every run that was missed.

The buffer. A task runs every hour (0 * * * *), with catchup_buffer set to 600 and boot_delay at its default of 5. The Agent is down from 13:50 and started again at 14:50, missing the 14:00 run. The catch-up would start at 14:50:05, and the next regular run at 15:00 is within 10 minutes of that, so the catch-up is skipped and the task runs at 15:00 as normal. Without the buffer, it would run at 14:50 and again at 15:00.

The window. The twice-daily task has catchup_max_age set to 28800 (8 hours). The Agent is stopped at 11:00 on Friday and started again at 21:00. The 12:00 run was missed 9 hours ago, outside the window, so it is skipped and the task next runs at 00:00. Use a window for tasks whose results go stale: there is no point sending a midday notification in the evening.

First enable. Catch-up is enabled for the first time on an Agent with existing schedules. Nothing runs: the Agent only records a baseline for each schedule. From then on, any occurrence it misses is caught up.

Schedule identity and tags

The history keeps one record per schedule, so the Agent needs to recognise a schedule from one start to the next.

A schedule with a tag is identified by its tag. You can change its action, args or anything else, and it keeps its history. Changing its when expression starts a new baseline, as above.
A schedule without a tag is identified by its action and args. If you change either of them, the Agent sees a new schedule: the old history is removed and a new baseline recorded, so an occurrence missed while you were making the change won't be caught up. Two untagged schedules with the same action and args can't be told apart, so only the first is tracked and a warning is logged.

Give every schedule a unique tag. Tags can be any length. They are hashed (xxh128) to form the history's keys, so a long tag costs nothing. A schedule with the same tag as an earlier one isn't registered at all, unless it sets overwrite.

What counts as an execution

An execution counts as soon as it starts, not when it finishes. If it fails, the Agent retries it as usual (task.retries), and the retries are part of the same execution, so they don't count as new ones. Once it has finished, its history record has one of these statuses:

Status	Meaning	Caught up again?
`complete`	It finished successfully.	No
`error`	It failed, including all of its retries.	No
`timeout`	It was killed for running past its timeout.	No
`running`	It was still running when the Agent stopped, and the Agent was killed before it could record anything else.	Only with `rerun_interrupted`
`cancelled`	It was cancelled while running, usually because the Agent was shutting down.	Only with `rerun_interrupted`
`baseline`	It was recorded without running.	No
`skipped`	Its catch-up was skipped by `max_age`, `buffer`, or because the next run was due first.	No

A failed execution isn't caught up, because catch-up is for runs that were missed, not runs that went wrong. Use an error handler to deal with failures.

Warning

rerun_interrupted runs an interrupted execution again from the start. Only enable it for tasks that are safe to run twice, because an interrupted task may have done some or all of its work before it was stopped.

What isn't caught up

Only CRON schedules from the Agent configuration are tracked. These are not:

@reboot schedules, which run every time the Agent starts anyway;
schedules with a Unix timestamp as their when;
anything started by application code with exec(), execDelay(), schedule() or interval(). These only live in the Agent's memory, and are lost if it stops.

The history file

The history is kept in a Hazaar\Util\BTree file, schedule.db in the application's runtime directory by default (see task.catchup.file). Each execution is recorded with a single small, in-place write.

Records of schedules that are removed from the configuration are deleted when the Agent starts, and the file is then compacted.
It is safe to delete the file while the Agent is stopped. The next time it starts, every schedule is recorded as a new baseline, as if catch-up had just been enabled.
If the file is unreadable or corrupt, it is moved aside to schedule.db.corrupt-<timestamp> and a new history is started. The schedules are then baselined, so nothing is caught up that time.
If the file can't be created or written, catch-up is disabled and an error is logged. It never stops the Agent from starting or from running its tasks.

Each record contains:

Field	Description
`tag`	The schedule's tag, or `null` if it doesn't have one.
`action`	The scheduled action.
`when`	The CRON expression the record was made against.
`due`	The occurrence the last execution represents, as a Unix timestamp. This is when it was due, not when it ran, so a catch-up of the 12:00 run that ran at 14:00 records 12:00.
`started`	When the last execution started. Not set for a schedule that has only been baselined.
`finished`	When the last execution finished, or `null` if it hasn't.
`status`	One of the statuses in What counts as an execution.
`exitCode`	The exit code of the last execution's process, if it has finished.
`catchup`	`true` if the last execution was a catch-up.

To look at the history, open it with Hazaar\Warlock\Agent\Schedule\History. This is safe to do while the Agent is running, because each read and write locks the file:

use Hazaar\Warlock\Agent\Schedule\History;
use Hazaar\Warlock\Logger;

$history = History::open('/path/to/app/.runtime/schedule.db', new Logger());
foreach ($history->all() as $record) {
    printf("%-20s %-10s due %s\n", $record['tag'] ?? $record['action'], $record['status'], date('c', $record['due']));
}

Deployment

Containers and Kubernetes. The history must survive the Agent's container being replaced. Put the runtime directory on a persistent volume, or point task.catchup.file at an absolute path on one. Otherwise each new container starts with no history, so nothing is ever caught up (and nothing is wrongly run either).
More than one Agent. Each Agent keeps its own history and catches up independently. If several Agents run the same schedules, they already each run every occurrence, and each one will also catch up a missed occurrence. Run configured schedules on one Agent only.
Clocks and timezones. Schedules are evaluated in the Agent's timezone (UTC by default). If the system clock goes backwards, so that the history is in the future, the schedule is re-baselined rather than run. If you change the Agent's timezone, a schedule's most recent due time moves, which can cause one catch-up or skip one occurrence.

Logging

Catch-up writes these messages to the Agent's log:

Level	Message
`NOTICE`	`Opening the schedule history: <file>`
`DEBUG`	`Schedule '<name>' has no history. Recording a baseline at <time>`
`NOTICE`	`Schedule '<name>' changed from '<old>' to '<new>'. Recording a new baseline.`
`WARN`	`The history of schedule '<name>' is in the future. Has the system clock changed? Recording a new baseline.`
`NOTICE`	`The last execution of schedule '<name>', due at <time>, was interrupted.`
`NOTICE`	`Skipping the catch-up of schedule '<name>', missed at <time>, because <reason>.`
`NOTICE`	`Schedule '<name>' missed its execution at <time>. Running a catch-up at <time>`
`NOTICE`	`Scheduled <n> catch-up executions`
`NOTICE`	`Running the catch-up of schedule '<name>', missed at <time>`
`WARN`	`Schedule '<name>' has the same identity as another schedule, so it can't be caught up. Give each schedule a unique tag.`
`NOTICE`	`Removing the schedule history of '<name>', which is no longer scheduled`
`ERROR`	`The schedule history file is unreadable or corrupt (<error>). Moving it to <file> and starting a new history.`
`ERROR`	`Schedule catch-up is disabled because the schedule history is unavailable.`, after a message giving the reason.
`ERROR`	`Unable to write the schedule history: <error>`

<name> is the schedule's tag, or its action if it doesn't have one. The reasons a catch-up is skipped are:

it is older than the catch-up window of <max_age> seconds
the next scheduled execution is due first
the next scheduled execution is within the catch-up buffer of <buffer> seconds