# Heartbeat monitors

> Watch cron jobs, backups and workers: your job calls a secret StatusTick URL at start and end, and StatusTick alerts on missed, late, failed or stuck runs.

A heartbeat monitor watches something that runs on a schedule: a backup, a cron job, a queue worker. Instead of StatusTick calling your service, your job calls StatusTick each time it runs.

## How it works

1. Create a heartbeat monitor and set how often the job runs. You can set a simple interval (for example every 24 hours) or a cron expression with a time zone.
2. Set a **grace period**: how far from its scheduled time a run may start before it counts as late or missed.
3. Copy the monitor's unique ping URL and call it from your job: when it starts, and when it ends.

If no run arrives within the scheduled time plus the grace period, the monitor goes **down** and the normal incident and alert flow runs.

StatusTick keeps every run as a record: when it was scheduled, when it started and finished, how long it took, and how it ended. The API lists them with `GET /v1/monitors/{monitorId}/runs`, and the MCP server with `list_job_runs`.

## The ping URLs

Ping URLs look like `https://webhook.statustick.com/v1/ping/<token>`. A `GET`, `POST` or `HEAD` request counts as a ping. For example, at the end of a shell script:

```bash runnable
# Replace the URL with the ping URL from your monitor page
curl -fsS --retry 3 "$STATUSTICK_PING_URL" > /dev/null
```

The URL has two more endpoints:

| URL | Use it to |
| --- | --- |
| `<ping URL>` | Report that the run finished successfully. |
| `<ping URL>/start` | Report that a run started. |
| `<ping URL>/fail` | Report that the run failed. The monitor goes down at once, without waiting for the grace period. |

For example:

```bash runnable
curl -fsS "$STATUSTICK_PING_URL/start" > /dev/null
if ./backup.sh; then
  curl -fsS "$STATUSTICK_PING_URL" > /dev/null
else
  curl -fsS "$STATUSTICK_PING_URL/fail" > /dev/null
fi
```

### Why send /start

A success ping alone tells StatusTick that a run finished. With `/start` as well, StatusTick also knows:

- how long each run took,
- that a run is still going, so it can tell when a run overlaps the next one,
- that a run started and never finished, or did not finish in time,
- how long a run usually takes, so it can tell when one is running long.

Without `/start`, StatusTick can only report missed, late, off-schedule, repeated and failed runs.

### The run parameter

Add `?run=<id>` to every ping of one run, with the same id: the start, then the success or the failure. StatusTick then ties each start to its own finish, even when runs overlap or a finish arrives late. The id is 1 to 64 letters, digits, `-` or `_`; any other value gets `400 Invalid run parameter` and the ping is not recorded. Use something unique per run, such as the job's start time or your scheduler's job id.

Without a run id, a finish closes the oldest run that is still open. That is right as long as runs never overlap.

The same ping with the same run id within 60 seconds is recorded once, so a retry does not count as a second run.

Jobs inside a private network can send the same pings, run id included, to a private agent's [heartbeat relay](https://statustick.com/docs/private-agents#heartbeat-relay).

## Examples

Each example sends a start, then a success or a failure, with one run id. Set `STATUSTICK_PING_URL` to the monitor's ping URL.

### Crontab

A small wrapper script runs any command and reports it:

```bash
#!/bin/sh
# /usr/local/bin/statustick-run: statustick-run <command> [args...]
RUN="$(date +%Y%m%d%H%M%S)-$$"
curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/start?run=$RUN" > /dev/null
if "$@"; then
  curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL?run=$RUN" > /dev/null
else
  code=$?
  curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/fail?run=$RUN" > /dev/null
  exit $code
fi
```

```bash
# crontab -e
STATUSTICK_PING_URL=https://webhook.statustick.com/v1/ping/<token>
0 2 * * * /usr/local/bin/statustick-run /opt/backup/backup.sh
```

### Kubernetes CronJob

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: registry.example.com/backup:1.4.0
              env:
                - name: STATUSTICK_PING_URL
                  valueFrom:
                    secretKeyRef:
                      name: statustick
                      key: ping-url
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              command: ["/bin/sh", "-c"]
              args:
                - |
                  RUN="$POD_NAME"
                  curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/start?run=$RUN"
                  if /app/backup.sh; then
                    curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL?run=$RUN"
                  else
                    curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/fail?run=$RUN"
                    exit 1
                  fi
```

The pod name is unique per run, so it makes a good run id.

### systemd timer

```ini
# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
EnvironmentFile=/etc/statustick/backup.env
ExecStartPre=/bin/sh -c 'curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/start?run=$INVOCATION_ID"'
ExecStart=/opt/backup/backup.sh
ExecStopPost=/bin/sh -c 'if [ "$SERVICE_RESULT" = success ]; then curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL?run=$INVOCATION_ID"; else curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/fail?run=$INVOCATION_ID"; fi'
```

```ini
# /etc/systemd/system/backup.timer
[Unit]
Description=Run the backup every night at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

`/etc/statustick/backup.env` holds `STATUSTICK_PING_URL=...`. systemd gives every run its own `INVOCATION_ID`, and `ExecStopPost` runs whether the job succeeded or not.

### GitHub Actions schedule

```yaml
name: Nightly report
on:
  schedule:
    - cron: "0 2 * * *"
jobs:
  report:
    runs-on: ubuntu-latest
    env:
      STATUSTICK_PING_URL: ${{ secrets.STATUSTICK_PING_URL }}
      RUN: ${{ github.run_id }}-${{ github.run_attempt }}
    steps:
      - run: curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/start?run=$RUN"
      - uses: actions/checkout@v4
      - run: ./scripts/report.sh
      - if: success()
        run: curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL?run=$RUN"
      - if: failure()
        run: curl -fsS -m 10 --retry 3 "$STATUSTICK_PING_URL/fail?run=$RUN"
```

GitHub starts scheduled workflows late at busy times. Give the monitor a grace period that covers that, for example 30 minutes.

## Run problems

StatusTick checks each run against the schedule. Times in alerts are in the monitor's time zone.

- **Down** problems (missed, not finished on time, lost, failed) open an incident and alert through the monitor's alert policy. A monitor that is down stays down, with one incident, until a run finishes successfully. If a run has a down and a degraded problem, down wins.
- **Degraded** problems (late, off schedule, double fire, overlap, running long) open no incident and page nobody. The first time each one appears since the monitor was last up, StatusTick sends one warning to the destinations in step 1 of the alert policy.

Every alert and warning names the problem and links to the run on the monitor's **Runs** tab.

### Missed

- **When:** a scheduled time passed, plus the grace period, and no run started or finished for it.
- **Monitor:** down.
- **Alert:** "Missed the 02:00 run".

### Late

- **When:** a run arrived after the grace period of its scheduled time, before the next one. It takes the place of the missed run for that time.
- **Monitor:** degraded; up again when the run finishes successfully.
- **Alert:** a warning, "The 02:00 run started late, at 02:41".

### Off schedule

- **When:** a run matches no scheduled time, for example someone ran the job by hand.
- **Monitor:** degraded.
- **Alert:** a warning, "Ran at 14:12, outside its schedule".

### Double fire

- **When:** a second run arrives for a scheduled time that already has one, for example the job runs on two machines. On an interval monitor: a run that starts less than the interval minus the grace period after the previous one. A new run after a failed run of the same time is a retry, not a double fire, and sends no alert.
- **Monitor:** degraded.
- **Alert:** a warning, "Ran twice for the 02:00 run (started 02:00:01 and 02:00:03)".

### Overlap

- **When:** a run sends `/start` while an earlier run is still running. Needs `/start`.
- **Monitor:** degraded.
- **Alert:** a warning, "The 03:00 run started while the 02:00 run was still running (running for 1 h 2 min)".

### Running long

- **When:** a run is still open after much longer than it usually takes: twice its usual run time, and at least 1 minute more. StatusTick learns the usual run time as the median of the last 20 successful runs that sent `/start`, leaving out runs during a maintenance window. It starts after 10 such runs; until then running long is off. The API shows it on the monitor as `usualRunTimeMs` and `runningLongAfterMs`. Needs `/start`.
- **Monitor:** degraded; up again when the run finishes successfully. That run then counts toward the usual run time.
- **Alert:** a warning, "Running for 47 min; usually 12 min".

### Not finished on time

- **When:** a run is still open when its scheduled time plus **finish within** has passed. Finish within is optional, 1 minute to 24 hours and longer than the grace period; set it through the API (`finishWithinSeconds`). Needs `/start`.
- **Monitor:** down.
- **Alert:** "The 02:00 run did not finish by 04:00".

### Lost

- **When:** a run started and sent no finish by the next scheduled time plus the grace period (and after finish within, if set). Needs `/start`. If its finish arrives later with its run id, StatusTick records it as finished late.
- **Monitor:** down.
- **Alert:** "The 02:00 run started and never finished".

### Failed

- **When:** the job calls `<ping URL>/fail`.
- **Monitor:** down at once.
- **Alert:** "The job reported a failure".

Paused monitors record no runs. During a maintenance window runs are still recorded; only the alerts are held back.

## Keep the URL secret

Anyone with the ping URL can report pings for the monitor. Store it like a password, for example in an environment variable or a secret store. If it leaks, make a new one from the monitor page; the old URL keeps working for 60 minutes so you can update the job, then stops.
