Consent Docs
Open console

Scheduled jobs

Every job the API runs on a schedule, when it runs, what it does and whether one replica or every replica does the work.

Reference4 min readFor project administrators and deployment operators

The API runs these jobs by itself; you do not set up cron.

How jobs run#

  • Every replica schedules every job. A job marked One takes a PostgreSQL advisory lock, so one replica does the work.
  • Times follow the time zone of the API process. That is UTC in the supplied image. The two anchor jobs always use UTC.
  • A missed job catches up. Each run works from the state of the database, not from a calendar.
  • A failed job writes a log line with "level":"error" and raises nothing else. A failed vendor list refresh is logged with "level":"warn". See the log lines to alert on.

Every 15 seconds#

Job What it does Replicas
Webhook outbox Sends up to 50 due events One
Consent Manager notifications Sends up to 50 due notifications to Consent Managers, retried as webhook events are One
Notice campaign work Makes the links of a queued batch, up to 5,000 at a time, and sends campaign.batch_issued. Completes withdrawals made on notice links that are over a minute old, and sets notice_withdrawals_pending One

Every hour#

Job What it does Replicas
Expiry sweep Marks consents past their expiry as EXPIRED, seals the change and sends consent.expired One
Inactivity erasure For projects with Erase after inactivity: sends subject.erasure_notice 72 hours ahead, then erases up to 500 people per project per run. Skips a project with no subscriber to the notice One
Breach reminders Sends breach.report_due 12 hours before a Board report is due One
Scheduled scans Queues weekly or monthly site scans that are due One
Usage alerts Sends usage.threshold_reached at 80% and 100% of a project's monthly limit One
Statistics roll-up Recomputes the last day's consent statistics One
Idempotency clean-up Deletes stored responses older than 24 hours One
Import clean-up Stops imports with no batch for 30 minutes and sends consent.imported. Discards checks not written within 24 hours One
Notice link copies Clears the encrypted copies of notice links whose batch can no longer be downloaded, 7 days after issue One

Every day#

Time Job What it does Replicas
00:15 UTC Daily anchors Anchors every closed day that has sealed events and no anchor yet, and sends evidence.anchored One
00:30 UTC Campaign anchors Anchors every closed day that has notice answers or issued batches, and sends campaign.anchored One
02:00 Vendor list refresh Downloads the IAB vendor list if the cached copy is older than 7 days Every
02:30 Project closure Deletes the data of each project whose closure date has passed, table by table, 5,000 rows at a time, and sends project.deleted. Waits while a legal hold is in force. Removes the project's webhook endpoints once that event is delivered, or after 7 days. See Close a project One
03:00 Evidence retention Only with ENFORCE_EVIDENCE_RETENTION=true. Deletes records whose consent ended and whose retention period is over, up to 500 per project per run. Writes what it deleted to the audit log One
04:00 Sandbox clean-up Deletes test-key records not changed for 30 days, up to 5,000 per run One
04:00 Held details on erased records Clears the details that DPDP Rule 8(3) kept for a year after an erasure Every. The statement is safe to repeat
05:00 Webhook log clean-up Deletes delivery logs and settled outbox events older than 30 days, failed events included One
05:00 Consent Manager notification clean-up Deletes delivered and failed notifications to Consent Managers older than 30 days One
06:00 Processor overdue check Sends one propagation.overdue for each processor with overdue live tasks, and sets processor_tasks_overdue One
07:00 DPDP rule set notice Sends one ruleset.changed to each project whose acknowledged rule set differs from the one in force. Also runs a minute after start and after the operator changes a date. See DPDP rule set One
08:00 Request reminders Sends subject_request.due_soon for open requests due within five days or overdue One

Jobs you can start by hand#

Job How
Daily anchors In the console, open Audit. Under Evidence anchors, choose Anchor closed days now. It anchors the closed days of that project
Every other job No manual trigger

Evidence retention#

Retention is off by default. Turning it on is a decision for the data owner, not only for the operator. Deletion cannot be undone.

docker-compose.override.ymlYAML
services:
  consent-api:
    environment:
      ENFORCE_EVIDENCE_RETENTION: 'true'

With it on, the job deletes a record only when all of these hold:

  • The consent has ended. A record still in force is never deleted for age.
  • The retention period is over. It counts from the record's expiry date. A record with no expiry date is never deleted.
  • A calendar year has passed since its last processing or last event.

The job keeps a hash of each deleted event, so earlier anchors still verify.

After a restart#

What What happens
Webhook events They wait in the database outbox. Deliveries continue
A job that was due while the API was down It catches up on its next run
Site scans that were queued or running They are marked failed. Start them again

↑↓ to move, Enter to open