Sync model and job scheduler
How integrations sync, retry, and fail, and how background jobs run.
Summary
- Each connected data source syncs on the frequency you choose (default daily). A failed sync is retried automatically; a sync that fails because the authorization has expired or been revoked stops until you reconnect.
- Everything that happens without a click is a background job: scheduled syncs, the daily scans and sweeps, and the work behind Generate and Sync now. Jobs run outside the web app and never on your browser's connection.
- Nothing runs on a timer per organization except your own syncs and training plans. The daily scans and sweeps run once for the whole platform and visit each organization in turn.
Concepts
| Term | Meaning |
|---|---|
| Data source | One connected integration: the provider, its configuration, encrypted credentials, a status, a sync frequency, and the time of the last and next sync |
| Data source status | Connected; Error (the last sync failed and will be retried); Needs reauthentication (the credentials need a human); Syncing (a manual sync is in flight) |
| Sync frequency | Hourly, Daily, Weekly, or Monthly, measured from the moment the previous sync finished |
| Sync history | One entry per sync with its outcome, a message, and counts of items processed, failures, and mappings generated |
| Job | One unit of background work with a type, a status (pending, running, completed, failed, or retrying), and a retry policy |
How it works
When a data source is due
The next sync time is computed from the moment the last one finished, plus the chosen frequency (a month is counted as 30 days). If the last sync ended in Error, the frequency is ignored and the next attempt is scheduled sooner. A data source that Needs reauthentication is never picked up automatically, because blindly retrying revoked credentials can lock the upstream account.
A due data source is skipped if a sync for it is already pending or running, so two syncs of the same source never overlap.
Sync now runs the same sync immediately, marks the source Syncing while it runs, and notifies you when it starts, finishes, or fails. Secrets never travel inside the job itself; the sync reads the encrypted credentials and refreshes any OAuth tokens at the moment it runs.
What one sync does and produces
- Tenant guard. The data source must belong to the organization the sync was scheduled for; otherwise the sync fails closed and nothing is written.
- Credential preparation. Decrypt stored credentials, refresh expiring tokens, and confirm the provider still answers.
- Collection. Read from the provider using only the permissions you granted (each provider page under Data sources lists what is and is not read). Each observation becomes an evidence record of type Automatic keyed by its identity in the provider. A record that already exists is updated in place; a record whose content has not changed only has its "last seen" time bumped, so its integrity digest is untouched. Control mappings are validated against the shared catalog; an unknown mapping is skipped with a warning rather than invented.
- Post-sync linking. Security findings are attached to assets and to pre-created risks, certificate domains are registered from domain evidence, third-party app grants are recorded, the privacy data map is refreshed from connector data, grant attribution proposals land in the review queue, and privacy drift is checked. Each of these is best-effort: a linking failure is logged and the sync still counts as a success.
- Outcome. The data source status is decided (see below) and a sync history entry is written.
A sync never deletes evidence. Evidence a provider stops returning simply stops being refreshed and keeps its last-seen time.
How background jobs run
Jobs are queued and executed by dedicated workers, separate from the web app. Each job is claimed by exactly one worker, so the same sync or scan cannot run twice at once. A job that fails is retried a small number of times with a delay between attempts; a job that reports a permanent failure (for example an authentication failure) is not retried. A worker that is restarted during maintenance re-queues the jobs it was running.
Notifications follow one rule: a job you started notifies you on success and failure; a job automation started notifies organization admins on failure and never on success.
What runs on its own
| Work | How often |
|---|---|
| Data source syncs | On each source's configured frequency |
| Evidence integrity attestation | Continuously; a new capture is chained and signed within minutes |
| Policy review roll-forward and reminders | Daily |
| Certificate scan of your configured domains | Daily |
| Privacy monitor scans of your monitored URLs | Daily |
| Vendor risk sweep (assessment of vendors not yet assessed) | Daily |
| AI inventory expiry sweep | Daily |
| Training campaign creation for recurring plans | On each plan's schedule |
Everything else is created when something happens: Generate for policies, organizational risks, and vendor risks; a vendor assessment when a vendor is created or updated; document gathering and AI inference when you work a questionnaire; privacy enrichment after a data-map ingestion; training invitations when a plan is activated; and a single certificate or privacy scan when you ask for one.
Edge cases and failure modes
How a failed sync is classified
The sync decides the data source's status from the error, and the status decides what happens next.
| What went wrong | Data source status | Next attempt |
|---|---|---|
| The provider answered but returned nothing usable | Stays Connected | Normal frequency |
| The provider rejected the credentials, the token has expired, or the app was uninstalled | Needs reauthentication | None until you reconnect |
| A transient network error | Unchanged | Retried automatically |
| Anything else | Error | Retried automatically, sooner than the normal frequency |
An authentication failure also creates an in-app notification for the person who authorized the connection, linking to Data Sources. Some providers report the reason, which is shown under Needs reauthentication.
Other things to know
- A sync can succeed with warnings. Post-sync linking is best-effort; check the sync history message if findings or data-map entries look incomplete.
- An empty sync is a data problem, not an auth problem. The source stays Connected and the next sync runs on schedule.
- A stuck job is recovered, not abandoned. A job that stops responding is failed and retried automatically.
- One regeneration at a time. Only one Generate job per kind is allowed to be active per organization; a second request while one is running is refused.
What you can influence
- Sync frequency per data source on the Data Sources page.
- Reconnect a data source that Needs reauthentication; nothing else will restart it.
- Sync now to run immediately with full notifications.
- Generate actions (policies, organizational risks, vendor risks) create their own jobs.
- Disconnect a data source to stop its syncs; existing evidence stays, with its last-seen time.
Do not rely on a retry after Needs reauthentication
Once a data source needs reauthentication, no schedule, sweep, or restart will touch it. Evidence from that source ages in place until someone reconnects.
Related guides
Last updated on