Sync model and job scheduler

How integrations sync, retry, and fail, and how background jobs run.

Summary

  • Each connected data source syncs on the frequency you choose (default daily). A failed sync is retried automatically; a sync that fails because the authorization has expired or been revoked stops until you reconnect.
  • Everything that happens without a click is a background job: scheduled syncs, the daily scans and sweeps, and the work behind Generate and Sync now. Jobs run outside the web app and never on your browser's connection.
  • Nothing runs on a timer per organization except your own syncs and training plans. The daily scans and sweeps run once for the whole platform and visit each organization in turn.

Concepts

TermMeaning
Data sourceOne connected integration: the provider, its configuration, encrypted credentials, a status, a sync frequency, and the time of the last and next sync
Data source statusConnected; Error (the last sync failed and will be retried); Needs reauthentication (the credentials need a human); Syncing (a manual sync is in flight)
Sync frequencyHourly, Daily, Weekly, or Monthly, measured from the moment the previous sync finished
Sync historyOne entry per sync with its outcome, a message, and counts of items processed, failures, and mappings generated
JobOne unit of background work with a type, a status (pending, running, completed, failed, or retrying), and a retry policy

How it works

When a data source is due

The next sync time is computed from the moment the last one finished, plus the chosen frequency (a month is counted as 30 days). If the last sync ended in Error, the frequency is ignored and the next attempt is scheduled sooner. A data source that Needs reauthentication is never picked up automatically, because blindly retrying revoked credentials can lock the upstream account.

A due data source is skipped if a sync for it is already pending or running, so two syncs of the same source never overlap.

Sync now runs the same sync immediately, marks the source Syncing while it runs, and notifies you when it starts, finishes, or fails. Secrets never travel inside the job itself; the sync reads the encrypted credentials and refreshes any OAuth tokens at the moment it runs.

What one sync does and produces

  1. Tenant guard. The data source must belong to the organization the sync was scheduled for; otherwise the sync fails closed and nothing is written.
  2. Credential preparation. Decrypt stored credentials, refresh expiring tokens, and confirm the provider still answers.
  3. Collection. Read from the provider using only the permissions you granted (each provider page under Data sources lists what is and is not read). Each observation becomes an evidence record of type Automatic keyed by its identity in the provider. A record that already exists is updated in place; a record whose content has not changed only has its "last seen" time bumped, so its integrity digest is untouched. Control mappings are validated against the shared catalog; an unknown mapping is skipped with a warning rather than invented.
  4. Post-sync linking. Security findings are attached to assets and to pre-created risks, certificate domains are registered from domain evidence, third-party app grants are recorded, the privacy data map is refreshed from connector data, grant attribution proposals land in the review queue, and privacy drift is checked. Each of these is best-effort: a linking failure is logged and the sync still counts as a success.
  5. Outcome. The data source status is decided (see below) and a sync history entry is written.

A sync never deletes evidence. Evidence a provider stops returning simply stops being refreshed and keeps its last-seen time.

How background jobs run

Jobs are queued and executed by dedicated workers, separate from the web app. Each job is claimed by exactly one worker, so the same sync or scan cannot run twice at once. A job that fails is retried a small number of times with a delay between attempts; a job that reports a permanent failure (for example an authentication failure) is not retried. A worker that is restarted during maintenance re-queues the jobs it was running.

Notifications follow one rule: a job you started notifies you on success and failure; a job automation started notifies organization admins on failure and never on success.

What runs on its own

WorkHow often
Data source syncsOn each source's configured frequency
Evidence integrity attestationContinuously; a new capture is chained and signed within minutes
Policy review roll-forward and remindersDaily
Certificate scan of your configured domainsDaily
Privacy monitor scans of your monitored URLsDaily
Vendor risk sweep (assessment of vendors not yet assessed)Daily
AI inventory expiry sweepDaily
Training campaign creation for recurring plansOn each plan's schedule

Everything else is created when something happens: Generate for policies, organizational risks, and vendor risks; a vendor assessment when a vendor is created or updated; document gathering and AI inference when you work a questionnaire; privacy enrichment after a data-map ingestion; training invitations when a plan is activated; and a single certificate or privacy scan when you ask for one.

Edge cases and failure modes

How a failed sync is classified

The sync decides the data source's status from the error, and the status decides what happens next.

What went wrongData source statusNext attempt
The provider answered but returned nothing usableStays ConnectedNormal frequency
The provider rejected the credentials, the token has expired, or the app was uninstalledNeeds reauthenticationNone until you reconnect
A transient network errorUnchangedRetried automatically
Anything elseErrorRetried automatically, sooner than the normal frequency

An authentication failure also creates an in-app notification for the person who authorized the connection, linking to Data Sources. Some providers report the reason, which is shown under Needs reauthentication.

Other things to know

  • A sync can succeed with warnings. Post-sync linking is best-effort; check the sync history message if findings or data-map entries look incomplete.
  • An empty sync is a data problem, not an auth problem. The source stays Connected and the next sync runs on schedule.
  • A stuck job is recovered, not abandoned. A job that stops responding is failed and retried automatically.
  • One regeneration at a time. Only one Generate job per kind is allowed to be active per organization; a second request while one is running is refused.

What you can influence

  • Sync frequency per data source on the Data Sources page.
  • Reconnect a data source that Needs reauthentication; nothing else will restart it.
  • Sync now to run immediately with full notifications.
  • Generate actions (policies, organizational risks, vendor risks) create their own jobs.
  • Disconnect a data source to stop its syncs; existing evidence stays, with its last-seen time.

Do not rely on a retry after Needs reauthentication

Once a data source needs reauthentication, no schedule, sweep, or restart will touch it. Evidence from that source ages in place until someone reconnects.

Last updated on