Delta Sync¶

This page explains the design behind Conflux's delta sync mechanism (CFX-84) — why it exists, how the cursor is derived, and the trade-offs in the chosen approach.

For the concrete field specification, see ExportRequest.modifiedAfter. For how WooCommerce applies the cursor, see WooCommerce Integration — Delta sync. For the scheduler that triggers delta and full syncs automatically, see Scheduled Sync.

Why Delta Sync¶

Every full bulk sync fetches every record from the source platform, regardless of what changed since the last run. For a store with tens of thousands of customers and orders, this means large API payloads and proportionally large database write loads even when only a handful of records have been updated.

Delta sync avoids that cost: after the first successful full sync, subsequent runs ask the source platform for only the records that changed since the last successful run. For platforms that support server-side date filtering (such as WooCommerce), this reduces both the payload size and the number of API round-trips.

The Cursor: SyncBatch.started_at¶

The cursor is derived from the started_at column of the most recent SyncBatch record for a given system_connection_id + capability combination that has a status of Completed.

BulkSyncJob computes this before dispatching any fetch jobs:

$lastBatch = SyncBatch::where('system_connection_id', $connection->id)
    ->where('capability', $capability)
    ->where('status', SyncBatchStatus::Completed)
    ->latest('started_at')
    ->first();

$modifiedAfter = $lastBatch?->started_at?->toIso8601String();

If no completed batch exists (first sync, or all previous attempts failed), $modifiedAfter is null and the integration performs a full fetch.

Why started_at, Not completed_at¶

Using started_at as the cursor rather than completed_at is deliberate.

A sync run takes time. Records modified on the source platform during the run — after it started fetching but before it finished — fall into a window that completed_at would miss: those records were modified before completed_at but after the first API call was made, so they are not in the response the integration received.

Using started_at ensures those records are included in the next run. Their date_modified value is later than started_at of the previous batch, so they appear in the filtered response. The trade-off is that some already-imported records will be re-fetched on every run (any record modified between started_at of the last batch and the moment the first API call was made). This is acceptable: the transformer's updateOrCreate path handles duplicates gracefully, and the StoreOutcome accounting correctly classifies them as NotTouched rather than Updated if nothing meaningful changed.

Why No New Table¶

The cursor is an emergent property of existing data. SyncBatch already records started_at at job creation time and status throughout the run. Deriving the cursor from these fields costs nothing in terms of schema changes or additional writes.

An alternative would be a dedicated sync_cursors table with one row per (system_connection_id, capability) updated on each completion. This adds write complexity (an additional update on the then() callback path) and introduces a second source of truth that could diverge from SyncBatch. Since SyncBatch already contains all the information needed, the implicit derivation is strictly simpler.

Only Completed Batches Advance the Cursor¶

A failed sync does not advance the cursor. The query filters on status = Completed — a Failed or InProgress batch is invisible to cursor derivation.

This is safe by design. If a run fails midway, some records may have been fetched but others were not reached. Using the failed batch's started_at as the next cursor would skip those unreached records indefinitely. By requiring Completed, the next run re-covers the same window from the last known-good checkpoint.

Per-Capability Cursors¶

Each capability (Customers, Orders, Products) has an independent cursor. BulkSyncJob derives a separate modifiedAfter value for each capability before dispatching its fetch jobs.

This means a successful Customers sync advances only the Customers cursor. If the Orders sync failed in the same overall run, the Orders cursor stays at the previous successful point and the next Orders sync re-covers from there — without affecting the Customers window.

Integration Responsibility¶

The cursor is computed by BulkSyncJob and passed to the integration via ExportRequest::$modifiedAfter. Integrations never derive or persist the cursor themselves.

What an integration does with the value is its own concern:

• Server-side filtering — pass modifiedAfter as a native API query parameter. WooCommerce supports modified_after on its REST endpoints and this is the most efficient approach.
• In-memory filtering — fetch all records and discard those with updatedAt before the cursor. Equivalent in correctness; less efficient in payload size.
• Ignore it — always perform a full fetch. Functionally correct; loses the efficiency benefit.

The contract does not enforce which approach an integration uses. An integration that ignores modifiedAfter falls back to full-sync behaviour on every run.

Interaction with Reconciliation¶

Delta sync changes what data the integration returns, but reconciliation is never run after a delta sync.

Reconciliation (see Sync Batch Reconciliation) identifies stale records by comparing last_bulk_sync_batch_id on entities against the completed batch ID. A delta sync run stamps only the records it touched — records not returned by the filtered API call are not stamped. If reconciliation ran after a delta sync, it would soft-delete every untouched record as stale, which would be incorrect.

BulkSyncJob prevents this by only dispatching ReconcileSyncJob in the then() callback when $modifiedAfter === null (i.e. a full sync). Delta runs skip reconciliation entirely.