Sync Batches¶

This page explains SyncBatch, the record every bulk sync run creates to track its own progress, and the Bus::batch() orchestration built on top of it. For how Conflux detects records deleted on the source platform, see Deletion Log Sync.

SyncBatch Records¶

Every bulk sync run creates one SyncBatch row per capability (Products, Customers, Orders). It is created at the start of the run, before any fetch jobs are dispatched, and tracks the run's progress through atomic counter increments.

The sync_batches table carries these counters:

capability is cast to SystemCapability and source to SyncBatchSource. The status field moves through a defined lifecycle (see Status lifecycle below).

Stamping last_bulk_sync_batch_id on Entities¶

Product, Customer, and Order each have a nullable last_bulk_sync_batch_id foreign key pointing at sync_batches. Every call to AbstractTransformer::store() writes the current batch's ID into last_bulk_sync_batch_id as one of the attributes it fill()s onto the record before save() — this is what lets StoreOutcome determination below filter out sync-driven no-op writes.

Webhooks do not stamp this field. A webhook event arriving during a bulk sync would update last_bulk_sync_batch_id to null (or to a different batch) on that record. By leaving last_bulk_sync_batch_id untouched for webhook-originated writes, the field reliably tracks only bulk sync membership.

How Bus::batch() Ties It Together¶

BulkSyncJob uses Laravel's Bus::batch() to coordinate all the fetch-and-persist jobs for a single capability run:

1. A SyncBatch record is created with status = InProgress. This still happens even when the probe finds nothing to fetch (totalPages === 0) — the batch exists purely so a deletion log fetch has something to attach its findings to, since records can be deleted on the source platform in a run that otherwise inserted or updated nothing.
2. A set of FetchBulkPageRangeJob instances is dispatched as a Laravel batch, one job per page-range slice (up to 5 pages per job).
3. Each FetchBulkPageRangeJob fetches its page range from the integration and, for each chunk of data, adds a PersistBulkDataJob to the running batch via $batchable->add(). This means persist jobs are added dynamically as data arrives rather than pre-declared upfront.
4. Each PersistBulkDataJob runs the transformer and then atomically increments the batch counters (total_source_items, inserted_items, updated_items, not_touched_items, failed_items).
5. When all jobs in the batch have finished, the then() callback fires: it dispatches FetchDeletionLogJob — see Deletion Log Sync — but leaves the SyncBatch InProgress. FetchDeletionLogJob itself marks the batch Completed once deletions have actually been processed, so a batch is never reported done while its deletion sweep is still pending or failed.
6. If any job fails, the catch() callback fires instead: the SyncBatch is marked Failed, and no deletion log fetch is dispatched.

How StoreOutcome Is Determined¶

AbstractTransformer::outcomeFor() is called after every save() and turns a change verdict the caller already computed into one of three StoreOutcome values — it does not inspect the model itself:

• Inserted — $model->wasRecentlyCreated is true: this is a new record.
• Updated — the caller determined the record changed: either its own fields did, checked via hasMeaningfulChanges() before the record was saved, or, for orders, a child row did — a line item — even though the record's own fields did not (outcomeIncludingChildren() covers that case). A customer's consent is not a child row in this sense: it is a record of the dependent customer_consents domain with its own change signal and its own outcome, which transform() does not count (ADR-0041).
• NotTouched — the model was not newly created and neither its own fields nor its children changed.

The counters count root records only. A consent's Inserted, Updated or NotTouched outcome is left out of inserted_items, updated_items and not_touched_items, and PersistBulkDataJob adds only root records to total_source_items. A dependent's failure is counted in failed_items, so it stays visible.

SYNC_NOISE_FIELDS is ['updated_at', 'last_bulk_sync_batch_id', 'source_created_at', 'source_updated_at']. updated_at and last_bulk_sync_batch_id change on every sync pass regardless of whether the underlying data changed: Eloquent always touches the first on save, and the second is explicitly written on every store. The two source timestamps behave differently. Both are written on every pass, but writing a column back the value it already holds leaves it out of getDirty(), so they only read as dirty when the source moved them: source_updated_at when a shop re-saved a record without touching anything Conflux mirrors, and source_created_at practically never, since a creation date does not normally change. Those are the passes the list exists to cover. hasMeaningfulChanges() filters all four out of $model->getDirty(), read before the record is saved, so a full sync that rewrites every row without changing it is not reported as Updated. All four are still written on every sync; they simply do not by themselves make a record deliverable. See ADR-0018 for why the source timestamps belong on this list.

SyncBatchSource: Bulk vs Webhook¶

SyncBatchSource distinguishes how a batch was initiated:

• Bulk — triggered by a full or delta export run (BulkSyncJob). These batches span all fetched pages of data for a capability (or zero pages, if there was nothing to fetch) and are the only source that drives a deletion log fetch.
• Webhook — reserved for webhook-originated batches. Webhook events process individual records rather than full data sets.

Status Lifecycle¶

InProgress → Completed  (bulk jobs succeeded, then FetchDeletionLogJob finished processing deletions)
           → Failed     (at least one bulk job failed)

A batch stays InProgress for the whole window between its bulk jobs finishing and FetchDeletionLogJob completing — including if that job fails and retries, in which case the batch simply stays InProgress until a run succeeds. This is deliberate: a batch is only Completed once its deletion sweep has actually happened, not merely dispatched.

A Failed batch leaves entity data in whatever partial state was reached. The next successful bulk sync will stamp last_bulk_sync_batch_id on the records it touches.

completed_at is set only on the Completed transition, by FetchDeletionLogJob; started_at is set at creation time.