Sync Batch Reconciliation¶

This page explains how Conflux detects and soft-deletes entity records that have been removed from the source platform, and why the system is designed the way it is. For the scheduler that triggers full syncs (and therefore reconciliation) automatically, see Scheduled Sync.

The Problem¶

A bulk sync run fetches all current data from the source platform and upserts it into Conflux. Records that existed in Conflux from a previous sync but are no longer present on the source platform would otherwise linger indefinitely — Conflux would show customers, products, or orders that no longer exist upstream. The reconciliation system solves this by identifying those stale records after each bulk sync completes and soft-deleting them.

SyncBatch Records¶

Every bulk sync run creates one SyncBatch row per capability (Products, Customers, Orders). It is created at the start of the run, before any fetch jobs are dispatched, and tracks the run's progress through atomic counter increments.

The sync_batches table carries these counters:

capability is cast to SystemCapability and source to SyncBatchSource. The status field moves through a defined lifecycle (see Status lifecycle below).

Stamping last_bulk_sync_batch_id on Entities¶

Product, Customer, and Order each have a nullable last_bulk_sync_batch_id foreign key pointing at sync_batches. This column is the mechanism by which reconciliation identifies which records were touched by a given run.

Every call to AbstractTransformer::store() writes the current batch's ID into last_bulk_sync_batch_id as part of the updateOrCreate payload. After the run completes, any entity for the same system_connection_id whose last_bulk_sync_batch_id does not match the completed batch was not seen during this sync — it is therefore stale.

Webhooks do not stamp this field. A webhook event arriving during a bulk sync would update last_bulk_sync_batch_id to null (or to a different batch) on that record, which would incorrectly flag a live record as stale and cause reconciliation to delete it. By leaving last_bulk_sync_batch_id untouched for webhook-originated writes, the field reliably tracks only bulk sync membership.

How Bus::batch() Ties It Together¶

BulkSyncJob uses Laravel's Bus::batch() to coordinate all the fetch-and-persist jobs for a single capability run:

1. A SyncBatch record is created with status = InProgress.
2. A set of FetchBulkPageRangeJob instances is dispatched as a Laravel batch, one job per page-range slice (up to 5 pages per job).
3. Each FetchBulkPageRangeJob fetches its page range from the integration and, for each chunk of data, adds a PersistBulkDataJob to the running batch via $batchable->add(). This means persist jobs are added dynamically as data arrives rather than pre-declared upfront.
4. Each PersistBulkDataJob runs the transformer and then atomically increments the batch counters (total_source_items, inserted_items, updated_items, not_touched_items, failed_items).
5. When all jobs in the batch have finished, the then() callback fires: it marks the SyncBatch as Completed and dispatches ReconcileSyncJob.
6. If any job fails, the catch() callback fires instead: the SyncBatch is marked Failed. ReconcileSyncJob is not dispatched — a partial sync must not delete records that were simply not reached.

How StoreOutcome Is Determined¶

AbstractTransformer::outcomeFor() is called after every updateOrCreate and returns one of three StoreOutcome values:

• Inserted — $model->wasRecentlyCreated is true: this is a new record.
• Updated — the model was not newly created, but $model->getChanges() contains at least one field outside SYNC_NOISE_FIELDS.
• NotTouched — the model was not newly created and only noise fields changed (or nothing changed at all).

SYNC_NOISE_FIELDS is ['updated_at', 'last_bulk_sync_batch_id']. Both fields change on every sync pass regardless of whether the underlying data actually changed: updated_at because Eloquent always touches it on save, and last_bulk_sync_batch_id because it is explicitly written on every store. Without filtering these out, every previously-seen record would be reported as Updated on every run.

Reconciliation: ReconcileSyncJob¶

ReconcileSyncJob receives the completed SyncBatch ID and performs the following steps:

1. Load the SyncBatch and determine the target model class from capability (Product, Customer, or Order).
2. Build a query for stale records: entities belonging to the same system_connection_id where last_bulk_sync_batch_id is either null or differs from the completed batch ID, and which are not already soft-deleted.
3. For Orders, collect the IDs of stale orders first and soft-delete their LineItem rows before soft-deleting the orders themselves. Line items do not carry last_bulk_sync_batch_id; their staleness is derived entirely from the parent order's staleness.
4. Soft-delete all stale entities via a bulk update(['deleted_at' => now()]).
5. Increment SyncBatch.deleted_items by the count of rows deleted.

Records with last_bulk_sync_batch_id = null are treated as stale because they predate the batch-tracking feature — they have never been confirmed present in any bulk sync.

Interaction with Delta Sync¶

ReconcileSyncJob is only dispatched after a full sync — runs where ExportRequest::$modifiedAfter is null. BulkSyncJob checks this in the then() callback and skips reconciliation for delta runs.

This is intentional: a delta sync only fetches records modified since the last cursor. Records absent from the response are not stale — they simply were not modified in the covered window. Reconciliation would incorrectly treat their unchanged last_bulk_sync_batch_id as evidence of staleness and soft-delete them.

See Delta Sync — Interaction with Reconciliation for more detail.

SyncBatchSource: Bulk vs Webhook¶

SyncBatchSource distinguishes how a batch was initiated:

• Bulk — triggered by a full export run (BulkSyncJob). These batches span all pages of data for a capability and are the only source that drives reconciliation.
• Webhook — reserved for webhook-originated batches. Webhook events process individual records rather than full data sets, so reconciliation is not meaningful for them and ReconcileSyncJob is never dispatched for a Webhook source batch.

Status Lifecycle¶

InProgress → Completed  (all jobs succeeded → ReconcileSyncJob dispatched for full syncs only)
           → Failed     (at least one job failed → no reconciliation)

A Failed batch leaves entity data in whatever partial state was reached. The next successful bulk sync will stamp last_bulk_sync_batch_id on the records it touches; a subsequent reconciliation will then clean up any records that were missed in both the failed run and the successful one.

completed_at is set only on the Completed transition; started_at is set at creation time.