Transformer Pipeline Design¶
This page explains the reasoning behind the pluggable transformation and storage pipeline (CFX-46) — the design decisions that shape how normalised DTOs become persisted Conflux records and how failures are handled.
For the exact field-by-field specification, see the Transformer Contract reference.
Where the Transformer Fits¶
CFX-45 defined the outbound integration contract: the boundary at which Conflux speaks to external platforms and receives normalised DTOs. CFX-46 defines the next stage: taking those DTOs and writing them into Conflux's own unified data store.
The full pipeline is:
The integration knows the system. The transformer knows Conflux's storage layer. Neither knows the other — they communicate only through the DTO contract.
Per-System Isolation Through Extension¶
AbstractTransformer provides a concrete default store() implementation that covers the common case: map the DTO fields to the unified Eloquent model and upsert by (system_connection_id, source_id).
A system that needs different behaviour — custom field normalisation, a different dedup strategy, pre-processing steps — creates its own subclass and overrides store(). Overrides are localised to that class; the default implementation is unaffected. Systems that fit the default need no subclass at all until a real divergence arises.
This avoids premature duplication: there is no value in two identical classes for WooCommerce and PrestaShop when they share the same logic. A subclass is introduced when, and only when, it carries something distinct.
Unified Models¶
Customer, Order, and Product aim to capture all fields present in the corresponding DTO. Every field that a platform integration can provide has a dedicated column so that downstream consumers can query it directly without having to parse JSON.
The models are "unified" in the sense that a customer from WooCommerce and a customer from PrestaShop both land in the same customers table. The source_platform column carries the origin identity; the system_connection_id column carries the credential context.
Deduplication by (system_connection_id, source_id)¶
Every transformer uses updateOrCreate keyed on (system_connection_id, source_id). This combination is the dedup key: the same external record, arriving via the same system connection, is always matched to the same Conflux row.
system_connection_id is included because the same system record could arrive via two different system connections (two shops on the same system). Without system_connection_id in the key, a customer from shop A would incorrectly match a customer from shop B if their system-native IDs happened to collide.
source_id alone is not unique across systems or system connections. The unique constraint on (system_connection_id, source_id) enforces this at the database level.
Failure Capture Without Discard¶
AbstractTransformer.transform() is final. It iterates item by item and catches any Throwable per item. A storage failure on one record does not abort the rest of the batch.
This is important for bulk imports. A single malformed record — an invalid enum value, a constraint violation, an unexpected null — should not roll back the hundreds of records that succeeded before it. The TransformationResult carries the complete failure list so the caller can log, alert, or schedule retries for the specific items that failed.
The result is always returned; the method never throws.
Why order.status Is Not an Enum¶
Customer.source_platform is cast to a SystemDriver enum. Order.status is a plain string.
The reasoning follows the same logic as in the integration contract: order statuses are system-specific with no stable shared vocabulary. A WooCommerce on-hold does not map cleanly to a PrestaShop awaiting_payment. Defining a normalised enum would require a lossy mapping or an ever-growing union. The raw system status is preserved so that downstream consumers can apply their own interpretation.
external_data as an Escape Hatch¶
external_data holds customAttributes — the platform-specific key/value pairs that the integration placed there because they fall outside the normalised DTO schema.
All fields that are part of the DTO schema are stored in dedicated columns for queryability. external_data exists for genuinely platform-specific extras that have no shared meaning across platforms and therefore do not belong in the unified schema.
What This Does Not Cover¶
- Downstream push — the transformer stores data; it does not push it to external consumers (rapidmail, User Platform). That is a separate pipeline stage not yet implemented.
- Incremental sync — the transformer is designed for full-batch re-runs. Incremental logic (detecting deletes, applying timestamps) is deferred to a later ticket.