Keeping data consistent across different systems.

A service reconciling shipments, states and history between AMXpress and a legacy API through workers, idempotent persistence and failure recovery.

Technologies used: NestJS · TypeScript · PostgreSQL · Prisma · Redis · BullMQ · Axios

The problem

The external system and Core differ in identities, states and available fields. Incomplete lists, updates and partial failures can leave inconsistent data. The portal needs useful local reads without aggressively querying legacy or treating a failed synchronization as complete.

My contribution

I implement the contract between Core and the Sync worker, external-model normalization, record reconciliation, retries and deduplication. Regression and integration tests on isolated PostgreSQL and Redis exercise replay, recovery and local reads.

What the product enables

Request without blocking operations

Core enqueues synchronization by customer, range and date criterion in legacy-sync. It also requests individual tracking refreshes. An explicit search can await the job; normal navigation reads locally and refreshes in the background.

Reconcile data and preserve identity

External IDs, customers, packages and history are normalized. Repeated processing preserves IDs and avoids duplicating known events. A partial payload does not erase a confirmed date; rescheduling updates the trip day.

Report actual completion

Counters distinguish created, updated, skipped and failed records. Only success publishes dashboard_sync_completed and marks freshness. If an item fails, the job fails and can retry over records already persisted.

How the pieces fit together

Core produces BullMQ jobs over Redis; an independent worker consumes with concurrency 1 and queries the legacy system sequentially. Sync transforms responses and writes to the operational PostgreSQL database through Prisma. Redis events let Core refresh dashboard and tracking. Historical Pub/Sub channels coexist with the queue but do not offer the same persistence guarantees.

MVP component relationships

Core API
Customer + range + criterion · deduplication
Redis / BullMQ
Sync Worker
Concurrency 1 · lists + details
Legacy API
Sync / Prisma
Statuses · dates · history
PostgreSQL
Sync → Redis events
Completion / dashboard refresh
Core → Angular

Engineering decisions

Separate requests, processing and reads

Core queues work and an independent worker adapts legacy; ordinary navigation reads PostgreSQL. Concurrency and request pacing are bounded to protect the provider. BullMQ retains pending work; Pub/Sub events announce changes but do not replace the queue or guarantee replay.

Reconcile by identity instead of recreating

I normalize IDs, states, packages and history before persistence. Updates reuse local identities and avoid duplicating known movements; partial payloads do not erase confirmed information. The shipment date is one example: it keeps its operational meaning separate from technical timestamps.

Recover failures without false success

A range may persist records before failing: it is not a global transaction. Failed items prevent completion and freshness publication. Retries reconcile existing writes; backoff and error classification distinguish transient failures from unrecoverable input, without promising exactly-once.

Challenges addressed

Partial failures without false success

A range is not a global transaction: some records may persist before failure. Recovery repeats work and reconciles by identity and history. No range completion or freshness is published while failed items remain.

Respect legacy capacity

Concurrency, sockets and request pacing are bounded, with timeouts and pauses between days. The observed contract does not certify high-volume lists or rule out silent truncation; the solution documents that limit instead of promising absolute consistency.

Demonstrable capabilities

Verifiable reconciliation and recovery

Integration with simulated legacy exercises normalization, identity and history, partial payloads, retries and post-job reads. Dates and rescheduling are tested as part of data consistency.

Explicitly scoped guarantees

In-memory locks and Pub/Sub have limits. Persistence protects identity and confirmed data without promising exactly-once or provider completeness.