Payments
Designing Reliable Payment Services
Authorization holds, capture at completion, webhook reconciliation, idempotent Celery tasks, and keeping money aligned with domain state machines under partial failure.
SystoBase Engineering · · 17 min
Key takeaways
- Money and domain state are two sources of truth that temporarily diverge — design for reconciliation.
- Authorize early, capture on completion — with explicit cancel/release paths.
- Every capture and refund must be idempotent at provider and database layers.
- Industry pattern: synchronous capture in request thread; SystoBase recommendation: outbox + payments queue with retry and DLQ.
- Webhooks are delivery, not truth — verify signatures and reconcile.
- Support tooling needs a safe read-only payment timeline per order or job ID.
Two sources of truth
Your PostgreSQL domain row says completed; Stripe says authorized_not_captured; wallet ledger missing debit — all coexist during incidents. Assuming HTTP 200 means both sides finalized is how payment services diverge from product truth. Any product that settles after work finishes — marketplaces, logistics, subscriptions with usage adjustments, mobility — hits this split.
Industry pattern: treat the payment provider response as immediate truth. SystoBase recommendation: an internal payment_intent state machine mirroring provider states, with scheduled reconciliation closing gaps.
Model payment states: initiated, authorized, capture_pending, captured, failed, refunded, disputed — orthogonal to order or job status, linked via foreign keys and invariants. Clust’s trip settlement is one instance of the same pattern.
Authorize and capture
Authorization validates the instrument and reserves funds before work starts. Capture at completion for the final amount — including adjustments such as tips, fees, or usage deltas. Preauth may differ from final; handle incremental auth or re-auth per product rules.
Release authorization on cancel paths promptly — visible holds anger users and raise chargeback rates whether the product is a trip, a delivery, or a booked service.
Cash and wallet flows still need ledger idempotency — “cash” does not mean “skip audit trail”.
Coupling to domain state
Transition to a billable completed state enqueues capture — never capture before domain state allows billing. Guard the capture worker with status=completed and an amount_finalized flag.
Industry pattern: capture in the same request as complete. SystoBase recommendation: complete transition commits, then an outbox enqueues capture — completion succeeds even if the provider is slow; the customer sees the job done while payment retries.
If capture fails persistently, keep the job completed but payment_status=outstanding — block the next purchase or booking per policy with clear in-app messaging.
Idempotency layers
Provider idempotency keys on capture and refund API calls — typically order_id or job_id + operation + attempt generation.
Database unique constraints on (job_id, capture) prevent double ledger entries when workers retry.
Industry pattern: retry until the provider 500 clears. SystoBase recommendation: exponential backoff, max attempts, DLQ, pager on DLQ — never infinite hammering during a provider outage.
Webhooks and async confirmation
Verify webhook signatures, persist the raw payload, process idempotently by event id. Respond 200 quickly — heavy work belongs on a dedicated payments queue.
Reconcile webhooks against internal state continuously for high-value drift metrics — missing webhooks happen.
Industry pattern: trust webhook over API response. SystoBase recommendation: merge both into a reconciliation ledger — conflicts escalate to an ops queue.
Partial failure recovery
Capture succeeded, domain rollback attempted — catastrophic if rollback deletes a completed job. Prefer forward repair: mark paid, notify the customer, never silent delete.
Payment succeeded, domain stuck in_progress — detect via invariant jobs, complete administratively after verification or refund if the work was invalid.
Document playbooks: auto-retry safe cases, freeze for review on ambiguity, never auto double-refund.
Monitoring money drift
Metrics: count of completed jobs vs captured in the last hour, authorization age histogram, webhook lag, DLQ depth on the payments queue.
Alert thresholds tuned per volume — relative spikes matter more than absolute counts early on.
Finance dashboards join domain_events and payment_events on a shared ID — engineering and ops share one timeline language.