Backend
Designing Reliable Domain State Machines
Explicit domain lifecycle states, transition guards in PostgreSQL transactions, audit events, and recovery when mobile clients and servers disagree — with ride lifecycle as a concrete example.
SystoBase Engineering · · 15 min
Key takeaways
- Hidden side states become production bugs — make every state explicit.
- Transitions belong in transactional boundaries with row-level guards.
- Industry pattern: update status column from anywhere; SystoBase recommendation: single domain service enforcing allowed transitions.
- Persist transition events, not just current status — ops needs history.
- Mobile optimistic UI must reconcile to server state machine truth.
- Cancellation variants (customer, provider, system, timeout) need distinct paths.
Core states
A production domain lifecycle needs explicit states — requested, searching/offered, accepted, in progress, completed, cancelled — plus payment substates. Skipping explicit substates hides bugs until ops invents spreadsheet workarounds. Marketplace orders, field jobs, and ride lifecycle all follow the same discipline.
Each state should map to customer and provider UX copy, push notification templates, and billing eligibility. Document the matrix before coding endpoints.
Industry pattern: string status field mutated by many controllers. SystoBase recommendation: enum with a centralized transition table and forbidden direct assignment linted in code review.
Transition guards
Guards encode who may transition and prerequisites: only the assigned provider marks arrived; only system or provider marks no-show after geofence and timer rules; a customer cannot jump to completed.
Race conditions love unconstrained updates — two tabs, retry POST, concurrent customer and provider actions. Guards use current_state checks in WHERE clauses so zero-row updates signal conflict.
Pair guards with idempotency keys on client actions so duplicate taps do not create illegal transition attempts that surface as 500 errors.
Transactional updates
Accept job: in one PostgreSQL transaction, verify the provider lease, update the job to accepted, mark the provider busy, emit an outbox event for notifications. Commit or rollback atomically.
Industry pattern: multi-step updates across requests. SystoBase recommendation: single transaction per transition with transactional outbox for downstream async — avoids “accepted in DB but provider still free in Redis”.
Isolation level READ COMMITTED is usually sufficient with careful row locks; Serializable for financial adjacency if you observe anomalies in testing.
The cancellation family
Cancellations are not one state — customer_cancelled_pre_accept, provider_declined, system_timeout, ops_cancelled, fraud_cancelled each imply different fees and availability restoration.
Fee rules attach to the transition, not the destination state alone. Refund authorization reversal may enqueue a payments-queue task after the cancel transition commits.
Failure mode: cancel succeeds in the app but payment authorization remains on hold — users see double pain. Tie payment side effects to the cancel transition outbox with a reconciliation job.
Audit and events
Append-only domain_events table: from_state, to_state, actor, reason, metadata JSON, created_at. Operators reconstruct disputes from this trail.
Industry pattern: overwrite status only. SystoBase recommendation: events as source of truth for history; current_status denormalized for query speed with invariant checks in CI.
Stream events to analytics and realtime channels — WebSocket payload carries state version for client ordering.
Client reconciliation
Mobile apps optimistically show the next state; the server may reject a transition on a stale version. Clients must poll or subscribe and roll back UI on conflict with clear messaging.
Include state_version or updated_at in API responses; require If-Match or a version field on transition POSTs.
Background app resume should always fetch the current snapshot — never assume local state after process death.
Recovery and stuck jobs
Stuck jobs: in_progress for hours, accepted never arrived, completed without payment. Scheduled reconciliation queries find invariant violations and enqueue repair or ops review tasks.
Automated repair for safe cases: release expired leases, move system_timeout from searching. Ambiguous money cases escalate — never auto-complete for billing.
Monitor count of jobs per state over time — a growing searching bucket often signals matching or dispatch outage.