Our Experience
Lessons from Building a Production Real-Time Platform
Field notes from SystoBase engineering on production real-time platforms (Clust and similar): payment/state sync failures, Celery queue pressure, reconciliation culture, and what we would design differently — without invented scale metrics.
SystoBase Engineering · · 18 min
Key takeaways
- The hardest outages join payments, state machines, and async workers — not raw API QPS.
- Queue pressure showed up as “UX bugs” long before infrastructure alerts fired.
- Payment/ride divergence required reconciliation jobs and ops tooling, not more retries.
- Industry marketing cites driver counts we cannot verify — we document patterns, not vanity metrics.
- Redis GEO plus PostgreSQL split reduced dispatch DB load when hot location was lifted.
- Invest early in ride_id-traced logs — it pays off in the first finance escalation.
Context without vanity metrics
SystoBase has shipped production real-time platforms — multi-sided products with real payments, matching, and operator tooling. Clust is our deepest reference in that class. We do not publish partner traffic numbers we cannot defend; the lessons below are about failure modes and design choices we would repeat or change.
This article describes recurring engineering patterns and failure modes observed in that work — payment capture lagging completion, Celery backlog after peak hours, dispatch races in dense pickup zones — not a benchmark report.
If you are evaluating vendors, ask for traced incident examples and invariant dashboards, not slide-deck scale numbers.
Payment and ride sync
The most emotionally charged support tickets join money and trip outcome: rider charged, trip shows cancelled; trip completed, no receipt; authorization hold days after cancel. These are systems bugs, not edge cases.
Root causes we saw repeatedly: capture task on default Celery queue behind bulk jobs; complete_ride HTTP succeeding while capture retry exhausted silently; webhook delay during provider incident without internal payment_state update.
Fix pattern: payment_intent state column, idempotent capture tasks on isolated queue, invariant monitors for completed rides without captured payment within SLA, ops UI showing joined ride_events and payment_events timeline.
Industry pattern: blame payment provider publicly. SystoBase recommendation: own the reconciliation layer — providers fail; your product must degrade honestly.
Celery queue pressure
During peak marketplace usage, API latency stayed flat while riders waited minutes for driver assignment timeouts to release — timeout tasks lived on the same queue as nightly reconciliation and marketing pushes.
Symptoms were subtle: drivers “stuck” reserved, push notifications batched, payment captures clustered hours later causing finance spikes. Without per-queue age metrics, teams debated mobile bugs instead of worker starvation.
Remediation matched our queue isolation article: ride_critical queue with dedicated workers and alerts on oldest task age. Not glamorous — immediately reduced ghost reservations in testing and production.
Lesson: classify tasks before peak season, not during it. Redis broker memory limits needed tuning once queue separation increased key count — monitor evictions.
Dispatch concurrency surprises
Staging with sequential tests missed double-assignment races visible only when multiple riders requested in the same metro cell. PostgreSQL row locks on driver availability with version check fixed classes of bugs that idempotency keys alone did not.
Offer token validation on accept — matching current offer generation — eliminated stale accepts after timeout task lag.
PostGIS service area checks caught invalid pickups early; Redis GEO alone could not enforce polygon rules without PostgreSQL validation.
Location pipeline lessons
Moving online driver positions to Redis GEO reduced write pressure on PostgreSQL during high online-driver periods. Cutover required shadow comparison of radius results — small drift from TTL ghosts until offline cleanup hardened.
Median location age alerting caught a mobile release that stopped foreground updates — dispatch quality dropped before crash reports spiked.
Trip trail retention policy deferred until storage growth — then painful. Plan tiered retention early even if v1 stores minimal trail.
Operations and reconciliation culture
Support escalations improved when ops could search ride_id and see state transitions plus payment attempts without SQL access. Engineering time building read-only admin views paid back quickly.
Scheduled reconciliation jobs emitted tickets to a queue ops owned — not Slack panic. Clear ownership between auto-repair (release hold) vs manual review (ambiguous trip) reduced midnight pages for engineering.
Runbooks written after incidents — first query always “show me ride timeline” — became onboarding material for new backend engineers.
What we would repeat
Explicit ride state machine with event audit table before scaling marketing — saved dispute resolution.
Idempotency on all mutating mobile endpoints from day one — cheaper than retrofit during retry bug firefight.
Separating hot location from transactional PostgreSQL early when dispatch DB write volume trended up — even before theoretical capacity limits.
Transactional outbox for notify/capture side effects — kept HTTP responses honest when async failed.
What we would change earlier
Queue metrics and DLQ alerting before first major promotional push — would have caught timeout starvation sooner.
Payment/ride invariant dashboards in staging with synthetic failure injection — not only happy-path QA.
Stricter lint preventing direct ride.status assignment outside domain service — hidden shortcuts accumulated tech debt.
Documented freshness SLOs per location consumer — reduced arguments between dispatch and rider map teams.