Scalability
Scaling Real-Time Backends Under Concurrent Load
Practical scaling phases for real-time backends: when Redis GEO becomes mandatory, how PostGIS queries change, PostgreSQL connection strategy, and Celery isolation as async volume grows.
SystoBase Engineering · · 20 min
Key takeaways
- Scale in phases tied to measurable pain — not hypothetical microservice splits.
- At roughly 1k online drivers, PostgreSQL-only location often still works; at 10k+ concurrent pings, hot stores dominate architecture.
- Shard by metro or tenant before chasing exotic database replacements.
- Connection pooling and read replicas solve different problems — do not confuse them.
- Celery queue isolation becomes mandatory as notification and reconciliation fan-out grows.
- Load tests must simulate overlapping dispatch in dense cells, not uniform random geography.
Phase 0: prove the domain
Early real-time backends can run as a modular monolith: Django or similar, PostgreSQL with PostGIS, Redis for cache, Celery for async. The goal is correct domain state, payments, and matching before premature distribution. Mobility fleets make the concurrent-load curve vivid — but marketplace and logistics products hit the same phases.
Industry pattern: split into microservices before product-market fit. SystoBase recommendation: enforce module boundaries in code (dispatch package, payments package) with clear interfaces; extract services when team or deploy conflict demands it.
Measure early: p95 ride request latency, assignment success rate, payment reconciliation backlog, Celery queue age by queue name.
The location inflection point
Thousand-driver scale often means low thousands of location writes per minute — manageable in PostgreSQL with careful indexing if writes are batched. Ten thousand online drivers at 0.2 Hz is 2,000 writes/sec — a different regime.
Redis GEO becomes the industry-standard hot layer for online positions. PostgreSQL retains rides, users, money, and trip trails. The inflection is as much about write amplification as driver count — high-frequency tracking products hit it sooner.
Migrating hot location to Redis without dual-write bugs requires a cutover plan: shadow reads, compare radii results, then flip dispatch reads with rollback switches.
PostgreSQL strategy
Use PgBouncer or equivalent pooler in transaction mode for API workers. Raw connection-per-worker does not scale to hundreds of gunicorn/uvicorn processes.
Read replicas help reporting and heavy admin queries — not dispatch writes. Replication lag can confuse operators viewing dashboards; tag queries with roles.
Industry pattern: one giant database forever. SystoBase recommendation: partition large audit tables by month, archive cold rides, and keep hot ride rows narrow — avoid TOAST-heavy JSON blobs on assignment paths.
PostGIS at scale
GiST indexes on geography columns are essential for ST_DWithin nearby queries. Keep queries sargable — filter by city_id or bounding box before expensive spatial predicates when possible.
Analyze query plans as driver density grows in downtown cells. Sometimes precomputed hex bins or supply heatmaps in Redis outperform repeated wide-radius PostGIS scans for analytics-driven repositioning.
PostGIS shines for service area validation and offline spatial analytics; pair it with Redis GEO rather than replacing it.
Async worker isolation
Background job volume scales with rides × notifications × webhooks × reconciliation. A single default Celery queue becomes a hidden outage vector — ride timeouts arrive late, payment captures stall.
Industry pattern: scale workers uniformly. SystoBase recommendation: dedicated queues for ride_critical, payments, notifications, analytics with min concurrency guarantees on critical queues.
Redis broker memory and visibility timeout settings matter. Poison tasks need dead-letter handling and alerting — not infinite retry loops.
Caching and read models
Cache driver profiles, vehicle metadata, and pricing tables with explicit TTL and cache bust on admin updates. Do not cache assignment state without version keys.
Materialized views or read models for operator dashboards reduce OLTP load — refresh on schedule or via event triggers, accepting minutes of lag for ops UI.
Industry pattern: cache everything indefinitely. SystoBase recommendation: cache only derived read-heavy data with documented invalidation; never cache authoritative ride state outside the database transaction.
Load testing honestly
Uniform random driver locations across a country hide dispatch races. Simulate clustered demand — stadium exit, airport queue — with overlapping assignment attempts.
Include Celery in load tests: completion bursts enqueue capture, receipt email, loyalty, analytics. API-only load tests lie about end-to-end capacity.
Define success criteria before tests: max queue age, max assignment latency, zero duplicate assignments in scripted race scenarios.
Operational headroom
Plan capacity for 2× expected peak before marketing campaigns. Mobility spikes are spiky — Friday nights, weather events, concerts.
Autoscale API on CPU and latency; autoscale Celery on queue depth, not CPU alone. Redis memory alerts prevent silent GEO evictions.
Scaling drivers without scaling support tooling creates human bottlenecks — admin search, manual reconciliation, and override actions must stay fast at 100k drivers.