Scalability
Scaling Background Jobs with Celery and Redis
Queue flooding, retry storms, worker prefetch pitfalls, and how to isolate latency-critical Celery tasks from bulk work on Redis brokers.
SystoBase Engineering · · 14 min
Key takeaways
- A growing queue with stable API latency still means user-visible failure in production.
- Industry pattern: one default queue; SystoBase recommendation: named queues by blast radius and SLO.
- Retries multiply load — cap retries and route poison messages to dead letters.
- prefetch_multiplier=1 for long tasks; tune acks_late for safety.
- Monitor queue depth, task age, and failure rate per task name.
- Beat schedules need isolation — cron storms can drown on-demand work.
Symptoms of queue pressure
API dashboards look green while customers wait minutes for receipts, providers stay reserved after missed offers, and payment captures lag completion. Celery backlog is a common root cause: the HTTP path succeeded, the async path did not. Mobility products surface this quickly; any product with settlement and fan-out shares the failure mode.
Operators report “random” instability — often correlated with marketing sends, nightly reports, or reconciliation crons sharing workers with timeout tasks.
Industry pattern: add workers until Redis CPU saturates. SystoBase recommendation: classify tasks, measure age per queue, scale critical queues first.
Typical production task mix
Latency-critical: offer timeout, status propagation, payment capture trigger, webhook processing. Seconds to tens of seconds.
Important but softer: push notifications, SMS, email receipts — users notice minutes of delay.
Bulk: analytics ETL, document batch validation, exports — should never starve latency-critical work.
Misclassification is common: a “simple” nightly script enqueued to default queue can block thousands of timeout tasks after a busy weekend.
Queue isolation
Define exchanges/routes in Celery so task names map to queues explicitly — no implicit default for new tasks in code review checklist.
Run worker pools subscribed only to specific queues in production. Critical pool always has minimum replicas, even if bulk pool scales to zero overnight.
Industry pattern: priority integer on shared queue. SystoBase recommendation: physical separation — Redis lists per queue — because starvation still happens with priority bugs and long tasks.
Redis broker operations
Monitor memory, evictions, and connected clients. Broker Redis should not share instance with GEO hot location unless carefully memory-capped — contention causes dual outages.
Visibility timeout and result backend TTL must align with longest task runtime; stuck unacked messages block capacity with acks_late enabled.
Use flower or Prometheus exporters for queue metrics — grep-ing Redis manually does not scale on call.
Retries and idempotency
Celery autoretry on any exception amplifies outages — a downstream DB blip becomes a retry storm. Retry only transient errors with exponential backoff and jitter.
Tasks must be idempotent: capture_payment(ride_id) twice must not double-charge. Use provider idempotency keys and database unique constraints on task dedupe keys.
Industry pattern: infinite retry until success. SystoBase recommendation: max retries + dead letter queue + alert on DLQ insert.
Worker tuning
Long I/O-bound tasks (PDF generation, provider API calls) suffer with high prefetch — one worker hoards messages. Set worker_prefetch_multiplier=1 for fairness on mixed-duration queues.
acks_late plus reject_on_worker_lost protects against killed workers mid-task — essential for payment tasks — at cost of possible duplicate execution; idempotency mandatory.
Autoscale celery workers on queue depth thresholds with cooldown — avoid flapping during brief spikes.
Observability
Dashboards: queue depth, oldest task age, task runtime p95 per name, failure rate, retry count, DLQ rate. Page when ride_critical age exceeds SLO.
Trace task enqueue from HTTP request via ride_id in task kwargs and structured logs — connects user complaint to stuck job.
Regular game days: deliberately pause critical workers and verify alerts fire before riders flood support.