Backend
Idempotency and Race Conditions in Concurrent APIs
Designing idempotent endpoints, database constraints, and compare-and-set patterns so retries, double taps, and concurrent assignment do not corrupt domain state.
SystoBase Engineering · · 16 min
Key takeaways
- Mobile networks guarantee retries — your API must guarantee safe replays.
- Idempotency-Key header plus server-side dedupe table is industry baseline.
- Database unique constraints catch races cheaper than distributed locks early on.
- Separate idempotency scope per endpoint and actor — keys are not global.
- Return same response body on replay — clients depend on stable outcomes.
- Load-test concurrent accept and cancel on same ride — races appear under overlap.
Why retries happen
Mobile clients timeout and retry. Intermediaries duplicate POSTs. Users double-tap accept. Background sync replays queued actions after reconnect. In concurrent APIs — especially matching, payments, and assignment — a non-idempotent write turns retries into corrupted state. Ride-sharing APIs make the races obvious; the patterns apply wherever two handlers can claim the same resource.
Industry pattern: document “do not retry POST” — unrealistic for mobile. SystoBase recommendation: design every mutating ride endpoint as idempotent within a time window with explicit client-supplied or server-generated keys.
Idempotency is not only payments — accept_ride, cancel_ride, complete_ride, rate_ride all need safe replay semantics.
Idempotency keys
Accept Idempotency-Key header (or body field) scoped to rider_id/driver_id plus endpoint. Store key hash, request fingerprint, response snapshot, expiry (24–72h typical).
On duplicate key with same fingerprint, return stored response with same HTTP status. On same key different body, return 409 — client bug or attack.
Industry pattern: Redis SETNX only. SystoBase recommendation: PostgreSQL idempotency_records table for durability across Redis restarts and auditable history.
Database constraints
Partial unique indexes enforce “one active ride per driver” and “one open assignment lease per driver”. Violations map to controlled 409 conflict, not 500.
UPDATE rides SET status=accepted WHERE id=? AND status=searching — check rowcount; zero rows means someone else won or state moved.
Compare-and-set on version column in same transaction as driver busy flag — atomic relative to concurrent accept attempts.
Dispatch race patterns
Two riders offered same driver: only one accept UPDATE succeeds; loser gets structured error and re-enters search pipeline without orphan accepted state.
Offer timeout task vs late accept: guard accept with offer_generation or token matching current offer — stale accepts rejected cleanly.
Industry pattern: last write wins on driver row. SystoBase recommendation: lease token validated in accept transaction; monitor rejected stale accepts as dispatch tuning signal.
Payment-adjacent APIs
complete_ride triggering capture must share idempotency with payment provider keys — store provider idempotency key derived from ride_id + capture_attempt.
Wallet debit/credit endpoints need ledger idempotency — unique (wallet_id, idempotency_key) on ledger entries.
Webhook handlers idempotent on provider event id — duplicate webhooks common during provider incidents.
Response replay semantics
Cached idempotent responses must include full body client needs — ride snapshot, error codes — not empty 204 on replay unless original was 204.
TTL expiry after window: either reject duplicate with 409 instructing client to fetch current state, or treat as new operation — document behavior.
Logging: log idempotency key and replay hit metric — spikes indicate client bug or network issues.
Testing races
Integration tests with parallel threads/processes hitting accept on same ride — assert exactly one success.
Chaos: kill API mid-transaction, verify no partial driver busy without accepted ride.
Property-based tests for state machine: random transition sequences never violate invariants when serialized vs parallel with locks.