Postgres vs Redis Jobs: Queue Trade-Offs
A workload-first comparison of PostgreSQL row queues and Redis Streams, covering enqueue atomicity, claims, pending work, replay, duplicates, receipts, and operations.
Postgres vs Redis jobs is a choice about where work becomes durable, how workers claim it, and which evidence survives a crash—not a generic speed race. Both row queues and stream groups can deliver robust systems, and both can duplicate effects when their application contract is incomplete.
This comparison maps one job lifecycle onto both engines, runs an identical scheduling fixture, and evaluates atomicity, replay, contention, pending work, maintenance, and operational ownership. The result is a workload decision with failure drills attached.
Frame Postgres vs Redis jobs around the truth boundary
Postgres vs Redis jobs should be decided from what must survive, what must be replayed, and which system already owns the transaction that creates the work. If an order row and its fulfillment job must appear atomically, placing both in PostgreSQL can make the boundary simple. If the workload needs high fan-out stream consumption, explicit pending-entry recovery, and a team already operating Redis durability, Streams can be a strong fit.
Neither engine turns arbitrary handlers into exactly-once execution. Workers can crash after an external effect and before acknowledgment, so jobs need idempotency, attempt records, and completion receipts. This comparison uses the same duration fixture for both queues to separate scheduling mechanics from unproven claims about production throughput.
The comparison starts by drawing where durable intent becomes true. A PostgreSQL row can be committed in the same transaction as domain state; a separate queue requires an outbox or another reconciliation boundary, so the benchmark records orphaned and duplicated intents rather than reporting enqueue latency alone.
Runnable artifact: The workload fixture schedules identical job durations across three workers and accounts for every completion.
Save this proof as job-queue-workload.test.mjs and run node job-queue-workload.test.mjs. Expected final line: PASS: queue workload accounted.
import assert from "node:assert/strict";
const run=(jobs,workers)=>{const lanes=Array(workers).fill(0);for(const ms of jobs){const i=lanes.indexOf(Math.min(...lanes));lanes[i]+=ms}return {makespan:Math.max(...lanes),completed:jobs.length}};
const fixture=[8,3,5,2,13,1,8,4];const pg=run(fixture,3),redis=run(fixture,3);assert.deepEqual(pg,redis);assert.equal(pg.completed,fixture.length);assert.ok(pg.makespan<=18);
console.log("PASS: queue workload accounted");
- Declared input
- Inspectable transformation
- Measured output
Build Postgres vs Redis jobs from delivery semantics
Define enqueue, claim, lease or ownership, heartbeat, acknowledgment, retry, dead-letter, cancellation, and completion states. At-least-once delivery is common because a job can reappear after ambiguous worker failure. The handler therefore checks an idempotency or operation key before repeating a side effect.
A background job queue also needs a durable relationship between the job and its business result: sent email is weaker than a provider message ID plus accepted timestamp. Do not count a row deletion or stream acknowledgment as proof that the user-visible outcome occurred. Keep the queue state machine identical in the comparison, then map each transition onto engine primitives so differences remain architectural rather than vocabulary-driven.
Delivery language is kept precise: both designs may retry after ambiguous failure, and exactly-once external effects require an idempotency or deduplication boundary outside the claim primitive. The article’s own state machine labels queued, claimed, performing, completed, retryable, and terminal states so an acknowledgment cannot be confused with business completion.
Use Postgres vs Redis jobs with row locking
A PostgreSQL worker can select ready rows ordered by priority and age using FOR UPDATE SKIP LOCKED, mark attempts inside a transaction, and let concurrent workers skip rows already locked. This SKIP LOCKED queue pattern is compact and keeps enqueue transactions close to domain data, but it needs careful indexes, short claim transactions, bounded polling, vacuum health, and contention measurement. Do not hold a row lock while calling an external service.
Claim the job with a lease or state update, commit, perform the effect, then write a receipt. Fairness can degrade when expensive jobs or repeated failures keep changing ordering. Partition or separate queues when one hot class dominates index scans and autovacuum cannot keep pace with churn.
The row-queue implementation cites PostgreSQL’s locking clause documentation beside claims about SKIP LOCKED. Its stress test creates more workers than immediately available rows, crashes one claimant, and verifies lease or recovery policy returns abandoned work without allowing long transactions to conceal a stalled queue.
Use Postgres vs Redis jobs with consumer groups
The Redis stream data type assigns IDs to appended entries, and consumer groups track delivered-but-unacknowledged work in a pending entries list. Workers read new messages, acknowledge completed ones, and claim stale pending entries according to policy. Redis Streams make replay and multiple consumer groups natural, but retention, trimming, persistence mode, replication, failover, memory, and pending-entry recovery remain product decisions.
Acknowledging before a durable external receipt loses recovery evidence; acknowledging too late can repeat effects. Keep business state outside a transient payload or store enough immutable identifiers to reload it. When Redis and the source-of-truth database are separate, enqueue atomicity usually needs an outbox or another reconciliation mechanism instead of optimistic dual writes.
Consumer-group behavior is grounded in the XREADGROUP command reference. The drill leaves entries pending, transfers ownership after a worker loss, and acknowledges only after the completion receipt is durable; retention settings are tested against the oldest unfinished entry rather than tuned from aggregate stream size alone.
| Concern | PostgreSQL rows | Redis stream | Evidence |
|---|---|---|---|
| Domain atomicity | Same transaction | Outbox usually | Orphan scan |
| Claim | Row state + lock | Consumer group | Crash drill |
| Replay | Query retained rows | Range + pending | Recovery time |
| Operations | Vacuum + indexes | Memory + persistence | Capacity receipt |
Read Postgres vs Redis jobs from official primitives
The PostgreSQL SELECT documentation defines SKIP LOCKED and explicitly notes its inconsistent view is suitable for queue-like access rather than general-purpose reads. The Redis stream introduction explains entry IDs, ranges, and consumer groups, while XREADGROUP documentation details grouped consumption and pending messages. These sources specify primitives, not a complete queue product.
Version the database, client, durability configuration, and failover assumptions used in a benchmark. A locally durable Postgres instance and an ephemeral Redis container are not comparable, nor are Redis append latency and a PostgreSQL workflow that also commits domain state in one transaction.
Official database and stream documentation describes primitives, not a universal queue winner. The article therefore keeps product claims conditional on workload, durability configuration, and operator skill, and it publishes exact engine versions and settings beside every measurement so a reader can distinguish documented behavior from this team’s benchmark result.
- 1Enqueue
Commit immutable intent beside the domain change.
- 2Claim
Assign bounded ownership without holding long locks.
- 3Perform
Use an idempotency key around external effects.
- 4Receipt
Persist outcome before final acknowledgment.
Benchmark Postgres vs Redis jobs with one workload
Generate a fixture with arrival bursts, several duration classes, priorities, retries, poison jobs, cancellations, worker loss, and external-effect ambiguity. Measure enqueue latency, claim latency, time to start, time to completion, throughput, duplicate attempts, replay duration, starvation, storage growth, maintenance work, and recovery after process and node failure. The runnable scheduler proves only that identical durations are accounted across three abstract workers; it does not crown either engine. Continue implementation with a Postgres worker queue, evidence through background-job completion receipts, semantics via event delivery guarantees, and transactional publishing in the outbox pattern.
One generated workload is replayed against both candidates with identical arrival times, durations, priorities, retries, and injected failures. Results include start latency distributions, completion latency, duplicate attempts, lost-work count, recovery duration, storage growth, and maintenance time; a throughput chart is rejected if either system silently drops poison or canceled jobs.
Choose Postgres vs Redis jobs by operating cost
Prefer PostgreSQL when jobs are moderate in volume, enqueue must share a transaction with relational state, SQL observability is valuable, and the database has measured headroom. Prefer the Redis stream approach when fan-out, replay, consumer groups, or very high event rates are central and Redis is already operated with the durability the work requires. Split workloads when one choice does not fit all jobs, but keep cross-system tracing and a shared completion vocabulary.
My default for a new product with one existing PostgreSQL database is to prove the row queue first. That avoids a second critical datastore until measured contention, fan-out, or replay requirements justify it. The boundary is workload evidence, not a cultural preference for fewer services.
My default to begin with relational rows is explicitly a team-and-system heuristic, not a database law. The decision changes when measured contention, replay volume, fan-out, or isolation requirements exceed the current database’s operating envelope, and the receipt names those reopening thresholds before implementation loyalty can harden into architecture.
Ship Postgres vs Redis jobs with failure drills
The receipt includes workload classes, truth owner, enqueue transaction, payload references, state machine, claim primitive, lease and heartbeat, acknowledgment, idempotency boundary, retry and backoff, poison handling, cancellation, priority and fairness, receipt schema, storage durability, retention, indexes or memory policy, worker concurrency, benchmark versions, failure drills, duplicate results, recovery time, maintenance, dashboards, alerts, cost, and rollback. Fail release when dual writes can strand work, a crashed worker leaves a job invisible forever, acknowledgment precedes necessary evidence, retries repeat irreversible effects, a poison job starves healthy work, or retention deletes unfinished state. A queue choice is complete only when the team can reconstruct what happened to one job across a worker and datastore failure.
The final drill commits a domain change, kills a worker during the external effect, restarts the queue datastore, and reconstructs one job from intent through completion receipt. If operators cannot explain whether the effect happened and what will retry, the system fails regardless of benchmark speed because its ambiguity has been transferred to users and support.
Postgres vs Redis jobs is not a universal speed contest. Decide Postgres vs Redis jobs from the durability boundary, replay model, contention profile, and operational system already carrying the product's truth.