# Publish/subscribe: why 2 ms jobs waited 31 seconds behind one slow service

URL: https://systemdesignsimple.com/publish-subscribe-one-slow-service/
Published: 2026-11-10
Updated: 2026-11-06
Category: System Design
Series: System Design, simply, Part 61: Publisher/subscriber
Publisher: System Design Simple (https://systemdesignsimple.com)

## Summary

When a publisher calls every subscriber in turn, its time per event is the sum of all the calls, so one slow subscriber sets the pace for everyone. In our virtual-time simulation (1,000 orders, one every 5 ms, 10 subscribers, one of them 50 ms slower) fast jobs averaged 31,484 ms with direct calls and 1.87 ms with pub/sub. The slow one still built a 902-event queue.

## Key facts

- **Direct calls, 10 subscribers, one slow:** Publisher time per event 67.97 ms against an order every 5 ms; peak publisher backlog 926 orders not yet started
- **Fast jobs, order placed to done:** 31,483.88 ms average with direct calls; 1.87 ms with pub/sub (p99 62,339.77 ms vs 2.36 ms)
- **Pub/sub publisher:** Assumed publish cost 0.07 ms per event at 10 subscribers; publisher backlog 0
- **The slow subscriber:** Still about 51 ms per event in both modes; its queue peaked at 902 with pub/sub and it finished at 51,511.67 ms instead of 67,960.95 ms
- **Without the slow one:** Direct calls at 10 subscribers still took 18 ms per event and averaged 6,502.54 ms for fast jobs; pub/sub 1.87 ms
- **Setup:** Our simulation (labs/sd-61, seed 61): virtual clock, hand-picked costs (0.3 ms call round trip, 1 to 2 ms work, +50 ms for one subscriber). Shape of the effect, not broker benchmarks

If one service has to tell ten others that an order was placed, calling them one after another makes every order wait for the sum of all ten calls, so the slowest service sets the pace for everyone. Publish/subscribe lets each subscription process events independently, so the publisher can continue without waiting for subscriber work.

## Prerequisites {#prerequisites}

A worker that gets a job every 5 ms and needs 18 ms per job has an offered load of ρ = λE[S] = 18/5 = 3.6, or 360% of one worker's capacity. The worker stays busy and its backlog grows while arrivals continue. The same queueing logic sits behind [latency versus throughput](/ten-lanes-still-slow/).

## Definitions {#definitions}

> Definition: Publisher: the service that announces an event (here, "order placed") without knowing who will act on it.

> Definition: Topic: a named channel the publisher sends each event to once. The broker makes the event available to every matching subscription.

> Definition: Subscription: an independent delivery stream and backlog for a consuming service, which can share processing among multiple consumer instances. In Kafka, the equivalent position is tracked per partition within each consumer group.

> Definition: Backlog: in our lab, the largest number of events that were placed but not yet started by a given party.

## How publish/subscribe works {#how}

1. The publisher sends the event to a topic and gets an acknowledgement from the broker. It does not wait for any subscriber.
2. The broker makes the event available to each matching subscription; storage depends on the implementation. Google Cloud Pub/Sub delivers each message to every subscription attached to the topic ([overview](https://cloud.google.com/pubsub/docs/overview)). RabbitMQ routes it through a fanout exchange to every bound queue, and in the [tutorial](https://www.rabbitmq.com/tutorials/tutorial-three-python) a message sent before any queue is bound is discarded. Kafka keeps one log per partition, and each consumer group tracks its own offset in it ([docs](https://kafka.apache.org/41/getting-started/introduction/)).
3. Each subscriber pulls or receives from its own subscription at its own speed and acknowledges what it finished.

[Image: A cork notice board on the left with ten lanes fanning out to ten service dots; nine lanes stream green copies, the invoice lane is orange with a growing stack | One post on the board, ten lanes. Each service reads its own lane at its own speed.]

With direct calls the publisher pays for every subscriber on every event. With a topic it pays for one publish, and the broker does the fan-out. Our lab assumes that publish costs 0.05 ms plus 0.002 ms per subscription, so 0.052 ms at one subscriber and 0.07 ms at ten.

## Why direct calls pile up: the arithmetic {#math}

With direct calls in sequence, the publisher's service time per event is the sum over subscribers of round trip plus work:

```text
S_direct = sum over i of (rtt + work_i)
         = 10 x 0.3 ms + nine fast jobs (1 to 2 ms) + one slow job (about 51 ms)
         = 67.97 ms per event   (lab output, seed 61)

arrivals every 5 ms  ->  offered load ρ = 67.97 / 5 = 13.6
```

At 13.6 the publisher's queue grows by about 12.6 events for every one it finishes. A fast subscriber receives event *n* only after the publisher reaches it, so it inherits the slow one's delay. Even without the slow subscriber, ten fast calls took 18 ms per event, an offered load of 3.6, and fast jobs averaged 6,502.54 ms.

With pub/sub the publisher's service time is the publish cost alone, 0.07 ms against a 5 ms gap, an offered load of about 0.014, so it never builds a backlog. Each fast subscriber is busy about 30% of the time (1 to 2 ms of work every 5 ms), which keeps its wait close to its own work time.

## Worked example {#example}

[Image: A leaning tower of order tickets next to the number 926 and the words not started | Direct calls: orders that arrived but had not even started, at the peak.]

```text
1,000 orders, one every 5 ms -> the last one arrives at 4,995 ms
direct: publisher starts one order every 67.97 ms (average; seeded work varies per event)
        by 4,995 ms it has started floor(4,995 / 67.97) + 1 = 74 orders
        1,000 - 74 = 926 orders placed but not started (lab: 926)
        order n waits about n x (67.97 - 5) ms before its calls start
        average over n = 0..999: about 499.5 x 62.97 = 31,453 ms,
        plus its place in the call sequence (lab: 31,483.88 ms)
pub/sub: fast completion ≈ 0.07 ms publish + 0.3 ms hand-off + 1.5 ms mean work = 1.87 ms
        slow subscriber: first start at 0.37 ms, then about 51.5 ms per event,
        so by 4,995 ms it has started about (4,995 - 0.37) / 51.5 + 1 ≈ 98 events
        (98 in the seeded run); 1,000 - 98 = 902 waiting (lab: 902)
```

Assumed by hand: round trip 0.3 ms, work 1 to 2 ms, the +50 ms on subscriber six, the hand-off 0.3 ms and the publish cost. Derived by the simulation: every backlog, wait and finish time.

## Code {#code}

```sh
cd labs/sd-61
node run.mjs   # Node.js 22, no packages, no network, under a second on a virtual clock
```

Compare the two `10 yes` rows: `pub ms/event` 67.97 against 0.07, `fast avg ms` 31483.88 against 1.87, and the slow subscriber's backlog, 926 against 902. The slow subscriber finished 16,449.28 ms earlier with pub/sub, because its work overlapped with the other subscribers instead of waiting behind their calls.

[Image: Full orange frame with 1.87 ms in large type, was 31,484 ms under it, and simulation, one Mac | Same 1,000 orders: fast jobs averaged 1.87 ms with pub/sub.]

## Failure modes and edge cases {#failure}

- The slow consumer's backlog grows. Pub/sub moved the 50 ms problem into the invoice service's own queue. Watch each subscription's oldest unacknowledged message age and message count, and add consumers where messages can be processed independently (the next part, competing consumers); a single ordered key remains limited by its serial processing rate. Or make the work cheaper.
- Isolation depends on broker capacity: growing queues can exhaust shared resources and throttle publishing. RabbitMQ, for example, blocks publishers when its [memory alarm](https://www.rabbitmq.com/docs/memory) fires. Monitor broker resource alarms alongside subscription backlog.
- Check retention and overflow policies. Configured expiry ([RabbitMQ TTL](https://www.rabbitmq.com/docs/ttl), for example) or storage limits can remove unread events; alert before the applicable limit.
- Redelivery. With RabbitMQ manual acknowledgements, closing a channel or connection requeues its unacknowledged deliveries; another consumer may receive them. Durability and acknowledgement mode are separate settings ([RabbitMQ](https://www.rabbitmq.com/docs/confirms#automatic-requeueing)).
- Ordering. Publish related events in causal order and configure the broker's ordering guarantee. In Google Pub/Sub, use the same ordering key and region, and enable subscription ordering ([docs](https://docs.cloud.google.com/pubsub/docs/ordering)). Preserve that order when processing messages concurrently.
- Hidden coupling. If checkout needs the invoice service's answer before replying to the user, it is a request, and that call should stay direct.

## Trade-offs {#tradeoffs}

| Option | Good for | Costs you | Use when |
| --- | --- | --- | --- |
| Direct calls in sequence | Simple code, immediate errors | Publisher time is the sum of every call | One or two fast, critical callees |
| Direct calls in parallel | Lower latency than sequence | Still waits for the slowest; failure handling per callee | The caller truly needs every answer now (our lab did not test this) |
| Publish/subscribe | Many independent reactions to one event | A broker to run, redelivery, backlog and broker resources to monitor | Subscribers can act later and at their own speed |

When you split a [monolith into services](/monolith-vs-microservices/), this choice comes up early. Services that keep direct calls find each other through [service discovery](/service-discovery/).

## Common mistakes {#mistakes}

- Expecting pub/sub to speed up a slow consumer. Scale or fix that consumer separately.

- Sharing one subscription between two different services, which splits messages between them. Give each service its own.

- Running without an alert on oldest message age per subscription.

- Writing the dedupe record separately from the business change. For database effects, insert the event ID under a unique constraint in the same transaction as the business change, then acknowledge after commit. A separate dedupe write leaves a crash window.

## In an interview {#interview}

Ask whether the caller needs each service's answer before it replies. If not, publish once to a topic with one subscription per service; if it does, keep that call direct.

## Practice {#practice}

**Q (Beginner): What does a topic do that a queue does not?**
It makes each message available to every subscription; a queue gives each message to one consumer.

**Q (Intermediate): Direct calls take 18 ms per event and events arrive every 5 ms. What happens?**
Offered load is 3.6, so the publisher's backlog grows by about 2.6 events per event served while arrivals continue.

**Q (Intermediate): Why did fast jobs wait 31 s with direct calls?**
They queued behind earlier orders that each took 67.97 ms.

**Q (Advanced): Why did the slow subscriber finish earlier with pub/sub?**
Only its own work limited it.

**Q (Advanced): How can a slow subscriber still slow the publisher?**
Its backlog fills shared broker memory or disk, and the broker throttles publishing.

**Q (Advanced): When should you keep a direct call?**
When the caller cannot answer its own user without the callee's result.

## Cheat sheet {#cheat}

| Term or formula | Meaning |
| --- | --- |
| S_direct = sum (rtt + work_i) | Publisher time per event with sequential calls |
| offered load ρ = λE[S] = S / arrival gap | Above 1, the backlog grows while arrivals continue |
| Topic | Publish once; every matching subscription can read the event |
| Subscription | An independent delivery stream and backlog for a consuming service, which can share processing among multiple consumer instances |
| At-least-once | Duplicates possible; dedupe in the same transaction |
| Oldest message age | The per-subscription backlog alert |

## Further reading {#reading}

- Google Cloud Pub/Sub overview and ordering guide: subscriptions, ordering keys and their requirements.
- Apache Kafka documentation: logs, partitions and consumer groups with their own offsets.
- RabbitMQ Publish/Subscribe tutorial, plus the memory alarm and acknowledgement pages.

This article describes general messaging patterns and public documentation; it does not describe any company's private systems, and the diagrams are conceptual. Every number comes from labs/sd-61 (seed 61), a virtual-time simulation with hand-picked costs. It shows the shape of the effect, not broker benchmarks.

## Frequently asked questions

**Does publish/subscribe make a slow consumer faster?**
No. In our simulation the slow subscriber still needed about 51 ms per event in both modes. The publisher and the other subscribers stopped waiting for it, while its own queue grew to 902 events.

**What is the difference between a topic and a queue?**
A queue hands each message to one consumer. A topic makes each message available to every subscription, and each subscription has its own delivery stream and backlog, which several instances of one consuming service can share.

**Can one slow subscriber still affect the others?**
Yes, through the broker. Its growing queue uses shared memory and disk, and brokers such as RabbitMQ block publishers when a resource alarm fires. Watch broker resources as well as each backlog.

## Sources

- [Pub/Sub overview, Google Cloud documentation](https://cloud.google.com/pubsub/docs/overview)
- [Ordering messages, Google Cloud Pub/Sub documentation](https://docs.cloud.google.com/pubsub/docs/ordering)
- [Apache Kafka 4.1 documentation: introduction](https://kafka.apache.org/41/getting-started/introduction/)
- [RabbitMQ tutorial: Publish/Subscribe](https://www.rabbitmq.com/tutorials/tutorial-three-python)
- [RabbitMQ: memory threshold and limit](https://www.rabbitmq.com/docs/memory)
- [RabbitMQ: time-to-live and expiration](https://www.rabbitmq.com/docs/ttl)
- [RabbitMQ: consumer acknowledgements and publisher confirms](https://www.rabbitmq.com/docs/confirms)