Publish/subscribe: why 2 ms jobs waited 31 seconds behind one slow service

If one service has to tell ten others that an order was placed, calling them one after another makes every order wait for the sum of all ten calls, so the slowest service sets the pace for everyone. Publish/subscribe lets each subscription process events independently, so the publisher can continue without waiting for subscriber work.

Prerequisites

A worker that gets a job every 5 ms and needs 18 ms per job has an offered load of ρ = λE[S] = 18/5 = 3.6, or 360% of one worker's capacity. The worker stays busy and its backlog grows while arrivals continue. The same queueing logic sits behind latency versus throughput.

Definitions

How publish/subscribe works

1. The publisher sends the event to a topic and gets an acknowledgement from the broker. It does not wait for any subscriber.
2. The broker makes the event available to each matching subscription; storage depends on the implementation. Google Cloud Pub/Sub delivers each message to every subscription attached to the topic (overview). RabbitMQ routes it through a fanout exchange to every bound queue, and in the tutorial a message sent before any queue is bound is discarded. Kafka keeps one log per partition, and each consumer group tracks its own offset in it (docs).
3. Each subscriber pulls or receives from its own subscription at its own speed and acknowledges what it finished.

With direct calls the publisher pays for every subscriber on every event. With a topic it pays for one publish, and the broker does the fan-out. Our lab assumes that publish costs 0.05 ms plus 0.002 ms per subscription, so 0.052 ms at one subscriber and 0.07 ms at ten.

Why direct calls pile up: the arithmetic

With direct calls in sequence, the publisher's service time per event is the sum over subscribers of round trip plus work:

S_direct = sum over i of (rtt + work_i)
         = 10 x 0.3 ms + nine fast jobs (1 to 2 ms) + one slow job (about 51 ms)
         = 67.97 ms per event   (lab output, seed 61)

arrivals every 5 ms  ->  offered load ρ = 67.97 / 5 = 13.6

At 13.6 the publisher's queue grows by about 12.6 events for every one it finishes. A fast subscriber receives event n only after the publisher reaches it, so it inherits the slow one's delay. Even without the slow subscriber, ten fast calls took 18 ms per event, an offered load of 3.6, and fast jobs averaged 6,502.54 ms.

With pub/sub the publisher's service time is the publish cost alone, 0.07 ms against a 5 ms gap, an offered load of about 0.014, so it never builds a backlog. Each fast subscriber is busy about 30% of the time (1 to 2 ms of work every 5 ms), which keeps its wait close to its own work time.

Worked example

1,000 orders, one every 5 ms -> the last one arrives at 4,995 ms
direct: publisher starts one order every 67.97 ms (average; seeded work varies per event)
        by 4,995 ms it has started floor(4,995 / 67.97) + 1 = 74 orders
        1,000 - 74 = 926 orders placed but not started (lab: 926)
        order n waits about n x (67.97 - 5) ms before its calls start
        average over n = 0..999: about 499.5 x 62.97 = 31,453 ms,
        plus its place in the call sequence (lab: 31,483.88 ms)
pub/sub: fast completion ≈ 0.07 ms publish + 0.3 ms hand-off + 1.5 ms mean work = 1.87 ms
        slow subscriber: first start at 0.37 ms, then about 51.5 ms per event,
        so by 4,995 ms it has started about (4,995 - 0.37) / 51.5 + 1 ≈ 98 events
        (98 in the seeded run); 1,000 - 98 = 902 waiting (lab: 902)

Assumed by hand: round trip 0.3 ms, work 1 to 2 ms, the +50 ms on subscriber six, the hand-off 0.3 ms and the publish cost. Derived by the simulation: every backlog, wait and finish time.

Code

cd labs/sd-61
node run.mjs   # Node.js 22, no packages, no network, under a second on a virtual clock

Compare the two 10 yes rows: pub ms/event 67.97 against 0.07, fast avg ms 31483.88 against 1.87, and the slow subscriber's backlog, 926 against 902. The slow subscriber finished 16,449.28 ms earlier with pub/sub, because its work overlapped with the other subscribers instead of waiting behind their calls.

Failure modes and edge cases

• The slow consumer's backlog grows. Pub/sub moved the 50 ms problem into the invoice service's own queue. Watch each subscription's oldest unacknowledged message age and message count, and add consumers where messages can be processed independently (the next part, competing consumers); a single ordered key remains limited by its serial processing rate. Or make the work cheaper.
• Isolation depends on broker capacity: growing queues can exhaust shared resources and throttle publishing. RabbitMQ, for example, blocks publishers when its memory alarm fires. Monitor broker resource alarms alongside subscription backlog.
• Check retention and overflow policies. Configured expiry (RabbitMQ TTL, for example) or storage limits can remove unread events; alert before the applicable limit.
• Redelivery. With RabbitMQ manual acknowledgements, closing a channel or connection requeues its unacknowledged deliveries; another consumer may receive them. Durability and acknowledgement mode are separate settings (RabbitMQ).
• Ordering. Publish related events in causal order and configure the broker's ordering guarantee. In Google Pub/Sub, use the same ordering key and region, and enable subscription ordering (docs). Preserve that order when processing messages concurrently.
• Hidden coupling. If checkout needs the invoice service's answer before replying to the user, it is a request, and that call should stay direct.

Trade-offs

When you split a monolith into services, this choice comes up early. Services that keep direct calls find each other through service discovery.

Common mistakes

In an interview

Ask whether the caller needs each service's answer before it replies. If not, publish once to a topic with one subscription per service; if it does, keep that call direct.

Practice

Cheat sheet

Further reading

• Google Cloud Pub/Sub overview and ordering guide: subscriptions, ordering keys and their requirements.
• Apache Kafka documentation: logs, partitions and consumer groups with their own offsets.
• RabbitMQ Publish/Subscribe tutorial, plus the memory alarm and acknowledgement pages.

This article describes general messaging patterns and public documentation; it does not describe any company's private systems, and the diagrams are conceptual. Every number comes from labs/sd-61 (seed 61), a virtual-time simulation with hand-picked costs. It shows the shape of the effect, not broker benchmarks.