Latency, Throughput and the Numbers Every Engineer Should Know

Nine hundred wind turbines, each reporting how it's doing every two seconds, feed a telemetry service called Galewatch that exists only in this course. Engineers read its dashboards over breakfast to decide which machine to climb today.

One of its wind farms is Tarrow Ridge: sixty turbines along a ridge, and a site office in a portable building at the bottom of the access road.

The Tarrow Ridge dashboard loaded in 0.4 seconds on a developer's laptop in the city, and nine seconds in that office.

Same build. Same database. Same time of day. The site technician had filed the ticket three times, the third time with a stopwatch video attached, which is how you know a bug report has turned into a grievance.

So somebody read the slow query log twice, in case something obvious had been missed on the first pass. There was nothing to find. The heaviest query on the page ran in three milliseconds. Every measurement taken inside the server said the server was fast, and every one of those measurements was correct.

An index nobody needed got added anyway. A loop got hoisted out of a render. Forty milliseconds came off the server. Tarrow Ridge still took nine seconds.

The time was never inside the server. It was in the gaps between the pieces, and the gaps have sizes you can look up.

Lesson 001 handed you latency and throughput as two different words. Here are their numbers.

The two words, sharpened

Latency is how long one operation takes. One request, one disk read, one network hop. Measured in milliseconds.

Throughput is how many operations finish per second. Requests a second, rows a second, megabytes a second.

They feel like the same quantity, and they are not, and the cleanest way to feel the difference is a van.

A Galewatch customer once asked for three years of their own history: 8.4 terabytes. Sending it over the Tarrow Ridge site link, which does about 20 megabits a second, would have taken thirty nine days. So Galewatch copied it onto drives, put the drives in a van, and drove 500 km overnight. Ten hours door to door, which works out at roughly 230 megabytes a second, about ninety times the throughput of that link.

The van beats the internet.

The van's latency is ten hours. If what you needed was one reading, right now, the van is the worst system ever built.

Hold both ideas at once, because systems get designed badly when somebody optimises one number and reports the other. "We improved throughput by 40%" is perfectly compatible with "every user now waits twice as long." Lesson 001 showed you exactly that: as arrivals crept up towards the service rate, throughput sat near its maximum while latency went vertical. The hockey stick at eighty percent is a latency story hiding inside a throughput number.

The latency ladder

Here is the table. It's the closest thing this field has to a multiplication table, and the engineers who seem to reason quickly in design reviews are usually just people who have it memorised.

Units first, because the abbreviations blur together. One second is 1,000 milliseconds (ms), one millisecond is 1,000 microseconds (us), one microsecond is 1,000 nanoseconds (ns). Every step down is a thousand times smaller.

Every one of those is an approximation, and every one is within a small factor of what you'll measure, which is all the precision this kind of thinking needs.

Now the trick that makes them stick. Multiply everything by a billion, so that one nanosecond becomes one second, and the ladder turns into a human calendar.

Sit with that last row for a second. To a processor that fetches a value from cache in a subjective second, asking a server in Virginia a question is a six year expedition. Your code doesn't experience waiting. It experiences geological time.

Which is why "just call the other service" is never free, and why the first thing an experienced engineer counts in a proposed design is crossings of this ladder.

Call it the latency ladder. Each rung is roughly a thousand times worse than a rung near it, and what you want to know about any piece of work is which rung it stands on, and how often it goes there.

Memorise ratios, not absolutes

Hardware gets faster. The widely circulated 2012 version of that table said a megabyte from memory took 250 microseconds; today it's nearer 50. Memorise absolutes and you'll be quoting stale figures at a design review in three years, with total confidence.

The ratios have barely moved in twenty years, so learn those instead. RAM is about 1,000 times faster than a mainstream SSD for a random read, and a couple of hundred times faster than the quickest NVMe drives. An SSD is about 100 times faster than a spinning disk for a random read. A round trip inside a data centre costs about 5 SSD reads, and about 5,000 memory reads. A round trip across the world is about 400 times a round trip inside one data centre.

Those four sentences will carry you further than the table will. And when somebody tells you new hardware changes everything, ask which ratio moved. Usually the answer is none of them.

One ratio will never move at all.

The speed of light tax

Light in a fibre optic cable travels at about 200,000 km a second, roughly two thirds of its speed in vacuum, because glass slows it down. That gives you an exchange rate you can do in your head:

200,000 km/sec  ->  5 microseconds per kilometre, one way
                ->  1 millisecond per 100 km, round trip

Hyderabad to Mumbai is about 620 km in a straight line, so a round trip between them cannot beat about 6 ms. In practice you measure 20 to 25, because fibre doesn't run in straight lines and every router, switch and firewall on the path takes its cut. The rule that survives contact with reality: take the straight line floor, then expect to pay two to four times it.

The word doing the work there is floor.

You can buy a faster server, a better database, a cleverer cache. You cannot buy a shorter distance. No optimisation and no vendor moves this number, short of physically putting the data nearer the user, which is precisely why content delivery networks and multi-region deployments exist. Lessons 022 and 050.

Call it the speed of light tax. Every kilometre between two parts of your system is charged, per round trip, forever.

Now watch what happens when you pay it sixty one times.

Count round trips, not bytes

Back to Tarrow Ridge. The site office reaches the Galewatch region over a fixed wireless link: 45 ms round trip, about 20 megabits a second. Nothing exotic, just a building at the end of a long road.

The dashboard does what dashboards do. It fetches the farm's list of turbines, and then, for each turbine, fetches that turbine's latest reading. Tarrow Ridge has sixty turbines, and the framework issues those calls one after another, because each one is awaited inside a loop.

1 call for the turbine list        1 x 45 ms  =    45 ms
60 calls, one per turbine         60 x 45 ms  = 2,700 ms

Sixty small calls, each carrying about 400 bytes of JSON. The bytes are irrelevant: 400 bytes on a 20 Mbps link is 0.16 ms of transmission, so all sixty responses together spend about a hundredth of a second on the wire. Almost every other millisecond is a browser in a portable building waiting for light to travel to a data centre and back, sixty one times, one trip at a time.

The server was innocent throughout. Each call also used about 5 ms of server time, another 300 ms across the sixty, and 2,745 ms of waiting plus 300 ms of work is the three seconds in the budget below.

Here's the whole nine seconds, at last. Nothing in it overlaps: each piece waits for the one above it to finish.

For small payloads, latency is round trips. If you want that page faster, don't make the JSON smaller. Make sixty calls into one call. Batching all sixty latest readings into a single request turns three seconds into 45 ms plus one query's worth of server time, call it a tenth of a second all in. Thirty times better, for an afternoon of work, and no amount of query tuning would ever have found it.

For large payloads, latency is bandwidth. The 2.4 MB bundle doesn't care how many round trips there are. Transmission time is size divided by bandwidth: 2.4 megabytes is 19.2 megabits, and at 20 megabits a second that's about a second of pure transmission. On a link shared with the rest of the site, and with the connection ramping up rather than starting at full speed, you measure closer to two.

So which one dominates? That isn't a matter of taste, it's a size, and you can compute it in ten seconds for any link. One round trip's worth of bytes is bandwidth times round trip time.

payload size where bytes start to matter = bandwidth x round trip time

1 Gbps  x 0.5 ms  = about 62 KB   (inside a data centre)
20 Mbps x 45 ms   = about 110 KB  (Tarrow Ridge)

Under that size, shrinking the payload is very nearly pointless. Over it, shrinking the payload is the whole game. Somebody on your team will spend a week compressing 3 KB responses inside a data centre, and it will be a week.

Count round trips, not bytes. Trace one user action through a design, count how many times it crosses the network, and check whether those crossings happen in series or in parallel. Series is what kills you. Ten calls issued concurrently cost roughly one round trip. Ten calls in a loop cost ten.

The disk half of the story

That 3,500 ms panel deserves its own look, because it lives on a different rung of the ladder, and it brings lesson 001's arithmetic back.

The panel answers "how did this farm behave over the last day", and it queries the hot readings table: a rolling fourteen days, after which everything older moves to cheaper storage. Galewatch keeps one of those tables per farm, so this query has only Tarrow Ridge's rows to get through. Sixty turbines at one reading every two seconds is 30 rows a second.

30 rows/sec x 86,400 = 2,592,000 rows per day
  x 200 bytes         = about 518 MB per day
  x 14 days           = about 7.3 GB

Without an index, the database has exactly one option: read every row and throw away the ones that don't match. On an SSD that streams at 2 GB a second, reading 7.3 GB takes 3.6 seconds. There's the number from the budget table, and it was never mysterious. The query isn't misbehaving. It's doing 3.6 seconds of honest work.

Notice how much that per-farm split was quietly worth. Fourteen days of all nine hundred turbines in one table is 109 GB, and the same unindexed scan would take nearly a minute.

Now let a customer contract ask for a year of the whole fleet, which lesson 001 put at 2.8 TB.

2,800 GB / 2 GB per second = 1,400 seconds

Twenty three minutes, to return one turbine's readings for one shift. No hardware you can buy makes that acceptable, and knowing so saves you the ten minutes you'd otherwise spend hoping.

With an index on (turbine_id, recorded_at), the database walks a tree a handful of levels deep to find where that turbine's day begins, then reads forward through rows that are already sitting next to each other. A few random reads at 100 microseconds, then a short sequential scan of about 8 MB. Under ten milliseconds. Same rows, same disk, same server.

What changed is how many rungs of the ladder you touched, and how many times you touched them. Lesson 010 builds the index intuition properly, including what indexes cost you on every write.

Little's law: the one formula worth memorising

Now tie the two words together, with a formula so simple it looks like it can't possibly be useful.

concurrency = throughput x latency

L = lambda x W

That's Little's law. The average number of requests in flight inside your system equals how many arrive per second multiplied by how long each one stays. It holds for any stable system, with no assumptions at all about how the traffic is distributed, which is rare and rather wonderful.

Picture a room with one door. Ten people go in per minute, each stays five minutes, so there are fifty people in the room. Nothing deeper than that. Call it the busy room.

Galewatch's ingestion endpoint takes 450 readings a second, and each write takes 20 ms.

450 per second x 0.02 seconds = 9 writes in flight

Nine. That's how many workers or connections need to be busy at once, and it tells you that a pool of eight will quietly be the bottleneck while every dashboard reports a relaxed CPU.

Turn it around, which is where the law earns its keep. Postgres allows 100 connections, and each of these writes holds one for 20 ms.

100 / 0.02 seconds = 5,000 writes per second, maximum

A ceiling you now know without running a load test. Galewatch is at 450, so there's room. The four person online bookshop from lesson 001, Marlow Books, has the same 100 connection limit and four application processes holding pools of 25 each, so it has none.

There's a nastier reading of the same formula. If latency rises, concurrency rises with it at the same arrival rate. Let those writes slow from 20 ms to 200 ms, perhaps because the working set no longer fits in RAM, and the same 450 a second now needs 90 workers instead of 9, against a limit of 100. The pool empties, requests start queueing for a worker rather than for the database, and latency rises again.

That's the hockey stick from lesson 001, seen from the other side. Slowness eats concurrency, and running out of concurrency causes slowness. The two feed each other, which is why systems don't degrade gently.

Averages lie, and the tail decides

One last number habit, and it's the one that separates people who have run production systems from people who have read about them.

Say Galewatch's ingestion API responds in 30 ms on average. Healthy enough. But an average is one number standing in for millions of different experiences, and it hides the worst of them by construction.

Use percentiles instead. p50 is the median, the value half of requests come in under. p99 is the value 99 percent of requests come in under. A realistic profile for that API:

The average of that distribution lands near 30 ms and describes nobody: slower than the typical request, far faster than the painful ones. The slow tail has ordinary causes, not mystical ones. A cache miss. A garbage collection pause. A lock held slightly too long. A retry after a dropped packet on a link that runs beside a wind farm.

Now the part that surprises people. Go back to the Tarrow Ridge dashboard, and suppose you did the sensible thing and fired all sixty turbine calls in parallel. The page finishes when the slowest one finishes. If each call independently has a one percent chance of landing in the tail:

chance all 60 are fast      = 0.99^60 = 0.55
chance at least one is slow = 45%

Forty five percent of page loads hit a 400 ms call. Your backend's one in a hundred event is the page's coin flip. The tail becomes the typical the moment one user action fans out into many calls, and that is why large systems care about p99 with an intensity that looks obsessive from outside. Nobody is chasing the last millisecond for a hundredth of their users. They're chasing the experience of half of them.

Set your alerts and your targets on percentiles, never on averages. And when you count round trips in a design, remember that fanning ten calls out in parallel doesn't cost you one round trip. It costs you the worst of ten.

Recap

The latency ladder. Cache 1 ns, RAM 100 ns, SSD 100 us, one data centre round trip 0.5 ms, disk seek 10 ms, across the world 200 ms. Each rung is roughly a thousand times a nearby rung. If a nanosecond were a second, a call to Virginia would take six years.

Ratios outlive absolutes. RAM is about 1,000 times faster than an SSD read, an SSD about 100 times faster than a spinning disk, a global round trip about 400 times slower than a local one. Hardware gets faster; the ratios stay put.

The speed of light tax. About 1 ms of round trip per 100 km, two to four times that in practice. It's a floor that no engineering removes. Only moving the data closer helps.

Count round trips, not bytes. Below roughly bandwidth times round trip time, about 62 KB inside a data centre, latency is round trips and batching is the fix. Above it, bandwidth rules and size is the fix. Series crossings hurt; parallel ones mostly don't.

The busy room. Concurrency equals throughput times latency. It sizes your pools without a load test, and it explains why rising latency consumes concurrency and then causes more latency.

The tail becomes the typical. Measure p50, p95 and p99, never the average. When one action fans out into sixty calls, a one in a hundred slow call becomes a coin flip for the page.

Check your understanding

1. A page makes 12 API calls, one after another, from a phone whose round trip to your servers is 60 ms. Each call uses 4 ms of server time. Where does the page's time go, and what is the single highest value change you could make?

2. You're asked whether to compress a 3 KB JSON response served inside one data centre over a 10 Gbps link. Compute one round trip's worth of bytes, then say whether compression is worth it and why.

3. Galewatch's ingestion endpoint handles 400 writes a second at 80 ms each. How many are in flight? The worker pool is 24. What happens on the day the database slows from 80 ms to 250 ms?

4. A report scans a 900 GB table on an SSD that streams at 1.5 GB a second. What is the fastest that scan can possibly finish, and what would you say to a colleague who thinks the query "just needs more CPU"?

5. Marlow Books' API has a p50 of 15 ms and a p99 of 600 ms. Someone proposes a home page that loads 30 personalised widgets in parallel, one API call each. Roughly what fraction of loads will feel slow, and what would you propose instead?

Next lesson

003 What "Available" Actually Means: Nines, Downtime and Promises. Today you put numbers on how long things take; next lesson puts numbers on how often they work at all, turns nines into hours you can argue about, and finishes the availability arithmetic lesson 001 started.