How Scale Giants Handle Millions of Requests

Imagine standing in line at a busy coffee shop during rush hour. There’s a constant flow of customers, some ordering complex drinks, some just a black coffee. The baristas behind the counter are like your servers handling requests. They need to process each order efficiently without letting the line explode.

Now, replace coffee orders with millions of API requests per second, and you’re starting to understand the scale companies like Uber, Netflix, and Amazon operate at. Handling high traffic isn’t about magic; it’s about smart architecture, efficient resource management, and building systems that fail gracefully.

None of the six patterns below is exotic, and you’ve probably read the names of all of them before. What usually gets left out is the second half of each one, which is the new problem it hands you the moment you adopt it. That second half is where the real work is, so it gets equal space here.

1. Event-Driven Architecture: Don’t Block the Line

Let's use Uber as an example. Each ride request is like a customer in that coffee line. The system doesn’t stop everything for one request. Instead, Node.js-style event-driven systems are perfect here:

// Pseudo Node.js request handler
server.on('request', async (req, res) => {
  const ride = await fetchRideData(req.body.rideId); // async, non-blocking
  res.send(ride);
});

Each request is non-blocking, meaning the server can start processing the next request immediately, even if one request is waiting on a database or an external API.

The thing worth understanding is why that works, because it changes what you do with it. Node runs your JavaScript on a single thread. Waiting on a database is cheap not because Node is fast, but because waiting costs nothing at all: the request is parked, the thread moves on, and the callback runs when the answer arrives. One barista can take a hundred orders as long as the machines do the brewing.

Which brings the catch. Non-blocking applies to input and output, not to work. The moment you do something CPU-heavy in a request handler, that single thread is yours alone, and every other request in the process waits behind it.

// This one line stops every other request on the server
app.post('/receipt', (req, res) => {
  const pdf = renderPdfSync(req.body.trip);   // 400ms of pure CPU
  res.send(pdf);
});

// Better: get the CPU work off the request path
app.post('/receipt', async (req, res) => {
  const jobId = await queue.publish('render_receipt', req.body.trip);
  res.status(202).send({ jobId });            // tell them where to collect it
});

Four hundred milliseconds of PDF rendering doesn’t slow down one request. It slows down every request currently on that server, including the health check. This is the most common way a Node app that behaved perfectly in testing falls over under real traffic, and it’s hard to spot because nothing errors. Everything just gets slower together, which looks like a database problem and isn’t one.

The fix is to get the CPU work off the request path: a worker thread, a queue, or a separate service. If you want the full picture of how the single thread schedules all this, I wrote about it in the event loop.

2. Microservices: Small Teams, Small Services, Big Impact

Big companies don’t run one giant monolith. They use microservices. Using Uber as our example, each service handles a specific task:

• Matching riders with drivers
• Calculating fares
• Handling notifications
• Trip history

Think of each microservice as a separate barista in our coffee shop analogy. Each one specializes, so they never bottleneck the rest of the shop.

[Ride Service] <---> [Fare Service] <---> [Notification Service]

By decoupling services, they can scale independently. If ride requests surge, only the ride-matching service needs more servers. That is the real prize, and it’s worth being precise about it: the win is independent scaling and independent deployment, not tidiness.

Here is the cost nobody mentions in the diagram. Every function call you turn into a network call becomes something that can fail, arrive twice, or hang. The first two you will plan for. The third is the one that takes production down.

// A call to another service is a call to something that can hang.
// Set the deadline yourself, and decide now what happens when it passes.
async function getFare(trip) {
  try {
    const res = await fetch(fareServiceUrl, {
      signal: AbortSignal.timeout(800),   // give up, out loud, after 800ms
    });
    return await res.json();
  } catch (err) {
    if (err.name === 'TimeoutError') return estimatedFare(trip);
    throw err;
  }
}

A service with no timeout on its dependencies is not independent. It’s coupled by latency, and when the fare service gets slow, the ride service piles up open connections waiting for it and dies too. One slow service quietly becomes an outage in four. Every call out of your process needs a deadline you chose on purpose and a decision about what to do when it passes.

So don’t start here. A monolith with clean internal boundaries will take you a long way, and it’s far easier to debug when something is wrong at 2am. Split a service out when a specific part genuinely needs to scale differently or belongs to a different team, not because the architecture diagram looks better with boxes.

3. Message Queues: Don’t Drop Orders

Even the best baristas can get overwhelmed if too many customers arrive at once. Similarly, Uber uses queues like Kafka or RabbitMQ to buffer requests:

// Example: pseudo-queue
queue.publish('ride_request', { userId, location });
queue.consume('ride_request', async (message) => {
  await matchDriver(message.userId, message.location);
});

Queues allow the system to process requests at a manageable pace, preventing server crashes and lost data. It helps to be blunt about what you’re actually buying: a queue does not make your system faster. It makes your system survivable. Work that arrives faster than you can handle it becomes a growing line instead of a crash, and a crash loses the orders while a line does not.

The catch here is delivery. Almost every queue worth using guarantees a message is delivered at least once, which is a polite way of saying some messages arrive twice. If a consumer processes a message and dies before acknowledging it, the queue does the responsible thing and hands it to someone else. Now you have charged the same rider twice.

queue.consume('charge_rider', async (message) => {
  // The same message can legitimately arrive twice.
  // Make the second delivery do nothing.
  const seen = await db.insertIfAbsent('processed_events', message.eventId);
  if (!seen) return;

  await chargeCard(message.riderId, message.amount);
});

That is idempotency, and it is not optional. Every consumer that changes something needs to be safe to run twice on the same message, usually by recording an event id and ignoring anything it has already seen.

Two more things to watch. A queue that only grows is a broken system with a calm-looking dashboard, so track queue depth as a first-class metric rather than an afterthought. And give yourself a dead letter queue, because one message that fails every single time will otherwise be retried forever and block everything behind it.

4. Horizontal Scaling: Add More Baristas

If your coffee shop gets too busy, you hire more baristas. In tech terms, this is horizontal scaling:

• Adding more servers
• Using load balancers
• Auto-scaling cloud infrastructure

[Load Balancer] --> [Server 1]
                 --> [Server 2]
                 --> [Server 3]

Horizontal scaling ensures your system can handle traffic spikes like morning rush hours or special events. It also has one hard prerequisite that catches people out: your application has to be stateless. If anything important lives in the memory of a single process, the second server breaks it.

// Works on one server. Breaks the moment you add a second one.
const sessions = {};
sessions[userId] = { cart, loggedInAt };

// Works on any number of servers, because no server owns the truth
await redis.set(`session:${userId}`, JSON.stringify({ cart, loggedInAt }));

The symptom is memorable. Users get logged out at random, carts empty themselves, and it only happens to some people some of the time, because it depends entirely on which server the load balancer picked. Sessions, caches and uploaded files all need to live somewhere every instance can reach, such as Redis, the database, or object storage.

Sticky sessions, where the balancer pins a user to one server, will make the symptom disappear. Be careful with that: it hides the problem rather than fixing it, and it comes back when that server restarts. While you’re there, give the balancer a real health check endpoint too, or it will keep confidently routing traffic to an instance that stopped working ten minutes ago.

5. Caching: Don’t Make Customers Wait

Not every request needs to go to the database. Uber caches frequently accessed data such as driver locations, surge pricing, or traffic info, so responses are fast and cheap:

async function getDriver(driverId) {
  const cached = await cache.get(driverId);
  if (cached) return cached;

  const driver = await db.fetchDriver(driverId);
  await cache.set(driverId, driver, 60);  // stale by up to 60s, and that is fine here
  return driver;
}

Caching is like remembering the regulars’ favorite coffee. No need to ask every time.

The useful question is not what you can cache but what you should, and there’s a decent rule for it: cache things that are read far more often than they change, and that nobody is harmed by seeing slightly out of date. A driver’s location a few seconds stale is fine, and it is read constantly. An account balance is not fine, and you should not cache it just because it is easy to.

Then there’s the failure mode that only shows up when you’re already busy. A popular key expires, and every request that wanted it misses at the same moment, so a thousand of them hit the database together. Your cache just aimed a spike at the exact system it was protecting. The usual defences are to serve the stale value while one request refreshes it in the background, or to let a single request rebuild the key while the others wait briefly.

One last thing that pays for itself: cache the computed answer, not the raw row. Caching a database record saves you one query. Caching the assembled response saves you the query, the joins, and all the work you did afterwards to shape it.

6. Observability: Watch the Shop

Finally, big companies monitor everything: latency, errors, queue sizes. Metrics and logging help spot bottlenecks before they become disasters. Tools like Prometheus, Grafana, and ELK stacks are part of the mix. Think of it as a manager constantly watching the coffee line, making sure no one waits too long, and adjusting staffing on the fly.

"Monitor everything" is not advice, though, so here is the shorter version of what to actually look at.

Measure percentiles, not averages. An average response time of 200ms can easily hide one percent of users waiting eight seconds, and at a million requests that one percent is ten thousand people having a bad day. Watch p95 and p99, because that is where your users live.

Watch saturation, not just errors. Queue depth, event loop lag, connection pool usage and memory all start moving well before anything actually fails. Errors tell you that you are already too late; saturation tells you that you have about twenty minutes.

And only alert on things you would get out of bed to fix. An alert nobody acts on trains you to ignore the one that mattered, which is a worse position than having no alerting at all.

Real Takeaways for Node.js Apps

It doesn’t matter whether you’re building an app for a mom-and-pop shop or the next global ride hailing app, you can learn from the approach big companies use. Use non-blocking I/O and keep CPU work off the request path. Break your app into services when a part of it genuinely needs to scale on its own. Buffer spikes with message queues, and make every consumer safe to run twice. Scale horizontally, which means keeping your app stateless. Cache what is read often and tolerates being stale. And monitor saturation, not just errors.

The order matters more than the list does. Most teams reach for microservices first, because it is the most visible change and the one that feels like real architecture. It is also the most expensive, and it is rarely the actual bottleneck. A more useful order is: measure first so you know what is slow, then cache, then get CPU work off the request path, then add a queue, then scale horizontally, and only then split services apart.

Uber’s architecture is the correct answer to Uber’s problem. Copying the answer before you have the problem mostly buys you Uber’s operational cost at your traffic, and that is a trade nobody enjoys making twice.