3 Reasons Your Stream-Based Logic Is Still Causing Memory Spikes

I was staring at a memory usage graph last Tuesday that looked less like a healthy application and more like the North Face of the Eiger. Straight up, no plateau in sight. The culprit was a microservice designed specifically to *prevent* this—a data migrator built entirely on the WHATWG Streams API. We were "streaming" data from a legacy database into a new cloud bucket, yet the container was hitting its 2GB limit and restarting every twenty minutes.

It’s a common trap. We’ve been told for years that streams are the silver bullet for memory management. "Don't load the whole file into memory," they say. "Use streams to process data chunk by chunk."

But here’s the reality I’ve learned the hard way: a ReadableStream is not a magic vacuum that regulates data flow. It is often just a very fancy, very expensive wrapper around a memory leak if you don't respect the underlying mechanics. If your "streaming" logic is still causing OOM (Out of Memory) errors, you’re likely falling for one of these three architectural traps.

1. The "Push" Fallacy: You’re Enqueuing Without Asking

The most frequent mistake I see—and the one I made last week—is treating the start method of a ReadableStream like a main execution block.

When you create a new ReadableStream({ start(controller) { ... } }), the code inside start runs immediately. If you put a for loop or a while loop inside that block to fetch data and call controller.enqueue(chunk), you aren't actually streaming in the way the API intended. You are pushing.

// THE OOM TRAP: The "Firehose" Stream
const stream = new ReadableStream({
  async start(controller) {
    const dataSource = await getLargeDataset(); // Imagine 1 million rows
    
    for (const record of dataSource) {
      // We are enqueuing as fast as the CPU can loop.
      // We aren't checking if the consumer is actually ready.
      controller.enqueue(record); 
    }
    controller.close();
  }
});

The problem here is that controller.enqueue() doesn't wait for the data to be read. It simply pushes the data into the stream's internal queue. If your data source (the database) is faster than your data sink (the network request uploading the file), that internal queue grows indefinitely.

You’ve essentially just moved your "giant array in memory" into a "giant queue in a stream object." The memory footprint is the same, but now it's harder to debug because it's hidden inside a stream's internal state.

The Fix: Monitoring desiredSize

The ReadableStreamDefaultController has a property called desiredSize. This is the single most important signal in the entire Streams API. It tells you how much space is left in the queue before it reaches the "High Water Mark" (HWM).

- If desiredSize is positive, you have room to grow.
- If it’s zero, the queue is full.
- If it’s negative, you have overfilled the queue and should stop immediately.

To fix the firehose, you have to turn your logic into a state machine that respects that signal.

const stream = new ReadableStream({
  async start(controller) {
    this.iterator = (await getLargeDataset())[Symbol.asyncIterator]();
  },

  async pull(controller) {
    // The pull method is the engine. 
    // It's called when the internal queue has room.
    const { value, done } = await this.iterator.next();

    if (done) {
      controller.close();
    } else {
      controller.enqueue(value);
    }
    
    // If desiredSize is still high, the browser/Node will call pull() again.
    // If it's low, it waits. This is backpressure.
  }
});

By moving the logic to pull(), you give the stream control. The consumer dictates the pace, not the producer.

2. Ignoring Backpressure in the Middle of the Chain

Sometimes we aren't the ones defining the source; we’re just piping it. But piping isn't a "set it and forget it" operation.

If you’re using pipeThrough or pipeTo, the API handles backpressure for you—*usually*. But the moment you manually consume a stream or write a custom TransformStream, you become responsible for the pressure valve.

I recently saw a codebase where a developer was trying to log the progress of a stream. They did something like this:

// THE "SNEAKY LEAK"
readable
  .pipeThrough(new TransformStream({
    transform(chunk, controller) {
      // Do some heavy processing
      const transformed = heavyTransform(chunk);
      controller.enqueue(transformed);
    }
  }))
  .getReader()
  .read()
  .then(function process({ done, value }) {
    if (done) return;
    
    // Imagine some async tracking logic here
    db.logs.insert({ size: value.length }); 
    
    // Wait... we aren't awaiting the next read correctly
    // or we are triggering multiple reads without waiting 
    // for the previous one to finish.
    return reader.read().then(process);
  });

When you manually call reader.read(), you are pulling a trigger. If you don't wait for the processing of that chunk to finish before calling read() again, or if you have an un-awaited promise inside your transform function, you’re creating a backlog.

In a TransformStream, the transform method can return a Promise. Use it. If you return a promise from transform, the stream will not call transform again until that promise resolves. This is how you signal backpressure up the chain.

// THE CORRECT TRANSFORM
const slowTransform = new TransformStream({
  async transform(chunk, controller) {
    // By making this async, we tell the source to "slow down"
    // The source's 'pull' or 'desiredSize' will reflect this delay.
    const result = await someExternalApiCall(chunk);
    controller.enqueue(result);
  }
});

If you omit that await, the source will keep pumping data into the transform stream, thinking it’s ready for more, while the someExternalApiCall functions pile up in the microtask queue, holding onto all that memory-intensive chunk data.

3. The "DesiredSize" is just a suggestion (The Buffer Bloat)

Here is something that gets people every time: desiredSize can be negative, and controller.enqueue() will let you keep enqueuing anyway.

The Streams API doesn't actually *stop* you from breaking your own rules. The "High Water Mark" (HWM) is a soft limit. By default, for a ReadableStream, the HWM is usually 1 (meaning 1 chunk, not 1 byte). For a ByteLengthQueuingStrategy, it might be 16KB.

If you have a source that generates data in massive 50MB chunks, and your HWM is set to 1, your desiredSize will drop from 1 to -49,999,999 the moment you enqueue that first chunk.

If you don't check for that negative value, you’ll keep enqueuing.

// DANGEROUS: Ignoring the negative signal
async function produce(controller) {
  while (true) {
    const chunk = await getChunk();
    controller.enqueue(chunk);
    
    // If you don't check this, you're just using a buffer
    if (controller.desiredSize <= 0) {
      // We should stop and wait... but how?
      break; 
    }
  }
}

The "how" is usually where people give up and just let the memory spike. To do this correctly, you need to create a mechanism to "pause" your production. In many cases, this means utilizing the pull method I mentioned earlier, because the stream's internal controller calls pull only when desiredSize goes back above zero.

But what if you’re wrapping an event-based API, like a web socket? You can't just "pull" from a web socket; it pushes when it wants to.

In that case, you have to implement a real buffer and an acknowledgment system, or use the socket's own backpressure mechanisms (like pause() and resume() in Node.js net modules).

Practical Example: A Memory-Safe File Generator

Let's look at a real-world scenario. You want to generate a massive CSV file in the browser or Node.js without crashing the tab.

function createSafeCSVStream() {
  let rowCount = 0;
  const maxRows = 1000000;

  return new ReadableStream({
    // pull() is called automatically when the consumer wants data
    // and the internal queue is below the High Water Mark.
    async pull(controller) {
      // Generate a "batch" of rows to be efficient, 
      // but not so many that we ignore backpressure.
      const batchSize = 100; 
      
      for (let i = 0; i < batchSize && rowCount < maxRows; i++) {
        const row = `user_${rowCount},${Math.random()},${Date.now()}\n`;
        controller.enqueue(row);
        rowCount++;
      }

      if (rowCount >= maxRows) {
        controller.close();
      }
      
      // Once this function finishes, if desiredSize is still > 0,
      // the engine will call pull() again immediately.
      // If the consumer is slow (e.g., a slow disk write),
      // the engine will wait before calling pull() again.
    },
    
    // High Water Mark of 10 chunks.
    // Since we enqueue 100 lines at a time, 
    // we will have roughly 1000 lines in memory at most.
  }, { highWaterMark: 10 });
}

This structure is inherently safe. If the consumer (the part calling reader.read() or the WritableStream at the end of the pipe) slows down, pull() stops getting called. The rowCount stays where it is. The CPU idles. Memory stays flat.

Why the "Wrapper" approach fails so often

I think developers reach for the "push-style" start method because it feels like a standard async function. We like loops. We like async/await. The pull model feels "inverted" and weird.

But the "Wrapper" approach (wrapping a big fetch in a stream) fails because it creates a producer-consumer mismatch.

In a perfect world, your producer (the DB) and consumer (the S3 upload) move at the same speed. In the real world, the network fluctuates. The S3 bucket might throttle you. Your CPU might spike. If your stream logic doesn't have a way to tell the producer to "hang on a second," your memory is the only place that data has to go.

A quick word on pipeTo and pipeThrough

If you are piping:
source.pipeTo(dest)

Make sure dest is a proper WritableStream. If dest is a custom writable, you must ensure its write(chunk) method returns a Promise that only resolves when the data is truly handled (e.g., written to disk or acknowledged by a socket). If your write method resolves immediately while an internal buffer grows, you’ve just moved the memory leak to the *end* of the pipe.

The Checklist for Low-Memory Streams

Before you ship that streaming logic, run through this mental checklist:

1. Is my `start` method doing too much? If your start method contains a loop that enqueues data, you're probably ignoring backpressure. Move that logic to pull.
2. Am I checking `desiredSize`? If you are manually enqueuing in response to events (like socket.on('data')), you must check if desiredSize is negative and tell the source to pause.
3. Do my transforms return Promises? Ensure any TransformStream you write awaits any asynchronous work inside the transform hook.
4. What is my High Water Mark? If your chunks are large (MBs), a default HWM of 1 or 16 might be too much. If your chunks are tiny (strings), it might be too low, causing overhead.

Streaming isn't about the ReadableStream object itself; it’s about the *negotiation* between the source and the destination. If you aren't listening to the desiredSize signal, you aren't negotiating—you're just shouting data into a void, and your RAM is the only thing catching the echoes.

Next time you see a memory spike in a stream-based app, don't look at the data. Look at the signals. Usually, the stream was trying to tell you it was full—you just weren't listening.