Why Is Your Canvas Video Overlay Dropping Frames?

I spent three days once trying to debug what I thought was a memory leak in a computer vision project. I was drawing bounding boxes over a 24fps video stream using a standard requestAnimationFrame loop, and the boxes kept "shivering." They weren't just off; they were vibrating back and forth like they were caffeinated. I assumed my math was wrong or my coordinates were drifting. Turns out, my math was perfect—my timing was just garbage.

The 60Hz Trap

Most of us were taught that requestAnimationFrame (rAF) is the gold standard for web performance. It syncs your code with the browser’s repaint cycle, usually hitting a sweet 60 frames per second.

But here’s the problem: Your video isn't running at 60fps.

Most cinematic video is 24fps. Webcams are often 30fps. High-end footage might be 60fps. When you use rAF to draw a video frame onto a canvas, you're essentially trying to mash two different gears together that have different tooth counts.

If you’re running a 60Hz display and watching a 24fps video:
1. rAF fires every 16.6ms.
2. The video frame only changes every 41.6ms.
3. You end up drawing the exact same video frame to the canvas two or three times in a row, then jumping to a new one.

This mismatch causes "judder." Your overlays feel disconnected from the underlying pixels because they are updating on a schedule that the video doesn't care about.

Enter requestVideoFrameCallback

A few years ago, the W3C (and specifically the Chrome team) realized this was a nightmare for anyone doing heavy video processing. They gave us requestVideoFrameCallback (rVFC).

Instead of saying "tell me when the screen repaints," rVFC says "tell me when the video engine has a brand new frame ready for me to work with."

It changes everything. Your code only runs when there is actually something new to draw. No wasted CPU cycles, no stuttering overlays.

The "Old" (Jittery) Way

const video = document.querySelector('video');
const canvas = document.querySelector('canvas');
const ctx = canvas.getContext('2d');

function render() {
  // This runs 60 times a second, even if the video 
  // hasn't moved a muscle.
  ctx.drawImage(video, 0, 0);
  
  // Logic for your overlay (the part that stutters)
  drawOverlay(); 
  
  requestAnimationFrame(render);
}

requestAnimationFrame(render);

The "New" (Smooth) Way

const video = document.querySelector('video');
const canvas = document.querySelector('canvas');
const ctx = canvas.getContext('2d');

function renderVideo(now, metadata) {
  // This ONLY runs when a new frame is actually 
  // pushed out by the video decoder.
  ctx.drawImage(video, 0, 0);
  
  // Your overlay is now perfectly in sync with the pixels
  drawOverlay();

  // Schedule the next frame
  video.requestVideoFrameCallback(renderVideo);
}

video.requestVideoFrameCallback(renderVideo);

Why the metadata object is a goldmine

One of the coolest parts about rVFC is the metadata object passed to the callback. It’s not just a "hey, I'm ready" signal; it gives you the dirty details of that specific frame.

function renderVideo(now, metadata) {
  console.log(metadata.presentationTime); // When the frame should be shown
  console.log(metadata.expectedDisplayTime); // When the browser will actually show it
  console.log(metadata.presentedFrames); // A count of how many frames we've shown
  console.log(metadata.width, metadata.height); // Great for handling dynamic resolution
  
  video.requestVideoFrameCallback(renderVideo);
}

This is massive for synchronization. If you're building a video editor or an AR filter, metadata.presentationTime is your source of truth. It allows you to map your external data (like tracking points) to the *exact* microsecond the frame was intended for.

Dealing with the real world (Compatibility)

As of today, requestVideoFrameCallback is supported in Chrome, Edge, and Safari. Firefox is the notable holdout (it's currently in development), so you still need a fallback.

Don't let the lack of 100% support stop you. You can write a simple wrapper that detects the feature and falls back to rAF if it’s missing.

function startRendering(video, callback) {
  if ('requestVideoFrameCallback' in HTMLVideoElement.prototype) {
    const internalCallback = (now, metadata) => {
      callback(now, metadata);
      video.requestVideoFrameCallback(internalCallback);
    };
    video.requestVideoFrameCallback(internalCallback);
  } else {
    // Fallback for Firefox
    const internalCallback = () => {
      callback(performance.now(), {}); 
      requestAnimationFrame(internalCallback);
    };
    requestAnimationFrame(internalCallback);
  }
}

The Verdict

If you are drawing to a canvas from a video source—whether it’s for green-screening, face-tracking, or just adding a custom play button—stop using requestAnimationFrame.

Using requestVideoFrameCallback reduces CPU usage because you aren't over-rendering. More importantly, it makes your app feel "pro." That weird, micro-stuttering ghost in the machine disappears, and your overlays finally look like they belong to the video rather than just floating precariously on top of it.