Text in canvas being wiped out with a video canvas underneath - graphics

Question

Note: Minimal working example here. Click on change text button, and you can see the text will appear for one frame and disappear.
I wanted to add an overlay to my video, so I stacked two canvases and filled text to the transparent top canvas.
However, the text does not stick. It disappears.
To test if I was filling the text correctly, I tried using a black canvas (no video) underneath.
I just needed to fillText once and the text stayed.
However, with the video canvas underneath, the text won't stick unless I draw text in requestAnimationFrame, which I believe keeps drawing the same text in every frame, which is unnecessary.
The text canvas is supposed to be separate from the video canvas. Why is this getting wiped out without requestAnimationFrame?
How can I fix it?
Minimal working example here, but the code is also shown below.
import "./styles.css";
import { useEffect, useRef, useState } from "react";
export default function App() {
const inputStreamRef = useRef();
const videoRef = useRef();
const canvasRef = useRef();
const overlayCanvasRef = useRef();
const [text, setText] = useState("Hello");
function drawTextWithBackground() {
const ctx = overlayCanvasRef.current.getContext("2d");
ctx.clearRect(
0,
0,
overlayCanvasRef.current.width,
overlayCanvasRef.current.height
);
/// lets save current state as we make a lot of changes
ctx.save();
/// set font
const font = "50px monospace";
ctx.font = font;
/// draw text from top - makes life easier at the moment
ctx.textBaseline = "top";
/// color for background
ctx.fillStyle = "#FFFFFF";
/// get width of text
const textWidth = ctx.measureText(text).width;
// Just note that this way of "measuring" height is not accurate.
// You can measure height of a font by using a temporary div / span element and
// get the calculated style from that when font and text is set for it.
const textHeight = parseInt(font, 10);
const textXPosition = canvasRef.current.width / 2 - textWidth / 2;
const textYPosition = canvasRef.current.height - textHeight * 3;
ctx.save();
ctx.globalAlpha = 0.4;
/// draw background rect assuming height of font
ctx.fillRect(textXPosition, textYPosition, textWidth, textHeight);
ctx.restore(); // this applies globalAlpha to just this?
/// text color
ctx.fillStyle = "#000";
ctx.fillText(text, textXPosition, textYPosition);
/// restore original state
ctx.restore();
}
function updateCanvas() {
const ctx = canvasRef.current.getContext("2d");
ctx.drawImage(
videoRef.current,
0,
0,
videoRef.current.videoWidth,
videoRef.current.videoHeight
);
// QUESTION: Now the text won't stick unless I uncomment this below,
// which runs drawTextWithBackground in requestAnimationFrame.
// Previosuly, with just a black canvas underneath, I just needed to fillText once
// and the text stayed.
// The text canvas is supposed to be separate. Why is this getting wiped out without requestAnimationFrame?
// drawTextWithBackground();
requestAnimationFrame(updateCanvas);
}
const CAMERA_CONSTRAINTS = {
audio: true,
video: {
// the best (4k) resolution from camera
width: 4096,
height: 2160
}
};
const enableCamera = async () => {
inputStreamRef.current = await navigator.mediaDevices.getUserMedia(
CAMERA_CONSTRAINTS
);
videoRef.current.srcObject = inputStreamRef.current;
await videoRef.current.play();
// We need to set the canvas height/width to match the video element.
canvasRef.current.height = videoRef.current.videoHeight;
canvasRef.current.width = videoRef.current.videoWidth;
overlayCanvasRef.current.height = videoRef.current.videoHeight;
overlayCanvasRef.current.width = videoRef.current.videoWidth;
requestAnimationFrame(updateCanvas);
};
useEffect(() => {
enableCamera();
});
return (
<div className="App">
<video
ref={videoRef}
style={{ visibility: "hidden", position: "absolute" }}
/>
<div style={{ position: "relative", width: "90vw" }}>
<canvas
style={{ width: "100%", top: 0, left: 0, position: "static" }}
ref={canvasRef}
width="500"
height="400"
/>
<canvas
style={{
width: "100%",
top: 0,
left: 0,
position: "absolute"
}}
ref={overlayCanvasRef}
width="500"
height="400"
/>
</div>
<button
onClick={() => {
setText(text + "c");
drawTextWithBackground();
}}
>
change text
</button>
</div>
);
}

Answer 1

Very simple fix
The canvas is being cleared due to how React updates state.
The fix
The simplest fix is to use just the one canvas and draw the text onto the canvas used to display the video.
From the working example change the two functions updateCanvas and drawTextWithBackground as follows.
Note I have removed all the state saving in drawTextWithBackground as the canvas state is lost due to react anyway so why use the very expensive state stack functions.
function drawTextWithBackground(ctx) {
const can = ctx.canvas;
const textH = 50;
ctx.font = textH + "px monospace";
ctx.textBaseline = "top";
ctx.textAlign = "center";
ctx.fillStyle = "#FFFFFF66"; // alpha 0.4 as part of fillstyle
ctx.fillStyle = "#000";
const [W, X, Y] = [ctx.measureText(text).width, can.width, can.height - textH * 3];
ctx.fillRect(X - W / 2, Y, W, textH);
ctx.fillText(text, X, Y);
}
function updateCanvas() {
const vid = videoRef.current;
const ctx = canvasRef.current.getContext("2d");
ctx.drawImage(vid, 0, 0, vid.videoWidth, vid.videoHeight);
drawTextWithBackground(ctx);
requestAnimationFrame(updateCanvas);
}

Answer 2

when you update the canvas from the video, all of the pixels in the canvas are updated with pixels from the video. Think of it like a wall:
you painted "here's my text" on the wall.
Then you painted the whole wall blue. (your text is gone, and the wall is just blue).
If you paint the whole wall blue, and then repaint "here's my text", you'd still see your text on the blue wall.
When you redraw the canvas, you have to re-draw everything you want to appear on the canvas. If the video "covers" your text, you've got to redraw the text.
working example at https://github.com/dougsillars/chyronavideo which draws a chyron (the bar at the bottom of a news story) on each frame of the canvas.