I'm trying to stream video over a websocket and process it server-side with Node.JS. The client is reading from a video file (.mp4) and sending it over the web socket via a stream object. However, I'm having trouble extracting frames from the stream at the server so it can be processed by opencv.
Do I need to break the video up into frames and stream each individual frame? What format can opencv most easily process in real time?
The end goal here is to enable opencv to process each frame of a video (in real time) that is being received by the server. Think I'm having some trouble understanding the paradigm here.
Related
I was creating an application in react for uploading video and using a REST API to send that to the server and store in S3. I also wanted the simple audio version of the video for some other tasks and I am confused as to what might be the better way:
Creating audio file on the fly when it is needed using node-ffmpeg package and not store it anywhere
Start converting the video file to audio on the browser client only, and posting that to the server for storage along with the video.
Just post the video to the server and use queue system for creating a new task for video conversion to audio and then save that to the S3 storage.
The second method seems to be saving some compute power on the server but it might be a problem if the video upload completes, audio conversion is still going on and the client disconnects.
Would appreciate some help, thanks.
I have a node.js server that's receiving WEBM blob binary data small packets through socket.io from a Webpage!
(navigator.mediaDevices.getUserMedia -> stream -> mediaRecorder.ondataavailable -> DATA . I'm sending that DATA back to the server. So that includes timestamp and binary data).
How do I stream those back on a http request in a never ending live stream that can be consumed by a HTML webpage simply by adding the URL in the VIDEO tag?
Like this:
<video src=".../video" autoplay></video>
I want to create a live video stream that and basically stream back my Webcam to an html page but I'm a bit lost how do I do that. Please help. Thanks
Edit: I'm using express.js to serve the app.
I just am not sure what I need to do on the Server with the coming webm binary blobs to serve it properly to be consumed by an html page on an endpoint /video
Please help :)
After many failed attempts I was finally able to build what I was trying to:
Live video streaming through socket.io.
So what I was doing was:
Start getUserMedia to start the web camera
Start a mediaRecorder set to record intervals of 100 ms
On each available chunk emit an event through socket.io to the server with the blob converted to base64 string
Server sends back base64 converted 100ms video chunk back to all connected sockets.
Webpage gets the chunk and uses mediaSource and sourceBuffer to add the chunk to the buffer
Attach the media source to a video element and VOILA :) the video would play SMOOTHLY. As long as you attach each chunk in order and you don't skip chunks (in which case it stops playing)
And IT WORKED! BUT was unusable.. :(
The problem is the mediaRecorder process is CPU intensive and the page cpu usage was jumping to 15% and the whole process was TOO SLOW.
There was 2.5 seconds latency on the video stream passing through socket.io and virtually the same EVEN if DON'T send the blobs through socket.io but render them on the same page.
Sooo I found out this works but DOESN'T work for a sustainable video chat service. It's just not designed for it. For recording a webcam video to playback later, mediaRecorder can work but not for live streaming.
I guess for live streaming there's no way around WebRTC, you MUST use WebRTC to send the video stream to either a peer or a server to send to other peers. DO NOT TRY to build a live video chat service with mediaRecorder. You're only gonna waste your time. I did that for you :) so you don't have to. Just look into webRTC. You may have to use a TURN server. Twilio provide STUN, TURN servers but it costs money. BUT you can run your own TURN server with Coturn and other services but I'm yet to look into that.
Thanks. Hope that helps someone.
I am using the combination of fragmented mp4 and websockets to stream a live video stream to web browser from where MSE takes over.
I have successfully fragmented into the appropriate fmp4 format using ffmpeg and have checked the data using an mpeg4parser tool.
Utilising a websocket server, the incoming data is broadcasted to all the browser clients connected via websocket. This works fine for both playback and live streaming(using rtsp stream as the input).
The problem I am facing occurs when a client tries to access the stream midway, i.e, once the ffmpeg stream has started. I have saved the init segment(ftyp + moov) elements in a queue buffer in the websocket server. This queue buffer sends this data to each new client on connection.
I believe this data is sent correctly since the browser console does not throw the 'Media Source Element not found' error. Yet no video is streamed when it receives the broadcasted moof/mdat pairs.
So a couple of questions I would like the answer to are:
1) I have observer that each moof element contains a sequence number in it's mfhd child element. Does this have to start from 1 always, which will naturally not be the case for a video stream accessed midway?
2) Is it possible to view the data in the browser client.js. At present all I can view is that my mediaBuffer contains a bunch of [Object ArrayBuffer]. Can I print the binary data inside these buffers?
3) From the server side the data seems to be sent in moof/mdat fragments as each new data arriving from the ffmpeg output to the websocket server begins with a moof element. This was noticed by printing the binary data in console. Is there a similar way to view this data in client side.
4) Does anyone have an idea of why this is happening? Some fragmented mp4 or ISO BMFF format detail that I am missing.
If any further detail is required for clarification please let me know, I will provide it.
Make sure your fragments include a base media decode time. Then set the video tag 'currentTime' to the time of the first fragment received.
I wanted to do a HTTP live stream on a screen cast with using ffmpeg, nodejs and html5 . I wanted it to be as real time as possible. However, I find that my video received by the client was behind by 1~2 seconds (On Chrome/Chromium). I am using vp8/webm as my codec.
I have eliminated the following factors as such:
1) Network: I have tried serving and receiving the video file locally by stating the video source to be 127.0.0.1:PORT or localhost:PORT
2) ffmpeg encoding speed:I have tried outputting the file locally, it the "delay" seems to be negligible.
3) Chrome internal buffer. The buffer was accounted to be 0.07s~0.08s.
On the nodeJS side, I have a child process that runs the ffmpeg command, and did a ffmpeg.stdout.pipe(res); <-- ffmpeg is child_process.spawn(...)
So it seems that the ffmpeg.std.pipe(res) of nodejs seems to be the one delaying the video stream. Am I correct in assuming so ? Is there anyway that I may reduce the delay ?
Go to WebRTC no need to implement any thing like codec,pipe,etc(already in chrome,opera,firefox)
Uses:
MediaCaptureAPI(access your cam and mic and convert object to URL, default they are using vp8 codec,etc)
RTCPeerconnectionAPI(send and receive media stream p2p)
RTCDatachannelAPI(send and receive data using p2p)
I have a server (linux) program that generates audio files (mp3). What
I need is to broadcast these files using http stream. The tricky part
is that the broadcast starts when the file to be transmitted is not
fully generated.
I tried to do this using mpd+mpc but once I use the "mpc play" command
only already existing part of the file is buffered and transmitted,
and the player disregards the part that appears after beginning of
playback.
Is there any way to send a mp3 http stream (using mpd or any other
server-side player) so that the player won't stop the playback as it
reaches the end of the part that was buffered initially?
Any ideas, please.
http://streamripper.sourceforge.net/ can record and broadcast the same stream
shotcast(or icecast, dont remember) was designed especially for this, and could re-encode your stream on the fly