I am recording screen with mediaRecorder in 10 second chunks of video.
Then, with node.js, we put those videos together in a single webm.
The problem is that we can't see the duration of the video when we play it.
We tried the package webm-duration-fix but I can only apply it to the video fragments, and when joining them it is broken
I'm using Javascript on the front and node js on the back. In the back I get the video snippet and then apply fs.appendFileSync
app.post('/upload/', async (req, res) => {
let dir = './videos';
let testeador = req.body.testeador;
let cliente = req.body.cliente;
let id = req.body.id;
let fragmento = req.body.fragmento;
let idVideo = generateRandomCode()
res.send('Archivo subido')
fs.mkdirSync(`${dir}/${cliente}/${testeador}`, {
recursive: true
});
fs.appendFileSync(`${dir}/${cliente}/${testeador}/${testeador}.webm`, req.files.video.data, (err) => {
if (err) throw err;
});
})
Related
I have a mp4 video of 31sec with no background Audio. Now I want to add 2 background audios(mp3) on this video i.e. - [0-15s] 1st Audio then [15-31s] 2nd Audio using fluent-ffmpeg. I have been searching a lot for this scenario but not getting right solution. Any lead in this much much appreciated.
const merge = (videoFileName = 'video.mp4') => {
try {
const start1=0, endInSec1=15, start2=16, endInSec2=16;
const audio1 = 'https://d3ezyqzf8yqw4x.cloudfront.net/279499-audio-1639134010235.mp3';
const audio2 = 'https://d3ezyqzf8yqw4x.cloudfront.net/279499-audio-1639130519365.mp3';
ffmpeg()
.input(directory + '/input/' + videoFileName)
.input(audio1)
.setStartTime(start1)
.setDuration(endInSec1)
.input(audio2)
.setStartTime(start2)
.setDuration(endInSec2)
.on('end', () => {
console.log("Done bro")
})
.on('error', (err) => {
console.log(":::::: Getting error at mergeVideo() : ", err);
})
.saveToFile(directory + '/out/' + 'output.mp4')
} catch (err) {
console.log("Getting error :::::::::: ");
}
}
Then I'm calling this method below
(() => {
merge();
})()
When I execute the above code getting an output.mp4 file which has no audio and the length of video is [0-15]sec.
Could someone please help me with it?
I have a script that sends MediaRecorder data over Websockets to the backend when the MediaStream is running. The backend then saves this data to a file to be replayed.
I am running into an issue where the result is coming out like this
The video is extremely glitchy (sometimes it comes out somewhat smooth but mostly it comes out glitchy) and I do not know why.
I have tested this on Chrome (latest version) and Microsoft Edge and both produce the same glitchy results. I did notice though that Firefox seems to be smoother but i did limited tests on Firefox.
My front end code:
const socket = io('/')
document.querySelector("#startReccy").addEventListener('click', async () => {
socket.emit('recID', ROOM_ID)
let screenStream = await navigator.mediaDevices.getDisplayMedia({
video: {
cursor: "never"
},
audio: true
})
let vidId = document.querySelector("#myLiveVid")
vidId.srcObject = screenStream;
let videoMediaRecObj = new MediaRecorder(screenStream, {
audioBitsPerSecond: 128000,
videoBitsPerSecond: 2500000
});
//Send to server
videoMediaRecObj.addEventListener('dataavailable', (e) => {
//console.log(e.data)
//socket.emit('writeInfoData', ROOM_ID, e.data)
e.data.arrayBuffer().then ( buf => {
socket.emit('writeInfoData', ROOM_ID, buf)
})
})
videoMediaRecObj.start(3000)
//When we stop
screenStream.addEventListener('ended', () => {
videoMediaRecObj.stop();
//screenStream.getTracks().forEach(track => track.stop());
})
})
The backend
const express = require('express')
const app = express()
server = require('http').Server(app)
const io = require('socket.io')(server)
var fs = require('fs');
const { v4: uuidV4 } = require('uuid')
app.set('view engine', 'ejs')
app.use(express.static("public"))
app.get("/", (req, res) => {
res.redirect(`${uuidV4()}`)
})
app.get('/:room', (req, res) => {
res.render('room', {roomId: req.params.room})
})
io.on('connection', socket => {
socket.on('screen-rec', (roomId)=> {
/* Seems to get room ID with no problem*/
console.log(`${roomId}`)
/* We join a room*/
socket.join(roomId)
socket.roomId = path.join(__dirname, 'media', roomId + '.webm')
socket.emit('startRecording')
})
/* Write media file to disk*/
socket.on('writeInfoData', (roomId, data) => {
/* trying to read the data */
//console.log(data) - This works
fs.appendFile(`file-${roomId}.webm`, data, function (err) {
if (err) {
throw err;
}
console.log(`${Date().toLocaleString()} - ${roomId} - Saved!`);
});
})
})
server.listen(3000, () => {
console.log("Listening")
})
I am thinking the issue is because they may be received out of order, which I guess i can solve by making continuous POST requests but I know Websockets run on TCP so the packets have to come in order and I am sending it every 2 seconds so it should have time to send the data and save to the disk before the next request comes in (not sure though, just my guess)
Is there something wrong in my implementation?
Edit: Solved by O. Jones. I thought having a bigger timing is better but it seems that I need a really small time. Thank you so much!
Each time MediaRecorder calls your ondataavailable handler it gives you a Blob in event.data. That Blob contains all the accumulated encoded data. At a combined datarate of 2.6 megabits per second, three seconds of that is almost a megabyte. That's a very large payload for a POST request, and for appendFile.
Call ondataavailable less often. Use .start(20) instead of .start(3000). You're still moving a lot of data, but moving it in smaller chunks makes it less likely that things will get out of order, or that you'll lose some.
And, consider using nodejs streams to write your files.
I am trying to scrape three levels of a webpage that link to each other, e.g. Home -> Jobs -> Open Positions. I then want to write the scraped data into a output.json file. The scraping works just fine, but the writing of the file is finished before the requests are due to their asynchronous nature.
The code below, using normal requests scrapes all the data, but is too "late" and thus the info does not get written into the file.
request(url, function(error, response, html){
var $ = cheerio.load(html);
$("tr").each(function(i, elem){
var club_url = $(this).children().first().children().attr("href");
club_url = url.substring(0,25) + club_url;
request(club_url, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
var club_name = $("h1.masthead-title").first().text().trim();
console.log(club_name);
clubs[i] = club_name;
var teams = {};
$("tr").each(function(i,elem){
var team_url = $(this).children().first().children().attr("href");
team_url = url.substring(0,25) + team_url;
request(team_url, function(error,response,html){
if(!error){
var $ = cheerio.load(html);
var team = $(".team-name").text().trim();
console.log(team);
teams[i] = team;
}
});
});
}
});
});
fs.writeFile('output.json', JSON.stringify(clubs, null, 4), function(err){
console.log('File successfully written! - Check your project directory for the output.json file');
});
Therefore I tried to use request-promise and re-write the code with it, so the writing would executed after the request promises are resolved.
app.get('/scrape', function(req, res){
var clubs = {};
url = 'https://norcalpremier.com/clubs/';
var options = {
uri: 'https://norcalpremier.com/clubs/',
transform: function (body) {
return cheerio.load(body);
}
};
rp(options).then(($) => {
var ps = [];
$("tbody tr").each(function(i, elem){
var club_url = $(this).children().first().children().attr("href");
club_url = url.substring(0,25) + club_url;
console.log(club_url);
var club_options = {
uri: club_url,
transform: function (body) {
return cheerio.load(body);
}
};
ps.push(rp(club_options));
});
Promise.all(ps).then((results) =>{
results.forEach((club)=>{
var $ = cheerio.load(club);
var club_name = $("h1.masthead-title").first().text().trim();
console.log(club_name);
clubs[i] = club_name;
})
}).then(()=>{
fs.writeFile('output.json', JSON.stringify(clubs, null, 4), function(err){
console.log('File successfully written! - Check your project directory for the output.json file');
});
res.send('Scraping is done, check the output.json file!');
}).catch(err => console.log(err));
})
})
However, I just don't get it to work and get a bad gateway error, funnily after the console logs that the file was written. Some I assume neither the scraping is working now nor waiting for the requests to be finished.
Note: the third request is cut in this version, because I need to get the second level running first.
What I want to achieve is to get information from each of the sites on level 2 and 3, basically the name, put it into a JSON object and then write this into a file. As said previously, the scraping of the relevant data on level 2 and 3 worked in the former version, but not the writing to the file.
Thanks, your help is so much appreciated!
Here's what I would do, make the function async and then do:
url = 'https://norcalpremier.com/clubs/'
// put the request code in a function so we don't repeat it
let $ = await get(url)
// get the club urls
let club_urls = $('td:nth-child(1) a[href*="/club/"]').map((i, a) => new URL($(a).attr('href'), url).href).get()
// await the responses. I used slice because I think this much concurrency will cause problems
let resolved = await Promise.all(club_urls.slice(0,2).map(club_url => get(club_url)))
// get the club names
let club_names = resolved.map($ => $("h1.masthead-title").first().text().trim())
// write the file, I think synchronously is a good idea here.
fs.writeFileSync('output.json', JSON.stringify(club_names))
I'll let you figure out the get function since I don't like to use request-promise
I have Node App that collects vote submissions and stores them in Cassandra. The votes are stored as base64 encoded encrypted strings. The API has an endpoint called /export that should get all of these votes strings (possibly > 1 million), convert them to binary and append them one after the other in a votes.egd file. That file should then be zipped and sent to the client. My idea is to stream the rows from Cassandra, converting each vote string to binary and writing to a WriteStream.
I want to wrap this functionality in a Promise for easy use. I have the following:
streamVotesToFile(query, validVotesFileBasename) {
return new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(`${validVotesFileBasename}.egd`);
writeStream.on('error', (err) => {
logger.error(`Writestream ${validVotesFileBasename}.egd error`);
reject(err);
});
writeStream.on('drain', () => {
logger.info(`Writestream ${validVotesFileBasename}.egd error`);
})
db.client.stream(query)
.on('readable', function() {
let row = this.read();
while (row) {
const envelope = new Buffer(row.vote, 'base64');
if(!writeStream.write(envelope + '\n')) {
logger.error(`Couldn't write vote`);
}
row = this.read()
}
})
.on('end', () => { // No more rows from Cassandra
writeStream.end();
writeStream.on('finish', () => {
logger.info(`Stream done writing`);
resolve();
});
})
.on('error', (err) => { // err is a response error from Cassandra
reject(err);
});
});
}
When I run this it is appending all the votes to a file and downloading fine. But there are a bunch of problems/questions I have:
If I make a req to the /export endpoint and this function runs, while it's running all other requests to the app are extremely slow or just don't finish before the export request is done. I'm guessing because the event loop being hogged by all of these events from the Cassandra stream (thousands per second) ?
All the votes seem to write to the file fine yet I get false for almost every writeStream.write() call and see the corresponding logged message (see code) ?
I understand that I need to consider backpressure and the 'drain' event for the WritableStream so ideally I would use pipe() and pipe the votes to a file because that has built in backpressure support (right?) but since I need to process each row (convert to binary and possible add other data from other row fields in the future), how would I do that with pipe?
This the perfect use case for a TransformStream:
const myTransform = new Transform({
readableObjectMode: true,
transform(row, encoding, callback) {
// Transform the row into something else
const item = new Buffer(row['vote'], 'base64');
callback(null, item);
}
});
client.stream(query, params, { prepare: true })
.pipe(myTransform)
.pipe(fileStream);
See more information on how to implement a TransformStream in the Node.js API Docs.
If I query the box REST API and get back a readable stream, what is the best way to handle it? How do you send it to the browser?? (DISCLAIMER: I'm new to streams and buffers, so some of this code is pretty theoretical)
Can you pass the readStream in the response and let the browser handle it? Or do you have to stream the chunks into a buffer and then send the buffer??
export function getFileStream(req, res) {
const fileId = req.params.fileId;
console.log('fileId', fileId);
req.sdk.files.getReadStream(fileId, null, (err, stream) => {
if (err) {
console.log('error', err);
return res.status(500).send(err);
}
res.type('application/octet-stream');
console.log('stream', stream);
return res.status(200).send(stream);
});
}
Will ^^ work, or do you need to do something like:
export function downloadFile(req, res) {
const fileId = req.params.fileId;
console.log('fileId', fileId);
req.sdk.files.getReadStream(fileId, null, (err, stream) => {
if (err) {
console.log('error', err);
return res.status(500).send(err);
}
const buffers = [];
const document = new Buffer();
console.log('stream', stream);
stream.on('data', (chunk) => {
buffers.push(buffer);
})
.on('end', function(){
const finalBuffer = Buffer.concat(buffers);
return res.status(200).send(finalBuffer);
});
});
}
The first example would work if you changed you theoretical line to:
- return res.status(200).send(stream);
+ res.writeHead(200, {header: here})
+ stream.pipe(res);
That's the nicest thing about node stream. The other case would (in essence) work too, but it would accumulate lots of unnecessary memory.
If you'd like to check a working example, here's one I wrote based on scramjet, express and browserify:
https://github.com/MichalCz/scramjet/blob/master/samples/browser/browser.js
Where your streams go from the server to the browser. With minor mods it'll fit your problem.