I have a NodeJS app and want to let user upload image/video from a given URL using Axios. But before downloading the media and store it on S3, I need to verify the width & height through ffprobe, and size of the media (can use Content-Length).
My current implementation is to get first 5 chunks and pipe it to write stream, but the video seems to be broken...
router.post('/upload', async (req, res) => {
const { file } = req.body;
try {
const rawPath = path.resolve(`./.temp/raw/${Math.random()}`);
const response = await axios.get(file, { responseType: 'stream' });
// Progress
const stream = response.data;
const chunks: Buffer[] = [];
const writeStream = fs.createWriteStream(rawPath);
let progress = 0;
await new Promise((resolve, reject) => {
stream.on('error', reject).once('close', () => resolve(rawPath));
stream.on('data', (data: Buffer) => {
progress += Buffer.byteLength(data);
console.log(progress);
chunks.push(data);
if (chunks.length >= 5) {
stream.destroy();
resolve(rawPath);
}
});
});
chunks.forEach((chunk) => {
writeStream.write(chunk);
});
writeStream.end();
res.json('ok');
} catch (e) {
console.log(e);
res.sendStatus(400);
}
});
Describe what you tried:
Axios piping to write stream, but can't seem to get partial media
What you expected to happen:
Get some chunks of video to be checked
What actually resulted:
Broken media after downloaded
Related
I am stuck at problem on how to write excel file to filestream and download it in browser.. I can only create new file in server but this is not what I want.. I don't want to create in on server (ok if it must be created then i also want to delete it when user downloads it in browser).
But I can't achieve the download..
So the general idea is that I read the csv file, than parse the data.
I also read a template Excele file which I overwrite and write it to the file stream. When I call the get API, then I can the download starts (I will integrate it in Angular app later)..
I am using Exceljs npm package.
I don't have any errors but code is not working as I want
I uploaded whole code in github so you can easily see the code and duplicate my code.
https://github.com/zigax1/mean-generate-download-excel/tree/master
My excel-builder script:
export const generateExcel = async (req: Request, res: Response) => {
try {
await csvParse();
res.setHeader("Content-disposition", `attachment;`);
res.contentType(
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
);
return res.status(200).json("Success");
} catch (err) {
return res.status(500).json("False");
}
};
const csvParse = async () => {
fs.createReadStream("./content/TestCsv.csv")
.pipe(csv.parse())
.on("error", (error: any) => console.log("Error"))
.on("data", (row: any) => {
let line: any = String(row);
line = line.split(";");
//let parsedData = line[0];
let parsedData = line;
allParsedData.push(parsedData);
})
.on("end", (rowCount: any) => {
let test = allParsedData.toString();
generateFile(test);
});
};
const generateFile = (data: any) => {
return new Promise<fs.ReadStream>((resolve, reject) => {
const workbook = new Excel.Workbook();
workbook.xlsx.readFile("./utilities/template.xlsx").then(() => {
workbook.xlsx.writeFile("./content/Test.xlsx").then(
() => {
let stream = fs.createReadStream("./content/Test.xlsx");
stream.on("close", () => {
fs.unlink("./content/Test.xlsx", (error) => {
if (error) {
throw error;
}
});
});
resolve(stream);
},
(err) => {
throw err;
}
);
});
});
};
Thanks to everyone!
const csv = require('fast-csv');
const fs = require('fs');
function exportCSVFile(res, path, data) {
const ws = fs.createWriteStream(path + ".csv");
ws.on("finish", function () {
res.download(path + ".csv", () => {
fs.unlinkSync(path + ".csv");
});
});
csv.write(data, {headers: true}).pipe(ws);
}
You use this export csv function for your response
As the title says, how do you make a direct download link with a file from mongoDB(GridFsBucket) using express?
The file should be downloadable from memory, as i dont want to save it temporarily on the server.
I have this method:
async function downloadFileFromDB(fileId) {
var gridfsbucket = new mongoose.mongo.GridFSBucket(mongoose.connection.db, {
chunkSizeBytes: 1024,
bucketName: 'filesBucket'
});
try {
const stream = gridfsbucket.openDownloadStream(fileId)
const fileBuffer = Buffer.from(stream)
return fileBuffer
} catch (err) {
stream.on('error', () => {
console.log("Some error occurred in download:" + error);
})
console.log(err);
}
}
And this route:
router.get('/download-file', async (req,res) => {
const fileId = req.query.fileId
const ObjectFileId = new ObjectId(fileId)
const fileBuffer = await fileFacade.downloadFileFromDB(ObjectFileId)
res.download(fileBuffer)
})
But res.download wants a path and not a buffer. Aswell im not sure i can make a buffer directly from the openDownloadStream method.
Can anyone help?
I believe you need to write the data to your res object. I accomplished this like:
const readStream = gridfs.openDownloadStreamByName(filename);
readStream.on("data", (chunk) => {
res.write(chunk);
});
readStream.on("end", () => {
res.status(200).end();
mongoClient.close();
});
readStream.on("error", (err) => {
console.log(err);
res.status(500).send(err);
});
So, you may just have to do:
res.write(fileBuffer).end();
//// Instead of doing:
// res.download(fileBuffer);
I have a problem when upload images by taking direct from OPPO camera. I tried this from other smartphone is working as expected.
This is my code
const form = new FormData();
function processImages() {
return new Promise((resolve, reject) => {
images.forEach(async (image, index, array) => {
try {
const data = await getBufferFromStreams(image);
form.append('images', data.fileBuffer, data.filename);
if (index === array.length - 1) {
resolve(true);
}
} catch (error) {
reject(error);
}
});
});
}
const success = await processImages();
const response = await this.post(`${url}`, form, {});
export const getBufferFromStreams = async (file) => {
if (!file) {
return new Error('file is undefined');
}
const { stream, filename } = await file;
const chunks = [];
return new Promise((resolve, reject) => {
stream.on('data', (chunk) => {
chunks.push(chunk);
});
stream.on('error', (error) => {
reject(error);
});
stream.on('end', () => {
resolve({ fileBuffer: Buffer.concat(chunks), filename });
});
});
};
I always get error "Network Error: Failed to Fetch" when uploading images when taking picture from camera only on OPPO smartphone.
Do you have encounter same issues and solved ?
I'm trying to upload an image to Firebase-storage from Node.js,
I followed the follow that gives firebase in their doc and all run fine but when the image is in the storage the size is 0 bytes and you can not see the preview.
This is my code:
const uploadImageToStorage = (file,filename) => {
let prom = new Promise((resolve, reject) => {
if (!file) {
reject('No image file');
}
let newFileName = `${file.originalname}_${Date.now()}`;
let fileUpload = bucket.file(newFileName);
const blobStream = fileUpload.createWriteStream({
metadata: {
contentType: file.mimetype
}
});
blobStream.on('error', (error) => {
reject('Something is wrong! Unable to upload at the moment.');
});
blobStream.on('finish', () => {
// The public URL can be used to directly access the file via HTTP.
const url = `https://storage.googleapis.com/${bucket.name}/${fileUpload.name}`;
resolve(url);
});
blobStream.end(file.buffer);
});
return prom;
}
This is my app.post method:
app.post('/Upload_img',multer.single("file"), function (req, res) {
console.log("Upload Imagennes");
let url = "";
let file= req.file;
if (file) {
uploadImageToStorage(file,file.name).then((success) => {
url = success;
res.status(200).send(url);
}).catch((error) => {
console.error(error);
});
}
}
The Storage:
I managed to upload a png file using your uploadImageToStorage function. I was calling it directly, without using multer:
(async () => {
const buffer = fs.readFileSync('./android.png');
const file = {
originalname: 'android.png',
mimetype: 'image/png',
buffer,
}
try {
return await uploadImageToStorage(file, file.originalname);
} catch (err) {
console.log(err);
}
})();
This works as expected. So the problem is probably in your Express.js code or in multer. Try logging req.file and req.file.buffer in the server code, and see it has the expected data.
I am trying to make this simple api with the help of this article .The primary purpose of reading this article is to learn react native but it starts with a server in nodejs .I couldn't get it running correctly .
Here is the link to article link
Here is the code for server.js file
import express from 'express';
import http from 'http';
import giphyapi from 'giphy-api';
import base64 from 'base64-stream';
// Initialize http server
const app = express();
// Register /gif endpoint that returns base64 encoded gif
app.get('/gif', async (req, res) => {
res.json({
gif: await fetchGif(),
});
});
// Launch the server on port 3000
const server = app.listen(3000, () => {
const { address, port } = server.address();
console.log(`Listening at http://${address}:${port}`);
});
// Fetch random GIF url with Giphy API, download and Base64 encode it
export const fetchGif = async () => {
const item = await giphyapi().random('cat');
return await encode(await download(item.data.image_url));
};
// File download helper
const download = async (url) => {
return new Promise((resolve, reject) => {
let req = http.get(url.replace('https', 'http'));
req.on('response', res => {
resolve(res);
});
req.on('error', err => {
reject(err);
});
});
};
// Base64 encode helper
const encode = async (content) => {
let output = 'data:image/gif;base64,';
const stream = content.pipe(base64.encode());
return new Promise((resolve, reject) => {
stream.on('readable', () => {
let read = stream.read();
if (read) {
output += read.toString();
}
else {
resolve(output);
}
});
stream.on('error', (err) => {
reject(err);
});
});
};
the error is occurring because the third-party library "base64-stream" does not have the "encoded" function, it may exist in previous versions.
To solve this problem, you need to change some lines of your code so that your server looks like the code below.
const express = require('express'); // instead of "import express from 'express';"
const http = require('http'); // instead of "import http from 'http';"
const giphyapi = require('giphy-api'); // instead of "import http from 'http';"
const { Base64Encode } = require('base64-stream'); // instead of "import base64 from 'base64-stream';"
// Initialize http server
const app = express();
// Register /gif endpoint that returns base64 encoded gif
app.get('/gif', async (req, res) => {
try {
const gif = await fetchGif();
res.json({ gif });
} catch (error) {
res.status(500).send({ error });
}
});
// Base64 encode helper
const encode = (content) => {
let output = 'data:image/gif;base64,';
const stream = content.pipe(new Base64Encode()); // instead of "const stream = content.pipe(base64.encode());"
return new Promise((resolve, reject) => {
stream.on('readable', () => {
let read = stream.read();
if (read) {
output += read.toString();
}
else {
resolve(output);
}
});
stream.on('error', (err) => {
reject(err);
});
});
};
// Launch the server on port 3000
const server = app.listen(3000, () => {
const { address, port } = server.address();
console.log(`Listening at http://${address}:${port}`);
});
// Fetch random GIF url with Giphy API, download and Base64 encode it
const fetchGif = async () => {
try {
const item = await giphyapi().random('cat');
const image = await download(item.data.image_url);
return await encode(image);
} catch (error) {
console.log('fetchGif', error);
}
};
// File download helper
const download = (url) => {
return new Promise((resolve, reject) => {
let req = http.get(url.replace('https', 'http'));
req.on('response', res => {
resolve(res);
});
req.on('error', err => {
reject(err);
});
});
};
base64-stream doesn't have any function called encode(..).
There are two Classes Base64Encode, Base64Decode which are used to stream.