I've been trying different things all day, but nothing seems to offer a simple, straight-forward way to write a ReadableStream (which is an image) to a file. I'm calling an API which returns a ReadableStream, but what then? I tried digging into the object a bit more, and followed it all the way to returning a Buffer[], which seems like it should be what needs to go into a fs.writeFile() but nothing works. The file gets created but I try to open the image and it says it can't open that file type (which file type they're talking about, I have no idea).
Here is my code that returns a Buffer[]. I can also cut off some of those chains to only return the ReadableStream body, but then that returns a PullThrough and I am already so lost. Very little about that class online. Any suggestions?
Here is the api I'm using: https://learn.microsoft.com/en-us/javascript/api/#azure/cognitiveservices-computervision/computervisionclient?view=azure-node-latest#generatethumbnail-number--number--string--computervisionclientgeneratethumbnailoptionalparams--servicecallback-void--
// Image of a dog.
const dogURL = 'https://moderatorsampleimages.blob.core.windows.net/samples/sample16.png';
await computerVisionClient.generateThumbnail(100, 100, dogURL, { smartCropping: true } )
.then((thumbResponse) => {
console.log(thumbResponse.readableStreamBody.readableBuffer.head.data)
fs.writeFile("thumb.jpg", thumbResponse.readableStreamBody.readableBuffer.head.data, "binary", (err) => {
console.log('Thumbnail saved')
if (err) throw err
})
})
Finally found a solution. I don't understand pipe() all that well, but when it's called from a ReadableStream with a filepath as parameter, it works.
The API response thumbResponse.readableStreamBody was the ReadableStream. So anyone who has a readable stream can use this solution. No need to call an API for anything.
// Image of a dog.
const dogURL = 'https://moderatorsampleimages.blob.core.windows.net/samples/sample16.png';
await computerVisionClient.generateThumbnail(100, 100, dogURL, { smartCropping: true } )
.then((thumbResponse) => {
const destination = fs.createWriteStream("thumb.png")
thumbResponse.readableStreamBody.pipe(destination)
console.log('Thumbnail saved')
})
Related
I am trying to resize and upload multiple files using multer. It's working but the problem is when I am sending the data back it is sent before even the data is processed. So I am getting an empty list. I am new to nodejs tried solutions online but couldn't find the right one for me. Can anyone help me solving this problem? How can I push data into the list before sending the response?
the code is Attached below...
router.post('/ads/images', upload.array('images',5), async(req,res)=>{
console.log(req.files);
var data = []
await req.files.every(async(file)=>{
var imageBuffer = await sharp(file.buffer).png().resize({
width:250,
fit: sharp.fit.cover,
position: sharp.strategy.entropy
}).toBuffer()
var thumbnailBuffer = await sharp(file.buffer).png().resize({
width:150,
height:150,
fit: sharp.fit.cover,
position: sharp.strategy.entropy
}).toBuffer()
console.log({imageBuffer,thumbnailBuffer});
data.push({imageBuffer,thumbnailBuffer})
})
console.log(data);
res.send(data)
},(error,req,res,next)=>{
res.status(400).send({error:error.message})
})
The problem is in your every method. Use regular loops in async methods.
Please use a regular for loop instead and apply once only the sharp transforms.
for(let file of req.files) {
(your code here)
}
Using async/await with a forEach loop
Your await call is not doing anything because every is not asynchronous. Switch from every to map then use Promise.all(..) to wait for all promises returned by map to complete:
await Promise.all(req.files.map(async(file) => {
// ...
}))
I am currently creating a web that uses a variable that I can store in a JSON format. My plan is to modify the value of the JSON every time there's a connection to a certain route. The problem is it just won't write.
I have tried to use fs.writeFile and fs.writeFileSync but none of them seem to work.
// Code I Have tried
const kwitansi = require('./no_kwitansi.json')
app.get('', async (req, res) => {
kwitansi.no_kwitansi += await 1
await fs.writeFile('../no_kwitansi.json', JSON.stringify(kwitansi, null, 2), function (e) {
if (e) {
throw new Error
} else {
console.log('Wrote to file')
}
})
await console.log(kwitansi)
await res.send(kwitansi)
})
// An Example of my JSON File
{
"no_kwitansi":4
}
You are trying to write to a place where you do not have permission. Note that you opened ./no_kwitansi.json, but you are trying to write to ../no_kwitansi.json (one directory back). If you are sure you can replace the original file, remove the extra . in the write line.
If the error persists, you also need to be sure that you have the proper permissions to write the file. If you are using *nix or mac, you can check this link.
I'm creating an application where users upload a pdf and extracts the text into JSON format. I am able to access the text, but I can't hold the response until the PDF extraction is complete. I'm unfamiliar with Formidable and I may be missing something entirely.
I am using Formidable for uploading and PDFReader for text extraction. The front-end and back-end are on separate servers, and the app is only intended for local use, so that shouldn't be an issue. I'm able to console.log the text perfectly. I would like to work with the text in JSON format in some way. I would like to append the text to the response back to the front-end, but I can't seem to hold it until the response is sent.
const IncomingForm = require("formidable").IncomingForm;
const { PdfReader } = require('pdfreader');
const test = new PdfReader(this,1);
module.exports = function upload(req, res) {
let str = ''
let form = new IncomingForm();
form.parse(req, () => {
console.log('parse')
});
form.on("file", (field, file) => {
test.parseFileItems(file.path, (err, item) => {
if (err){
console.log(err)
}
else if (item){
if (item.text){
console.log(item.text)
str += item.text
}
}
})
});
form.on("end", () => {
console.log("reached end/str: ", str)
});
};
I've attempted a number of different ways of handling the async functions, primarily within form.on('file'). The following attempts at form.on('file') produce the same effect (the text is console.logged correctly but only after form.on('end") is hit:
//Making the callback to form.on('file') async then traditional await
form.on("file", async (field, file) => {
//...
await test.parseFileItems(...)
//...
console.log(str) //After end of PDFReader code, shows blank
//Making cb async, then manually creating promise
form.on("file", async (field, file) => {
//...
let textProm = await new Promise ((res, rej) => //...
I've also attempted to convert the text manually from the Buffer using fs.readFile, but this also produces the same effect; I can only access text after form.end is hit.
A few things I see is that form.on('file') is hit first, then form.parse. It seems maybe I'm attempting to parse the document twice (Formidable and Pdfreader), but this is probably necessary.
Also, after reading through the docs/stackoverflow, I think I'm mixing the built-in middleware with form.parse/form.on/form.end with manual callbacks, but I was unsure of how to stick with just one, and I'm still able to access the text.
Finally, PDFReader accesses text one line at a time, so parseFileItems is run for every line. I've attempted to resolve a Promise.all with the PdfReader instance, but I couldn't get it to work.
Any help would be greatly appreciated!
I wrote a function that work like this
onNewZipFileRequested
{get all the necessary data}
.then{download all the files}
.then{create a zipfile with all those file}
.then{upload that zipfile} (*here is the problem)
.than{update the database with the signedUrl of the file}
Here is the relevant code
[***CREATION OF ZIP FILE WORKING****]
}).then(() =>{
zip.generateNodeStream({type:'nodebuffer',streamFiles:true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log("zip written.");
return bucket.upload(tempPath, { //**** problem****
destination: destinazionePath
});
});
}).then(()=>{
const config = {
action:'read',
expires:'03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato=>{
const daSalvare ={
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
On the client side, as soon as the app see the status change and the new Url, a download button (pointing to the new url) appears
Everything is working, but if I try to download the file immediately... there is no file yet!!!
If I wait same time and retry the file is there.
I noted that the time I have to wait depend on the size of the zipfile.
The bucket.upload promise should resolve on the end of the upload, but apparently fires too early.
Is there a way to know exactly when the file is ready?
I may have to make same very big file, it's not a problem if the process takes several minutes, but I need to know when it's over.
* EDIT *
there was a unnecessary nesting in the code. While it was not the error (results are the same before and after refactoring) it was causing some confusion in the answers, so i edited it out.
Id' like to point out that i update the database only after getting the signed url, and i get that only after the upload (i could not otherwise), so to get any result at all the promise chain MUST work, and in fact it does. When on the client side the download button appears (happens when 'status' become 'fatto') it is already linked to the correct signed url, but if i press it too early the file is not there (Failed - No file). If i wait some second (the bigger the file the longer i have to wait) then the file is there.
(English is not my mother language, if i have been unclear ask and i will try to explain myself better)
It looks like the problem could be that the braces are not aligned properly, causing a then statement to be embedded within another. Here is the code with the then statements separated:
[***CREATION OF ZIP FILE WORKING****]}).then(() => {
zip.generateNodeStream({type: 'nodebuffer', streamFiles: true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log('zip written.')
return bucket.upload(tempPath, {
destination: destinazionePath
})
})
}).then(() => {
const config = {
action: 'read',
expires: '03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato => {
const daSalvare = {
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
I am looking for a working way to to use the GM methods in CollectionFS transformWrite function depending o the image size. There is a size method implemented in GM but this works async and so it seems to be not possible to use.
I tried the following:
gm(readStream, fileObj.name()).size(function(err, dimensions){
if (err) {
console.log('err with getting size:');
console.log(err);
}
console.log('Result of media_size:');
console.log(dimensions);
// here do smth depends on the dimensions ...
gm(readStream, fileObj.name()).resize('1200', '630').stream().pipe(writeStream);
});
When i use the above snippet in the CollectionFS function I get this error:
Error: gm().stream() or gm().write() with a non-readable stream.
This seems to be a problem that I use a async function - when removing the async function the upload works perfectly but then I have no access to the dimensions of the uploaded image.
Is there a solution to get the dimensions of the image in a sync way when having just access to fileObj, readStream & writeStream ?
Edit:
Thanks Jasper for the hint with the wrapAsync. I tested it and have this code in use:
var imgsize;
var img = gm(readStream, fileObj.name());
imgsize = Meteor.wrapAsync(img.size, img);
console.log('call wrapAsync:');
var result;
try {
result = imgsize();
} catch (e) {
console.log('Error:');
console.log(e)
}
console.log('((after imgsize()))');
When take a look at the console.logs the script stops after "call wrapAsync" - also there is no error returning so its very difficult to tell whats the problem. I also tried this with the NPM package "imagesize" with Meteor.wrapAsync(imagesize); and then imgsize(readStream) which causes the same: No console log after "call wrapAsync:".
The core of the problem is not the asynchronous behavior of gm().size(), but the fact that you use the readStream twice. First you use it to get the size of the image, which empties readStream. Then you try to use it again to resize but because it has ended, you get an error telling you the stream is non-readable.
I found the solution at the bottom of the gm package's streams documenation:
GOTCHA: when working with input streams and any 'identify' operation
(size, format, etc), you must pass "{bufferStream: true}" if you also
need to convert (write() or stream()) the image afterwards NOTE: this
buffers the readStream in memory!
Based on that and the small example below, we can change your code to:
gm(readStream, fileObj.name()).size({ bufferStream: true }, function(err, dimensions){
if (err) {
console.log('err with getting size:');
console.log(err);
}
console.log('Result of media_size:');
console.log(dimensions);
// here do smth depends on the dimensions ...
this.resize('1200', '630').stream().pipe(writeStream);
});
Inside of the callback, this refers to the image you're working on and you can use it to continue your chain.
I tested this in a small sample meteor application, and it works!