I have a bunch of AVRO files in an S3 bucket. Each file contains a series of records. Every time a file is uploaded to the Bucket, a Lambda is triggered. I want to read the content of the AVRO file (the records) and save them in a more friendly format, for instance push all entries to an array so I could do stuff with it.
I am using the client-s3 from the aws-sdk lib.
I have tried with the following piece of code but I am not able to get a working result.
let client = new S3Client({})
let command = new GetObjectCommand(parameters)
const { Body } = await client.send(command)
I have tried other solutions too that I found on the Internet and AWS Docs but I do not seem to find a way to make this work.
Related
I'm confused on two things. I need to do these 2, but am not sure how:
Get the buffer of a PRIVATE image in cloud firestore from node
change the name of an imagefile thats in storage.(Either by setting the name as it is being uploaded, or changed afterward, both work).
Here is my current code that logs some giant thing that does indeed contain the file, but I can't find how to get the image data from it besides just the name of it/oth irrelivant stuff.
var theFile = bucket.file('Walks For Lives.png');
console.log(theFile)
Ideally, to do both of these things, I seek some way to do this:
var theFile = bucket.file('Walks For Lives.png');
var the imageBuffer = theFile.buffer;
theFile.name.set("newName");
I am using Gridfs to store large files in mognodb.
Now I am using PDFTron for pdf editing and want to watermark pdf.
The problem is i am not able to read file from Gridfs stream in pdftron nodejs sdk.
also i want to send it back to the client without storing it locally or anywhere else.
I am doing something like this...
const bucket = new mongodb.GridFSBucket(db);
const stream = bucket.openDownloadStream(ObjectId(file_id))
const pdfdoc = await PDFNet.PDFDoc.createFromFilter(stream);
the error i am getting is ...
TypeError: 1st input argument in function 'createFromFilter' is of type 'object'. Expected type 'Filter'. Function Signature: createFromFilter(PDFNet.Filter)
The PDFDoc.createFromFilter API is expecting a PDFNet Filter, not whatever GridFS is returning.
https://www.pdftron.com/api/pdfnet-node/PDFNet.PDFDoc.html#.createFromFilter__anchor
You can see this sample on creating a PDFDoc object from a Filter
https://www.pdftron.com/documentation/samples/node/js/PDFDocMemoryTest
Though the easiest is to write your GridFD stream to a buffer, and then pass that buffer to PDFDoc.createFromBuffer. https://www.pdftron.com/api/pdfnet-node/PDFNet.PDFDoc.html#.createFromBuffer__anchor
looking for some suggestions as I'm stuck as to how i can achieve my next task.
i have a csv file test.csv in S3 bucket. Now, i have to read this file in my Lambda function and convert to HTML table and come up with a URL that i can save on S3 so that i can later
email to users. when users click the URL they should see this HTML table.
Read lot of articles but not sure where to start,
I'd appreciate if anyone break above tasks into steps then i can start writing code to achieve this step by step and can get help if i get any issue
below is where i'm in my Lambda function
let readParam = {
Bucket: process.env.S3_BUCKET,
Key: process.env.CSV_OUTPUT,
};
s3Param.getObject(readParam, function(err, data){
const body = Buffer.from(data.Body).toString('utf8');
})
any help please... i'm trying to accomplish this in Nodejs
I set up my discord BOT using node.js. For my advantage, I would need to store some data on a external file, but I don't seem to be able to access it from my index.js file (the main Bot file).
I've tried having one static array in the external js/json files, but I can only retrieve undefined/empty values. Additionally, when I tried with a .txt file, once retrieved the content, I found it unable to call functions such as string.split().
Did I miss something in the package content perhaps?
Assuming the data you are storing is in UTF-8 encoding:
var fs = require('fs');
fs.readFile('path/to/file', 'utf8', function(err, contents) {
// code using file data
});
Assuming no errors contents will be a string of the data that is inside that file.
https://code-maven.com/reading-a-file-with-nodejs
I'm trying to save a remote image file into a database, but I'm having some issues with it since I've never done it before.
I need to download the image and pass it along (with node-request) with a few other properties to another node api that saves it into a mysql database (using sequelize). I've managed to get some data to save, but when I download it manually and try to open it, it's not really usable and no image shows up.
I've tried a few things: getting the image with node-request, converting it to a base64 string (read about that somewhere) and passing it along in a json payload, but that didn't work. Tried sending it as a multipart, but that didn't work either. Haven't worked with streams/buffers/multipart all that much before and never in node. I've tried looking into node-request pipes, but I couldn't really figure out how possibly apply them to this context.
Here's what I currently have (it's a part es6 class so there's no 'function' keywords; also, request is promisified):
function getImageData(imageUrl) {
return request({
url: imageUrl,
encoding: null,
json: false
});
}
function createEntry(entry) {
return getImageData(entry.image)
.then((imageData) => {
entry.image_src = imageData.toString('base64');
var requestObject = {
url: 'http://localhost:3000/api/entry',
method: 'post',
json: false,
formData: entry
};
return request(requestObject);
});
}
I'm almost 100% certain the problem is in this part because the api just takes what it gets and gives it to sequelize to put into the table, but I could be wrong. Image field is set as longblob.
I'm sure it's something simple once I figure it out, but so far I'm stumped.
This is not a direct answer to your question but it is rarely needed to actually store an image in the database. What is usually done is storing an image on storage like S3, a CDN like CloudFront or even just in a file system of a static file server, and then storing only the file name or some ID of the image in the actual database.
If there is any chance that you are going to serve those images to some clients then serving them from the database instead of a CDN or file system will be very inefficient. If you're not going to serve those images then there is still very little reason to actually put them in the database. It's not like you're going to query the database for specific contents of the image or sort the results on the particular serialization of an image format that you use.
The simplest thing you can do is save the images with a unique filename (either a random string, UUID or a key from your database) and keep the ID or filename in the database with other data that you need. If you need to serve it efficiently then consider using S3 or some CDN for that.