I need to write a lambda function and send a number in the api request field to genrate the number of QR codes and store them in a S3 bucket.I am using the serverless framework with the aws-nodejs template.
To describe the task briefly lets say I get a number input in the api request PathParameters and based on these number I have to generate those number of QR's using the qr npm package and then store these generated qr's in the s3 bucket
this is what i have been able to do so far.
module.exports.CreateQR = (event,context) =>{
const numberOfQR = JSON.parse(event.pathParameters.number) ;
for(let i=0;i<numberOfQR;i++){
var d= new Date();
async function createQr(){
let unique, cipher, raw, qrbase64;
unique = randomize('0', 16);
cipher = key.encrypt(unique);
raw = { 'version': '1.0', data: cipher, type: 'EC_LOAD'}
// linkArray.forEach( async (element,index) =>{
let qrcode = await qr.toDataURL(JSON.stringify(raw));
console.log(qrcode);
// fs.writeFileSync('./qr.html', `<img src="${qrcode}">`)
const params = {
Bucket:BUCKET_NAME,
Key:`QR/${d}/img${i+1}.jpg`,
Body: qrcode
};
s3.upload(params , function(err,data){
if(err){
throw err
}
console.log(`File uploaded Successfully .${data.Location}`);
});
}
createQr();
}
};
I have been able to upload a given number of images to the bucket but the issue i am facing is the images are not going in the order.I think the problem is with the asynchronous code. Any idea how to solve this issue
that's because you're not awaiting the s3 to upload, but instead you have a callback.
you should use the .promise of s3 and then await it, so you'll wait the file to be uploaded before move to the next one
I changed the example of code
See docs:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#promise-property
// see the async keyword before the lambda function
// we need it for use the await keyword and wait for a task to complete before continue
module.exports.CreateQR = async (event,context) =>{
const numberOfQR = JSON.parse(event.pathParameters.number) ;
// moved your function out of the loop
function createQr(){
// ...
// here we call .promise and return it so we get a Task back from s3
return s3.upload(params).promise();
}
for(let i=0;i<numberOfQR;i++){
// ....
// here we await for the task, so you will get the images being created and uploaded in order
await createQr();
}
};
hope it guide you towards a solution.
Related
I am working on an nodejs image server to read and write images on Oracle Object Storage.
The issue I am having is not getting the full image when using the function getObject using the Javascript api from oci-objectstorage
I have succesfully stored the following images.
1x1 image with the size of 70 bytes and another
5120 x 3200 image with size 2.9 MB
When I use the function getObject I am able to retrieve the full 1x1 image but when I attempt it with the 5120 x 3200 image, I can only get 15KB of 2.9MB
I used the following example from Oracle
https://github.com/oracle/oci-typescript-sdk/blob/master/examples/javascript/objectstorage.js
Below is the code that I am using to read the image from Oracle Object Storage
I have the below code in an async function.
router.get('/data/', async function (req, res, next) {
let path = req.query.image_data
fs.access(imagePath, fs.F_OK, async (err) => {
if (err) {
const provider = new common.ConfigFileAuthenticationDetailsProvider();
const client = new os.ObjectStorageClient({
authenticationDetailsProvider: provider
});
const compartmentId = config.COMPARTMENTID
const bucket = config.BUCKET
const request = {};
const response = await client.getNamespace(request);
const namespace = response.value;
const getObjectRequest = {
objectName: imagePath,
bucketName: bucket,
namespaceName: namespace
};
const getObjectResponse = await client.getObject(getObjectRequest);
const head = getObjectResponse.value._readableState.buffer.head.data.toString('base64')
const tail = getObjectResponse.value._readableState.buffer.tail.data.toString('base64')
await fs.writeFile(imagePath, completeImage, {encoding: 'base64'},function(err) {
if (err) return
res.sendFile(path, {root: './imagefiles'}) //using express to serve the image file
});
}
//file exists
res.sendFile(path, {root: './imagefiles'});
})
})
It seems to me that the head and tail both have the same data. I am trying to then write the image using fs.write which then with the large image only write a small portion of the image while with the small 1x1 image it writes the full image.
I am not sure if its an issue with my use of the async/await setup or I may have to use a better implementation using promises that may allow to download the full image.
Any ideas on how to tackle this?
Another small issue I am having is serving the image after writing it. On the webpage I get an error saying could not display the image because it contains errors. But after I refresh the page again, which finds the image since it now exists on disk, it is able to display the image and does not show the previous error.
I have a database with a bunch of dates and an online overview where you can view them, now I know I can copy a URL from my Google Agenda and import this in other calendar clients so I can view the events there.
I want to generate an Express endpoint where I fetch every event every time the endpoint is called and return it in a format that can be imported by other calendar clients. Now with packages like iCal-generator I could generate, read, and return the file whenever a user requests the URL. but it feels redudent to write a file to my storage to then read it, return it and delete it every time it's requested.
What is the most effiecent way to go about this?
Instead of generating the file/calendar data on every request, you could implement a simple caching mechanism. That is, upon start of your node app you generate the calendar data and put it in your cache with corresponding time to live value. Once the data has expired or new entries are inserted into your DB you invalidate the cache, re-generate the data and cache it again.
Here's a very simple example for an in-memory cache that uses the node-cache library:
const NodeCache = require('node-cache');
const cacheService = new NodeCache();
// ...
const calendarDataCacheKey = 'calender-data';
// at the start of your app, generate the calendar data and cache it with a ttl of 30 min
cacheCalendarData(generateCalendarData());
function cacheCalendarData (calendarData) {
cacheService.set(calendarDataCacheKey, calendarData, 1800);
}
// in your express handler first try to get the value from the cache
// if not - generate it and cache it
app.get('/calendar-data', (req, res) => {
let calendarData = cacheService.get(calendarDataCacheKey);
if (calendarData === undefined) {
calendarData = generateCalendarData();
cacheCalendarData(calendarData);
}
res.send(calendarData);
});
If your app is scaled horizontally you should consider using redis.
100% untested, but I have code similar to this that exports to a .csv from a db query, and it might get you close:
const { Readable } = require('stream');
async function getCalendar(req, res) {
const events = await db.getCalendarEvents();
const filename = 'some_file.ics';
res.set({
'Content-Type': 'text/calendar',
'Content-Disposition': `attachment; filename=${filename}`,
});
const input = new Readable({ objectMode: true });
input.pipe(res)
.on('error', (err) => {
console.error('SOME ERROR', err);
res.status(500).end();
});
events.forEach(e => input.push(e));
input.push(null);
}
if you were going to use the iCal generator package, you would do your transforms within the forEach method before pushing to the stream.
I am working on building a blog API for a practice project, but am using the data from an external API. (There is no authorization required, I am using the JSON data at permission of the developer)
The idea is that the user can enter multiple topic parameters into my API. Then, I make individual requests to the external API for the requested info.
For each topic query, I would like to:
Get the appropriate data from the external API based on the params entered (using a GET request to the URL)
Add the response data to my own array that will be displayed at the end.
Check if each object already exists in the array (to avoid duplicates).
res.send the array.
My main problem I think has to do with understanding the scope and also promises in Axios. I have tried to read up on the concept of promise based requests but I can't seem to understand how to apply this to my code.
I know my code is an overall mess, but if anybody could explain how I can extract the data from the Axios function, I think it could help me get the ball rolling again.
Sorry if this is a super low-level or obvious question - I am self-taught and am still very much a newbie!~ (my code is a pretty big mess right now haha)
Here is a screenshot of the bit of code I need to fix:
router.get('/:tagQuery', function(req, res){
const tagString = req.params.tagQuery;
const tagArray = tagString.split(',');
router.get('/:tag', function(req, res){
const tagString = req.params.tag;
const tagArray = queryString.split(',');
const displayPosts = tagArray.map(function(topic){
const baseUrl = "https://info.io/api/blog/posts";
return axios
.get(baseUrl, {
params: {
tag: tag
}
})
.then(function(response) {
const responseData = response.data.posts;
if (tag === (tagArray[0])){
const responseData = response.data.posts;
displayPosts.push(responseData);
} else {
responseData.forEach(function(post){
// I will write function to check if post already exists in responseData array. Else, add to array
}); // End if/then
})
.catch(function(err) {
console.log(err.message);
}); // End Axios
}); // End Map Function
res.send(displayPosts);
});
Node.js is a single thread non-blocking, and according to your code you will respond with the result before you fetching the data.
you are using .map which will fetch n queries.
use Promise.all to fetch all the requests || Promise.allsettled.
after that inside the .then of Promise.all || promise.allsettled, map your result.
after that respond with the mapped data to the user
router.get('/:tag', function (req, res) {
const tagString = req.params.tag;
const tagArray = queryString.split(',');
const baseUrl = "https://info.io/api/blog/posts";
const topicsPromises=tagArray.map((tobic)=>{
return axios
.get(baseUrl, {
params: {
tag: tag
}
})
});
Promise.all(topicsPromises).then(topicsArr=>{
//all the data have been fetched successfully
// loop through the array and handle your business logic for each topic
//send the required data to the user using res.send()
}).catch(err=>{
// error while fetching the data
});
});
your code will be something like this.
note: read first in promise.all and how it is working.
The current project I'm working on requires that multiple processes upload data to a single file in S3. This data comes from multiple sources in parallel, so in order to process all the sources as fast as possible we'll use multiple Nodejs instances to listen to the sources. There are memory and storage constraints, so load all ingested data to memory or store in disk and then perform a single upload is out of question.
To respect those constraints I implemented a streamed upload: it buffers a small portion of the data from a single source and pipes the buffer to an upload stream. This works really well when using a single nodejs process, but, as I mentioned, the goal is to process all sources in parallel. My first try was to open multiple streams to the same object key in the bucket. This simply overrides the file with the data from the last stream to close. So I discarded this option.
// code for the scenario above, where each process will open a separete stream to
// the same key and perform it's data ingestion and upload.
openStreamingUpload() {
const stream = require('stream');
const AWS = require('aws-sdk');
const s3 = new this.AWS.S3(/* s3 config */);
const passThrough = new stream.PassThrough();
const params = {
Key: 'final-s3-file.txt',
Bucket: 'my-bucket',
Body: passThrough
};
s3
.upload(params)
.promise();
return passThrough;
}
async main() { // simulating a "never ending" flow of data
const stream = openStreamingUpload();
let data = await receiveData();;
do {
stream.write(data);
data = await receiveData();
} while(data);
stram.close();
}
main();
Next I went to try the multipart upload that the S3 API offers. At first, I create a multipart upload, obtain its ID and store it in a shared space. After that I try to open multiple multipart uploads on all nodejs processes the cluster will be using, with the same UploadId obtained beforehand. Each one of those multipart uploads should have a stream that will pipe the data received. The problem I came across was that the multipart upload needs to know the part length beforehand and as I'm piping a stream that I don't know when will close or the amount of data it will pipe, it's not possible to calculate it's size. Code inspired by this implementation.
const AWS = require('aws-sdk');
const s3 = new this.AWS.S3(/* s3 config */);
async startMultipartUpload()
const multiPartParams = {
Key: 'final-s3-file.txt',
Bucket: 'my-bucket'
};
const multipart = await s3.createMultipartUpload(multiPartParams).promise();
return multipart.UploadId;
}
async finishMultipartUpload(multipartUploadId) {
const finishingParams = {
Key: 'final-s3-file.txt',
Bucket: 'my-bucket',
UploadId: multipartUploadId
};
const data = await s3.completeMultipartUpload(finishingParams).promise();
return data;
}
async openMultiparStream(multipartUploadId) {
const stream = require('stream');
const passThrough = new stream.PassThrough();
const params = {
Body: passThrough.,
Key: 'final-s3-file.txt',
Bucket: 'my-bucket',
UploadId: multipartUploadId,
PartNumber: // how do I know this part number when it's, in principle, unbounded?
};
s3
.uploadPart(params)
.promise();
return passThrough
}
// a single process will start the multipart upload
const uploadId startMultipartUpload();
async main() { // simulating a "never ending" flow of data
const stream = openMultiparStream(uploadId);
let data = await receiveData();;
do {
stream.write(data);
data = await receiveData();
} while(data);
stram.close();
}
main(); // all the processes will receive and upload to the same UploadId
finishMultipartUpload(uploadId); // only the last process to closm will finish the multipart upload.
Searching around, I came across the article from AWS the presents the upload() API method and says that it abstracts the multipart API to allow use of piped data streams to upload large files. So I wonder if its possible to obtain the uploadId from a streamed 'simple' upload, so I can pass this Id around the cluster and upload to the same object and still maintaining the streaming characteristic. Does anyone ever tried this type of scenario of a 'streamed multipart' upload?
So Basically I want to let the user have the option of uploading an image when they register. I have no idea where to start forever. I know that CouchDB supports attachments, but how exactly does that work with Cradle.
I found the following code in Cradle's documentation
saveAttachment: function (/* id, [rev], attachmentName, contentType, dataOrStream */) {
So I know it can save attachments. How would I pass in the image then? I'm assuming that in the html, i have to use
form(action='/upload', enctype='multipart/form-data', method='post')
input(type='file', name='upload')
input(type='submit', value='Upload')
But where do I go from there? Wouldn't this step save the image on the server somewhere. Then do I somehow need to get the address of the image and pass that to cradle to save it as an attachment in the CouchDB database.
Thanks in advance if you can help me out!
You need to take the incoming stream from the form and then send a stream to CouchDB via Cradle.
Sending the stream to Cradle is probably the easy bit. This example shows how to do it with a local file:
db.saveAttachment(
doc.id,
doc.rev,
attachmentId,
mimetype,
fs.createReadStream(path),
function( err, data ){
console.log(data);
}
);
The trickier bit in my opinion is managing incoming files. They arrive as a multipart stream rather than being saved to a file. My preference would be to outsource that code to formidable, either directly, or indirectly via connect-form if you're using Connect or Express.
My current connect-form code can be summarised to this:
req.form.complete(function(err, fields, files){
if ( err ) // handle err
else
{
db.saveAttachment(
doc.id,
doc.rev,
attachmentId,
mimetype,
fs.createReadStream(files.name),
function( err, data ){
console.log(data);
}
);
}
});
This isn't optimal for speed as it creates an actual file on disk rather than streaming data from one place to the other, but it is convenient and may satisfy a lot of use cases.
Another package you should be aware of if you're dealing with image upload is node-imagemagick, which as you might expect from the name is the node.js wrapper for ImageMagick.
I wrote up some documentation on attachments that hopefully will be merged into the cradle readme soon. For now though here is the relevant section
Attachments
Cradle supports writing, reading, and removing attachments. The read and write operations can be either buffered or streaming
Writing
You can buffer the entire attachment body and send it all at once as a single request. The callback function will fire after the attachment upload is complete or an error occurs
Syntax
db.saveAttachment(idData, attachmentData, callbackFunction)
Example
Say you want to save a text document as an attachment with the name 'fooAttachment.txt' and the content 'Foo document text'
var doc = <some existing document>
var id = doc._id
var rev = doc._rev
var idAndRevData = {
id: id,
rev: rev
}
var attachmentData = {
name: 'fooAttachment.txt',
'Content-Type': 'text/plain',
body: 'Foo document text'
}
db.saveAttachment(idAndRevData, attachmentData, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})
Streaming
You can use a read stream to upload the attachment body rather than buffering the entire body first. The callback function will fire after the streaming upload completes or an error occurs
Syntax
var doc = savedDoc // <some saved couchdb document which has an attachment>
var id = doc._id
var rev = doc._rev
var idAndRevData = {
id: id,
rev: rev
}
var attachmentData = {
name: attachmentName // something like 'foo.txt'
'Content-Type': attachmentMimeType // something like 'text/plain', 'application/pdf', etc.
body: rawAttachmentBody // something like 'foo document body text'
}
var readStream = fs.createReadStream('/path/to/file/')
var writeStream = db.saveAttachment(idData, attachmentData, callbackFunction)
readStream.pipe(writeStream)
When the streaming upload is complete the callback function will fire
Example
Attach a pdf file with the name 'bar.pdf' located at path './data/bar.pdf' to an existing document
var path = require('path')
var fs = require('fs')
// this document should already be saved in the couchdb database
var doc = {
_id: 'fooDocumentID',
_rev: 'fooDocumentRev'
}
var idData = {
id: doc._id,
rev: doc._rev
}
var filename = 'bar.pdf' // this is the filename that will be used in couchdb. It can be different from your source filename if desired
var filePath = path.join(__dirname, 'data', 'bar.pdf')
var readStream = fs.createReadStream
// note that there is no body field here since we are streaming the upload
var attachmentData = {
name: 'fooAttachment.txt',
'Content-Type': 'text/plain'
}
db.saveAttachment(idData, attachmentData, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
}, readStream)
Reading
Buffered
You can buffer the entire attachment and receive it all at once. The callback function will fire after the download is complete or an error occurs. The second parameter in the callback will be the binary data of the attachment
Syntax
db.getAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to read back an attachment that was saved with the name 'foo.txt'
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
db.getAttachment(id, attachmentName, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})
Streaming
You can stream the attachment as well. If the attachment is large it can be useful to stream it to limit memory consumption. The callback function will fire once the download stream is complete. Note that there is only a single error parameter passed to the callback function. The error is null is no errors occured or an error object if there was an error downloading the attachment. There is no second parameter containing the attachment data like in the buffered read example
Syntax
var readStream = db.getAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to read back an attachment that was saved with the name 'foo.txt'. However the attachment foo.txt is very large so you want to stream it to disk rather than buffer the entire file into memory
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
var downloadPath = path.join(__dirname, 'foo_download.txt')
var writeStream = fs.createWriteStream(downloadPath)
var readStream = db.getAttachment('piped-attachment', 'foo.txt', function (err) { // note no second reply paramter
if (err) {
console.dir(err)
return
}
console.dir('download completed and written to file on disk at path', downloadPath)
})
readStream.pipe(writeStream)
Removing
You can remove uploaded attachments with a _id and an attachment name
Syntax
db.removeAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to remove an attachment that was saved with the name 'foo.txt'
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
db.removeAttachment(id, attachmentName, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})
FYI, for future readers, the calling parameters have changed since then, so this appears to be no longer valid. Check the source as the documentation doesn't describe how to use it.