We are trying to migrate our zip microservice from regular application in nodejs Express to AWS API Gateway integrated with AWS Lambda.
Our current application sends request to our API, gets list of attachments and then visits those attachments and pipes their content back to user in form of zip archive. It looks something like this:
module.exports = function requestHandler(req, res) {
//...
//irrelevant code
//...
return getFileList(params, token).then(function(fileList) {
const filename = `attachments_${params.id}`;
res.set('Content-Disposition', `attachment; filename=${filename}.zip`);
streamFiles(fileList, filename).pipe(res); <-- here magic happens
}, function(error) {
errors[error](req, res);
});
};
I have managed to do everything except the part where I have to stream content out of Lambda function.
I think one of possible solutions is to use aws-serverless-express, but I'd like a more elegant solution.
Anyone has any ideas? Is it even possible to stream out of Lambda?
Unfortunately lambda does not support streams as events or return values. (It's hard to find it mentioned explicitly in the documentation, except by noting how invocation and contexts/callbacks are described in the working documentation).
In the case of your example, you will have to await streamFiles and then return the completed result.
(aws-serverless-express would not help here, if you check the code they wait for your pipe to finish before returning: https://github.com/awslabs/aws-serverless-express/blob/master/src/index.js#L68)
n.b. There's a nuance here that a lot of the language SDK's support streaming for requests/responses, however this means connecting to the stream transport, e.g. the stream downloading the complete response from the lambda, not listening to a stream emitted from the lambda.
Had the same issue, now sure how you can do stream/pipe via the native lambda + API Gateway directly... but it's technically possible.
We used Serverless Framework and were able to use XX.pipe(res) using this starter kit (https://github.com/serverless/examples/tree/v3/aws-node-express-dynamodb-api)
What's interesting is that this just wraps over native lambda + API Gateway so, technically it is possible as they have done it.
Good luck
Related
Beginner here, I'm using Firebase real time database and I need my API to constantly return that value when something has been added see my code below.
apiCalls.get('/api/getallusers',function(req,res){
userFunc.getAllUsers(function(err,result){
if (err) return res.status(500).send('internal server error!');
res.status(200).write(JSON.stringify(result));
res.end();
return res;
})
})
this will return the error
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
but if i remove res.end it will show 1 record and constantly load until the page times out..
is what I'm doing possible or are there different ways to do it.
also I'm using firebase cloud functions for this api.
UPDATE:
Uploaded the API but it does not return anything...
here is the link https://us-central1-testproject-e6819.cloudfunctions.net/api1/api/getUser
tried axios and Event Source
Firebase functions logs the values but it does not return it..
If you're viewing the API response like a web page, your browser is buffering the data it's received until there's enough of it to form a more full page. Your browser is expecting content that ends, not some endless stream of data.
You should remove .end() if you expect to be able to continue to write to the output stream.
Also, I recommend using the Server-Sent Events (SSE) protocol for this. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events It provides a nice standards-based abstraction that makes it very easy to handle event streams client-side.
const eventSource = new EventSource('https://api.example.com/someApi');
eventSource.addEventListener('userupdate', (e) => {
console.log(e.data);
});
Server-side, there are a couple Express-based middlewares to make this even easier than it already is.
Operations in Cloud Functions must be relatively short-lived and end deterministically. There is no way to keep a connection open from Cloud Functions to the client.
Typically consider what triggers the need to send new data. For example, if it is triggered by the fact that a new user is registered, you can use trigger your Cloud Functions from Firebase Authentication. Then the function could for example write to the Realtime Database (or Cloud Firestore), and your client/app listens to the database for realtime updates. That way you're using all the pieces of Firebase in the way they're designed: Cloud Functions for short-lived updates triggered from events in the system, and the Realtime Database or Cloud Firestore for sending realtime updates.
If that doesn't work for your use-case, you'll need a runtime environment that allows you to keep processes alive. Something like App Engine flex, Kubernetes, or many other options come to mind for that.
I'm trying to query a DynamoDB table from a Lambda function (for an Alexa skill), but when I send the intent that calls require('aws-sdk'), the skill seems to hang and timeout. The Alexa test page just says "There was a problem with the requested skill's response", and I don't see any errors in the CloudWatch logs. I have the skill set up to catch any errors and return them as a spoken response, so I'm sure it's not an uncaught exception. I've also tried wrapping the require in a try/catch block, but that didn't work either.
This is the module that gets loaded with require if the test database intent request is received:
const AWS = require('aws-sdk');
module.exports = () => {
return 'Success!';
};
If I comment out require('aws-sdk'), the function works properly and Alexa responds with "Success".
Why does my skill break when all I'm doing is requiring the aws-sdk module?
I'm very new to AWS and this is my first experience trying to access a DynamoDB table in a Lambda function.
The Lambda function is uploaded as a zip that contains my source code, package.json (that includes aws-sdk as a dependency), and the node_modules folder.
After hours of debugging, I've found that changing import * as AWS from 'aws-sdk'; to import {DynamoDB} from 'aws-sdk'; (or {CloudFront} or whatever you actually use) made the timeout issue disappear. Mind you, the time to actually connect to DynamoDB was never an issue for me, it was always the import line where the timeout happened.
This can be fixed by either increasing the timeout or the memory allotted to the lambda function.
This is probably because the SDK is too big to be imported by the default timeout value of 3 seconds and the default memory value of 128 MB.
This is why it will probably work if you try importing smaller components like only DynamoDB.
Lambda, when using NodeJS, uses a callback continuation model. Your module should export a function that takes three parameters: event, context, and callback.
Event provides input parameters.
The other two are used for returning control from your handler function, depending on the NodeJS version you are using.
Try adding the three parameters that I mentioned and the, from within your exported handler function, call:
module.export = function(event, context, callback) {
callback(‘success’);
}
Bear in mind, I wrote this on mobile off the top of my mind, so you may need to make small afjustments to the code but the idea is the same. Don’t return directly from the function, but call the callback to supply the response as a continuation. And note that in earlier versions of NodeJS, prior to version 4, you would have to use the context to set success or failure, rather than calling the callback.
For more details, see the Lambda with NodeJS tech docs on AWS.
The other thing to keep in mind is that for Alexa specifically, the response should be in the correct format. That is a JSON response that contains all the necessary elements as explained in the Alexa Skills Kit tech docs.
The Alexa ASK sdk that you’ve included generates those responses but I thought I should point you to the actual docs in case you were going to try building the response by hand to understand how it works.
I want to make a Google Cloud Function with HTTP trigger that call another function (example: changeString). I know that I can include the function changeString in index.js. However, I want to reuse changeString so others Google Cloud Functions can call it.
exports.helloWorld = function helloWorld(req, res) {
var result = changeString(req.body.string);
res.send(result);
};
I know that there is a similar question, but it did not solve my problem.
I was wondering about this myself and I think the answer is that you don’t call the function. Instead, you should send a payload to the PubSub service from your HTTP Cloud Function. A secondary Cloud Function subscribes to the PubSub topic and consumes the payload (which is Base64 encoded).
As #user3158158 brings out, you would publish a message to a Pub/Sub service. This is highlighted here: https://cloud.google.com/functions/docs/calling/pubsub#publishing_a_message_from_within_a_function
I shows the sample syntax at the above link, I needed to do the same thing today.
The Alexa skill docs will eventually allow you to send webhooks to https endpoints. However the SDK only documents lambda style alexa-sdk usage. How would one go about running Alexa applications on one's own server without anything abstracting Lambda? Is it possible to wrap the event and context objects?
You can already use your own endpoint. When you create a new skill, in the configuration tab, just choose HTTPS and provide your https endpoint. ASK will call your endpoint where you can run anything you want (tip, check ngrok.com to tunnel to your own dev machine). Regarding the event and context objects; your endpoint will receive the event object information. You don't need the context object for anything, that just lets you interact with Lambda-specific stuff (http://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html). Just make sure that you comply with the (undocumented) timeouts by ASK and you are good to go.
Here's a way to do this that requires only a small change to your Skill code:
In your main index.js entry point, instead of:
exports.handler = function (event, context) {
use something like:
exports.myAppName = function (funcEvent, res) {
Below that, add the following workaround:
var event = funcEvent.body
// since not using Lambda, create dummy context with fail and succeed functions
const context = {
fail: () => {
res.sendStatus(500);
},
succeed: data => {
res.send(data);
}
};
Install and use Google Cloud Functions Local Emulator on your laptop. When you start and deploy your function to the emulator, you will get back a Resource URL something like http://localhost:8010/my-project-id/us-central1/myAppName.
Create a tunnel with ngrok. Then take the ngrok endpoint and put it in place of localhost:8010 in the Resource URL above. Your resulting fulfillment URL will be something like: https://b0xyz04e.ngrok.io/my-project-id/us-central1/myAppName
Use the fulfillment URL (like above) under Configuration in the Alexa dev console, selecting https as the Service Endpoint Type.
Folks,
I would like to set up a message queue between our Java API and NodeJS API.
After reading several examples of using aws-sdk, I am not sure how to make the service watch the queue.
For instance, this article Using SQS with Node: Receiving Messages Example Code tells me to use the sqs.receiveMessage() to receive and sqs.deleteMessage() to delete a message.
What I am not clear about, is how to wrap this into a service that runs continuously, which constantly takes the messages off the sqs queue, passes them to the model, stores them in mongo, etc.
Hope my question is not entirely vague. My experience with Node lies primarily with Express.js.
Is the answer as simple as using something like sqs-poller? How would I implement the same into an already running NodeJS Express app? Quite possibly I should look into SNS to not have any delay in message transfers.
Thanks!
For a start, Amazon SQS is a pseudo queue that guarantees availability of messages but not their sequence in FIFO fashion. You have to implement sequencing logic into your app if you want it to work that way.
Coming back to your question, SQS has to be polled within your app to check if there are new messages available. I implemented this in an app using setInterval(). I would poll the queue for items and if no items were found, I would delay the next call and in case some items were found, the next call would be immediate bypassing the setInterval(). This is obviously a very raw implementation and you can look into alternatives. How about a child process on your server that pings your NodeJS app when a new item is found in SQS ? I think you can implement the child process as a watcher in BASH without using NodeJS. You can also look into npm modules if there is already one for this.
In short, there are many ways you can poll but polling has to be done one way or the other if you are working with Amazon SQS.
I am not sure about this but if you want to be notified of items, you might want to look into Amazon SNS.
When writing applications to consume messages from SQS I use sqs-consumer:
const Consumer = require('sqs-consumer');
const app = Consumer.create({
queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
handleMessage: (message, done) => {
console.log('Processing message: ', message);
done();
}
});
app.on('error', (err) => {
console.log(err.message);
});
app.start();
See the docs for more information (well documented):
https://github.com/bbc/sqs-consumer