I'm trying to lift all the functions in an Amazon S3 object using when/node.
var when = require('when');
var nodefn = require('when/node');
var AWS = require('aws-sdk');
var s3 = new AWS.S3();
var promisedS3 = nodefn.liftAll(s3);
when(promisedS3.listBuckets())
.then(function(data) {
console.log(data);
})
However, it looks like a request object is being printed out. I'm kind of at a loss as to what is happening here, I can get the correct results if I individually lift functions like so:
var listBucketsP = nodefn.lift(s3.listBuckets.bind(s3));
Any ideas?
Try this:
nodefn.liftAll(s3.__proto__, undefined, s3);
Then just do
s3.listBuckets().then(function(data) {
console.log(data);
});
This worked for me.
Explanation: the methods you are trying to modify are not part of the s3 object itself, but of its prototype. When's node.liftAll's 3 argument version takes the source object first, an optional transformation function, and finally the destination object (to attach the lifted functions onto).
So we're taking the functions from the prototype and attach the promisified versions to the object we're working with.
Related
I am trying to use util.promisify to convert AWS Document client get function to a promise based utility. But it does not seem to behave as expected;
// This does not work as expected
const docClient = new AWS.DynamoDB.DocumentClient();
let docClientGet = require('util').promisify(docClient.get);
However when i do usual promise conversion like this,
let docClientGet = function (params) {
return new Promise((resolve, reject) => {
docClient.get(params, function (err, data) {
if (err) {
return reject(err);
}
return resolve(data);
})
})
};
And use it in an async function like this:
await docClientGet(params);
It works!.
I wonder where I am wrong in understanding util.promisify
If the method you are promisifying needs to be associated with the object it is on (which it looks like it does in your case, then this code:
let docClientGet = utils.promisify(docClient.get);
will not retain the association with the docClient object. What happens is that the promisified docClient.get() gets called without the this value set to the docClient object and it can't do its job properly.
You can work around that with this:
utils.promisify(docClient.get.bind(docClient));
The promisify doc does not make this clear because it uses an example from the fs library whose methods do not need to be associated with the fs object in order to work properly.
P.S. It's a bit unusual to put the util library into a variable named utils. That is likely to confuse some people reading your code.
I'm trying to run a specific function from an existing app via AWS Lambda, using the JS SDK to invoke Lambda from my node.js app. Since I'm overwriting the existing function, I'll have to keep its basic structure, which is this:
overwrittenFunction = function(params) {
//get some data
return dataArray;
}
..so I need to end up with an array that I can return, if I'm looking to keep the underlying structure of the lib I use the same. Now as far as I know, Lambda invocations are asynchronous, and it's therefore not possible to do something like this:
overwrittenFunction = function(params) {
lambda.invoke(params, callback);
function callback(err,data) {
var dataArray = data;
}
return dataArray;
}
(I've also tried similar things with promises and async/await).
afaik I have two options now: somehow figure out how to do a synchronous Lambda invocation, or modify my library / existing app (which I would rather not do if possible).
Is there any way to do such a thing and somehow return the value I'm expecting?
(I'm using node v8.9.4)
Lambda and async/await are a bit tricky, but the following is working for me (in production):
const lambdaParams = {
FunctionName: 'my-lambda',
// RequestResponse is important here. Without it we won't get the result Payload
InvocationType: 'RequestResponse',
LogType: 'Tail', // other option is 'None'
Payload: {
something: 'anything'
}
};
// Lambda expects the Payload to be stringified JSON
lambdaParams.Payload = JSON.stringify(lambdaParams.Payload);
const lambdaResult = await lambda.invoke(lambdaParams).promise();
logger.debug('Lambda completed, result: ', lambdaResult.Payload);
const resultObject = JSON.parse(lambdaResult.Payload)
Wrap it all up in a try/catch and go to town.
You can use async await but as the AWS SDK uses node callback pattern you'll need to wrap the function with the built-in promisify.
const promisify = require('utils').promisify
const aws = require('aws-sdk');
const lambda = aws.Lambda();
const invoke = promisify(lambda.invoke);
async function invocation(params) {
try {
return await invoke(params);
} catch (err) {
throw new Error('Somethings up');
}
}
const data = invocation(params);
I'm trying to update one value after a write completes (in a Cloud Function) but it just wont work (I'm sure this is a stupidly simple problem). Code below:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const firebase = require('firebase');
admin.initializeApp(functions.config().firebase);
exports.createMessage = functions.https.onRequest((request, response) => {
const json = JSON.parse(request.query.json); // == "{'start':0, 'end':0}"
json.start = firebase.database.ServerValue.TIMESTAMP;
admin.database().ref('/messages/').push(json).then(snapshot => {
//Here is the problem. Whatever I try here it won't work to retrieve the value.
//So, how to I get the "start" value, which has been written to the DB (TIMESTAMP value)?
var startValue = snapshot.ref.child('start').val();
snapshot.ref.update({ end: (startValue + 85800000) }).then(snapshot2=>{
response.redirect(303, snapshot.ref);
});
});
});
Is the problem that I'm using admin.database()?
This code:
var startValue = snapshot.ref.child('start').val();
doesn't actually retrieve any values. Take a look at the docs for DataSnapshot. Reach into that snapshot directly with child() - you don't need the ref. Maybe this is what you meant?
var startValue = snapshot.child('start').val();
I'm not sure if there's a bug in Firebase or if I'm using it wrong, but if I try to call any method on the snapshot-reference I will only get an error saying: TypeError: snapshot.xxx is not a function where xxx is the function name i try to use (for example: child(...), forEach(...), etc).
However, the following seems to fix the issue with the snapshot:
admin.database().ref('/messages/').push(json).once('value').then(snapshot => {
instead of:
admin.database().ref('/messages/').push(json).then(snapshot => {
My uneducated guess is that the then-promise, for the push-function returns some faulty snapshot since the only thing that seems to work is snapshot.key.
Also, if I'm not mistaken, doesn't my solution make two reads now, instead of one? Since push will write and then (supposedly) read and return the written value and then I read it once more with once(value).
Does anyone has any further insights into this problem?
It seems it is not possible to pass around some code (containing data and functions) that is invoked as a AWS lambda function within another AWS lambda function.
For example, take this customConfigLambda:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
exports.handler = (event, context, callback) => {
callback(null, callbackPayload);
};
When I call the previous AWS lambda function in another AWS lambda function like here:
var AWS = require('aws-sdk');
AWS.config.update({accessKey: '123', secretAccessKey: 'abc', region: 'us-east-1' });
var lambda = new AWS.Lambda({region: 'us-east-1'});
exports.handler = (event, context, callback) => {
var params = {FunctionName: 'customConfigLambda'};
lambda.invoke(params, function(err, callbackPayload) {
if (err) {
// do nothing
}
else {
console.log('callbackPayload:', JSON.stringify(callbackPayload, null, 2));
}
});
};
Then I can see only callbackPayload.fooData but not callbackPayload.fooFunction.
How can I have some callbackPayload.fooFunction(s) shared between multiple other AWS lambda functions?
As of AWS Reinvent 2018, Amazon has introduced Lambda Layers.
Lambda Layers, a way to centrally manage code and data that is shared across multiple functions.
The idea is that now you can put common components in a ZIP file and upload it as a Lambda Layer. Your function code doesn’t need to be changed and can reference the libraries in the layer as it would normally do instead of packaging them separately.
See the docs on Using the Callback Parameter at:
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html#nodejs-prog-model-handler-callback
It says this about the result (the callbackPayload in your code):
result – is an optional parameter that you can use to provide the
result of a successful function execution. The result provided must be
JSON.stringify compatible. If an error is provided, this parameter is
ignored.
To be JSON.stringify compatible you cannot have any functions there. See the http://json.org/ to see what is valid JSON (only strings, numbers, objects, arrays, true, false and null).
If you want to share code between your AWS Lambda functions in a broad sense, you have to require the same Node module in both of them, so that you can make a common set of functions available to all of your AWS Lamda handlers. But you cannot pass around arbitrary code between them because those will not survive the JSON.stringify.
As a test you can try running this in the browser:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
alert(JSON.stringify(callbackPayload));
(see DEMO)
or this in Node:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
console.log(JSON.stringify(callbackPayload));
and see the result:
{"fooData":"fooFromData"}
The functions is stripped out during the serialization process.
Of course you could do something like this:
callbackPayload.fooFunction
= function() {return 'fooFromFunction'; }.toString();
and get a JSON result:
{"fooData":"fooFromData","fooFunction":"function () {return 'fooFromFunction'; }"}
which you could theoretically eval on the other end but I wouldn't really recommend it.
In Meteor, on the server side, I want to use the .find() function on a Collection and then get a Node ReadStream interface from the curser that is returned. I've tried using .stream() on the curser as described in the mongoDB docs Seen Here. However I get the error "Object [object Object] has no method 'stream'" So it looks like Meteor collections don't have this option. Is there a way to get a stream from a Meteor Collection's curser?
I am trying to export some data to CSV and I want to pipe the data directly from the collections stream into a CSV parser and then into the response going back to the user. I am able to get the response stream from the Router package we are using, and it's all working except for getting a stream from the collection. Fetching the array from the find to push it into the stream manually would defeat the purpose of a stream since it would put everything in memory. I guess my other option is to use a foreach on the collection and push the rows into the stream one by one, but this seems dirty when I could pipe the stream directly through the parser with a transform on it.
Here's some sample code of what I am trying to do:
response.writeHead(200,{'content-type':'text/csv'});
// Set up a future
var fut = new Future();
var users = Users.find({}).stream();
CSV().from(users)
.to(response)
.on('end', function(count){
log.verbose('finished csv export');
response.end();
fut.ret();
});
return fut.wait();
Have you tried creating a custom function and piping to it?
Though this would only work if Users.find() supported .pipe()(again, only if Users.find inherited from node.js streamble object).
Kind of like
var stream = require('stream')
var util = require('util')
streamreader = function (){
stream.Writable.call(this)
this.end = function() {
console.log(this.data) //this.data contains raw data in a string so do what you need to to make it usable, i.e, do a split on ',' or something or whatever it is you need to make it usable
db.close()
})
}
util.inherits(streamreader,stream.Writeable)
stream.prototype._write = function (chunk, encoding, callback) {
this.data = this.data + chunk.toString('utf8')
callback()
}
Users.find({}).pipe(new streamReader())