First of all I am a newbie in NodeJS and want to imporve my skills on it.
I have a table in Airtable and want to get all the elements from it.
Easy with the airtable api for nodejs.
But what I want to do is push and save these elements in a tab for the future(JSON, excel ...).
To do so, I am using callbaks since the call is async.. I heared about Promises but it's very new to me, and I am hardly uderstanding it..
Here is my code for now:
var Airtable = require('airtable');
Airtable.configure({
endpointUrl: 'https://api.airtable.com',
apiKey: 'keyKWYJPOEObWhNt2'
});
var base = Airtable.base('app4qIwfmG0ZKAdBH');
var view = "Main View";
var tab = [];
base('Table 1').select({
view : view}).eachPage(function page(records, fetchNextPage){records.forEach(function(record){
tab.push({
"Name": record.get('Name'),
"Notes": record.get('Notes')
});
});
fetchNextPage();
pushToArray(tab);
}, function done (error){
if(error){ console.log(error);
console.log(tab);}
});
function pushToArray(tab) {
TabToJson(tab);
return tab;
};
function TabToJson(tab){
console.log(tab);
return JSON.stringify(tab);
};
How can I implements promises? Is it necessary here? I don't want to end up with dozen of callback functions..
Thank you all and have a nice day!
Careful here! You're on the right track with realizing that this function is async and that you want to wait until every iteration of #eachpage has resolved before writing outputting your JSON, like a Promise would. But Airtable was kind enough to already provide what you're looking for: the callback
function done (error){
if(error){ console.log(error);
console.log(tab);
}
}
will run immediately after the last successful call to #fetchNextPage. This is where you should have your JSON-writing logic. You would want something like
function done (error){
TabToJson(tab);
if(error){ console.log(error);
console.log(tab);
}
}
You don't need your function pushToArray, as you've already pushed the individual records from Airtable into your array 'tab' in each call to #page. Furthermore, if you want to do more than log your JSON output, which your question makes it seem, you should look into Node's File System Library. checkout the fs#writeFile method.
You can use async/await for it. Please make sure you can use await in an async function.
try {
const records = await base('Table 1').select({ view }).all()
records.map((record) => {
tab.push({
"Name": record.get('Name'),
"Notes": record.get('Notes')
});
pushToArray(tab)
})
} catch (e) {
console.error(e)
}
Related
I'm making a DNS Lookup API using Node.js and Express.js framework such that when it sends a POST request, it should return the addresses of different record types.
app.post('/', (req, res) => {
// Request format
// const l = {
// lookup: 'twitter.com',
// recordTypes: ['A', 'TXT']
// };
// Using destructor to fetch properties
const { lookup, recordTypes } = req.body;
console.log(lookup, recordTypes);
// For each record type
recordTypes.forEach(function(type) {
// setTimeout to get something async
setTimeout(function() {
dns.resolve(lookup.toLowerCase(), type, (err, addresses) => {
console.log(type);
if (err) {
return console.log(`\nType(${type}):\n`, err);
}
result = result + JSON.stringify({ type: `${type}`, response: { addresses } });
console.log(result);
});
}, 2000);
});
res.send(result);
});
It logs the correct stuff in the console but when it comes to the response, it returns an empty string. I used setTimeout to mimic the asynchronous nature of the request but it just does not work.
Please assume that I have declared stuff like result etc. because it is working. Also, please don't to redirect me to the Node.js documentation because I have already read that stuff and that's not the problem here. The problem is that I need to get every record type in an array and send that back as a response.
Here's what I have tried:
Tried to push response for each record type in the result array,
Tried to use a for of loop instead of forEach
Please help!
The way I'm reading your code is that for each item in the array you correctly use callbacks to do each individual bit of processing.
However, remember that forEach itself is not asynchronous. Thus you are setting up a bunch of tasks that will complete sometime, then returning undefined... then your results start to trickle in.
There's a couple ways to correctly. As you are using callbacks here I will use that style. You want to get a callback when all items in an array have been completely processed. The async module does this very well, providing a lot of high quality methods that act on arrays and such and give you a way to have a callback when they are all over.
Your function will look something like:
let res = []
async.each( recordTypes,
( type, done ) => {
dns.resolve(lookup.toLowerCase(), type, (err, addresses) => {
result = result + JSON.stringify({ type: `${type}`, response: { addresses } });
done(err)
} )
},
(allOverError) => {
res.send(result);
}
)
Notice there are two function parameters here: the first one is called for every item in the list, and the last is called when every item in the list has been completely processed.
There are other ways too, promises or the async/await keywords (confusing because of the name of the async module), but callbacks are good.
I am building a small node.js website with a user interface that features a dropdown with a list of countries.
Previously the list of countries was hard coded in a json file that I would read:
exports.countries = require('./json/countries.json');
Then I realized I shouldn't hard code it like that when I can do a distinct query to the the list from the mongodb database.
db.collection.distinct('c', {}, function(err, data) {
// something
});
But then there's the question of how to extract the value of the data variable in that callback function. I discovered that this works:
db.collection.distinct('c', {}, function(err, data) {
if (err) {
throw Error('mongodb problem - unable to load distinct values');
} else {
exports.countries = data;
}
});
I am new to node.js and this seems fishy to me. Is this OK code? Is it better do this with generators or promises? If I wanted to use generators or promises to do this, how would I do that?
The end result where this is used is in a template. ref.countries is the actual list of countries using my fishy code. If I had a Promise instead of the list of countries, how would I change this code?
<% ref.countries.forEach(function(c) { -%>
<option value="<%= c %>">
<%= ref.isoCodes[c] -%>
</option>
<% }); -%>
I am using node v6.10.3.
Your export that you say "works" is impossible to use because the code that loads your module would have no idea when the exports.countries value has actually been set because it is set in an asynchronous call that finishes some indeterminate time in the future. In addition, you have no means of handling any error in that function.
The modern way of doing this would be to export a function that, when called, returns a promise that resolves to the data (or rejects with an error). The code loading your module, then calls that exported function, gets the promise, uses .then() on the promise and uses the data in the .then() handler. That could look something like this:
function getCountries() {
return new Promise(function(resolve, reject) {
db.collection.distinct('c', {}, function(err, data) {
if (err) {
reject(err);
} else {
resolve(data);
}
});
}
}
module.exports.getCountries = getCountries;
The caller would then do something like this:
const myModule = require('myModule');
myModule.getCountries().then(function(countries) {
console.log(countries);
// use country data here
}).catch(function(err) {
// deal with error here
});
Most databases for node.js these days have some level of promise support built in so you often don't have to create your own promise wrapper around your DB functions like was shown above, but rather can use a promise directly returned from the DB engine. How that works is specific to the particular database/version you are using.
If you are using the list of countries in a template rendering operation, then you will have to fetch the list of countries (and any other data needed for the template rendering) and only call res.render() when all the data has been successfully retrieved. This probably also leads to what you should do when there's an error retrieving the necessary data. In that case, you would typically respond with a 5xx error code for the page request and may want to render some sort of error page that informs the end-user about the error.
I am using Node 6.10 so I don't have async and await but if I did they would help me here:
https://developers.google.com/web/fundamentals/getting-started/primers/async-functions
Instead I can use the asyncawait library:
https://github.com/yortus/asyncawait
Code looks like this:
var async = require('asyncawait/async');
var await = require('asyncawait/await');
const db = require('_/db');
function getDistinctValues(key) {
return new Promise((resolve, reject) => {
db.collection.distinct(key, {}, function(err, data) {
if (err) {
throw Error('mongodb problem - unable to load distinct values');
} else {
resolve(data);
}
});
});
};
async(function () {
exports.countries = await(getDistinctValues('c'));
exports.categories = await(getDistinctValues('w'));
})();
Now I can be sure ref.countries and ref.categories are available after this is loaded.
I'm a bit new to RxJS and it is kicking my ass, so I hope someone can help!
I'm using RxJS(5) on my express server to handle behaviour where I have to save a bunch of Document objects and then email each of them to their recepients. The code in my documents/create endpoint looks like this:
// Each element in this stream is an array of `Document` model objects: [<Document>, <Document>, <Document>]
const saveDocs$ = Observable.fromPromise(Document.handleCreateBatch(docs, companyId, userId));
const saveThenEmailDocs$ = saveDocs$
.switchMap((docs) => sendInitialEmails$$(docs, user))
.do(x => {
// Here x is the `Document` model object
debugger;
});
// First saves all the docs, and then begins to email them all.
// The reason we want to save them all first is because, if an email fails,
// we can still ensure that the document is saved
saveThenEmailDocs$
.subscribe(
(doc) => {
// This never hits
},
(err) => {},
() => {
// This hits immediately.. Why though?
}
);
The sendInitialEmails$$ function returns an Observable and looks like this:
sendInitialEmails$$ (docs, fromUser) {
return Rx.Observable.create((observer) => {
// Emails each document to their recepients
docs.forEach((doc) => {
mailer.send({...}, (err) => {
if (err) {
observer.error(err);
} else {
observer.next(doc);
}
});
});
// When all the docs have finished sending, complete the
// stream
observer.complete();
});
});
The problem is that when I subscribe to saveThenEmailDocs$, my next handler is never called, and it goes straight to complete. I have no idea why... Inversely if I remove the observer.complete() call from sendInitialEmails$$, the next handler is called every time and the complete handler in subscribe is never called.
Why isn't the expected behaviour of next next complete happening, instead it's one or the other... Am I missing something?
I can only assume that mailer.send is an asynchronous call.
Your observer.complete() is called when all the asynchronous calls have been launched, but before any of them could complete.
In such cases I would either make an stream of observable values from the docs array rather than wrap it like this.
Or, if you would like to wrap it manually into an observable, I suggest you look into the library async and use
async.each(docs, function(doc, callback) {...}, function finalized(err){...})
I'm coming from a java background so a bit of a newbie on Javascript conventions needed for Lambda.
I've got a lambda function which is meant to do several AWS tasks in a particular order, depending on the result of the previous task.
Given that each task reports its results asynchronously, I'm wondering if the right way make sure they all happen in the right sequence, and the results of one operation are available to the invocation of the next function.
It seems like I have to invoike each function in the callback of the prior function, but seems like that will some kind of deep nesting and wondering if that is the proper way to do this.
For example on of these functions requires a DynamoDB getItem, following by a call to SNS to get an endpoint, followed by a SNS call to send a message, followed by a DynamoDB write.
What's the right way to do that in lambda javascript, accounting for all that asynchronicity?
I like the answer from #jonathanbaraldi but I think it would be better if you manage control flow with Promises. The Q library has some convenience functions like nbind which help convert node style callback API's like the aws-sdk into promises.
So in this example I'll send an email, and then as soon as the email response comes back I'll send a second email. This is essentially what was asked, calling multiple services in sequence. I'm using the then method of promises to manage that in a vertically readable way. Also using catch to handle errors. I think it reads much better just simply nesting callback functions.
var Q = require('q');
var AWS = require('aws-sdk');
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';
// Use a promised version of sendEmail
var ses = new AWS.SES({apiVersion: '2010-12-01'});
var sendEmail = Q.nbind(ses.sendEmail, ses);
exports.handler = function(event, context) {
console.log(event.nome);
console.log(event.email);
console.log(event.mensagem);
var nome = event.nome;
var email = event.email;
var mensagem = event.mensagem;
var to = ['email#company.com.br'];
var from = 'site#company.com.br';
// Send email
mensagem = ""+nome+"||"+email+"||"+mensagem+"";
console.log(mensagem);
var params = {
Source: from,
Destination: { ToAddresses: to },
Message: {
Subject: {
Data: 'Form contact our Site'
},
Body: {
Text: {
Data: mensagem,
}
}
};
// Here is the white-meat of the program right here.
sendEmail(params)
.then(sendAnotherEmail)
.then(success)
.catch(logErrors);
function sendAnotherEmail(data) {
console.log("FIRST EMAIL SENT="+data);
// send a second one.
return sendEmail(params);
}
function logErrors(err) {
console.log("ERROR="+err, err.stack);
context.done();
}
function success(data) {
console.log("SECOND EMAIL SENT="+data);
context.done();
}
}
Short answer:
Use Async / Await — and Call the AWS service (SNS for example) with a .promise() extension to tell aws-sdk to use the promise-ified version of that service function instead of the call back based version.
Since you want to execute them in a specific order you can use Async / Await assuming that the parent function you are calling them from is itself async.
For example:
let snsResult = await sns.publish({
Message: snsPayload,
MessageStructure: 'json',
TargetArn: endPointArn
}, async function (err, data) {
if (err) {
console.log("SNS Push Failed:");
console.log(err.stack);
return;
}
console.log('SNS push suceeded: ' + data);
return data;
}).promise();
The important part is the .promise() on the end there. Full docs on using aws-sdk in an async / promise based manner can be found here: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html
In order to run another aws-sdk task you would similarly add await and the .promise() extension to that function (assuming that is available).
For anyone who runs into this thread and is actually looking to simply push promises to an array and wait for that WHOLE array to finish (without regard to which promise executes first) I ended up with something like this:
let snsPromises = [] // declare array to hold promises
let snsResult = await sns.publish({
Message: snsPayload,
MessageStructure: 'json',
TargetArn: endPointArn
}, async function (err, data) {
if (err) {
console.log("Search Push Failed:");
console.log(err.stack);
return;
}
console.log('Search push suceeded: ' + data);
return data;
}).promise();
snsPromises.push(snsResult)
await Promise.all(snsPromises)
Hope that helps someone that randomly stumbles on this via google like I did!
I don't know Lambda but you should look into the node async library as a way to sequence asynchronous functions.
async has made my life a lot easier and my code much more orderly without the deep nesting issue you mentioned in your question.
Typical async code might look like:
async.waterfall([
function doTheFirstThing(callback) {
db.somecollection.find({}).toArray(callback);
},
function useresult(dbFindResult, callback) {
do some other stuff (could be synch or async)
etc etc etc
callback(null);
],
function (err) {
//this last function runs anytime any callback has an error, or if no error
// then when the last function in the array above invokes callback.
if (err) { sendForTheCodeDoctor(); }
});
Have a look at the async doco at the link above. There are many useful functions for serial, parallel, waterfall, and many more. Async is actively maintained and seems very reliable.
good luck!
A very specific solution that comes to mind is cascading Lambda calls. For example, you could write:
A Lambda function gets something from DynamoDB, then invokes…
…a Lambda function that calls SNS to get an endpoint, then invokes…
…a Lambda function that sends a message through SNS, then invokes…
…a Lambda function that writes to DynamoDB
All of those functions take the output from the previous function as input. This is of course very fine-grained, and you might decide to group certain calls. Doing it this way avoids callback hell in your JS code at least.
(As a side note, I'm not sure how well DynamoDB integrates with Lambda. AWS might emit change events for records that can then be processed through Lambda.)
Just saw this old thread. Note that future versions of JS will improve that. Take a look at the ES2017 async/await syntax that streamlines an async nested callback mess into a clean sync like code.
Now there are some polyfills that can provide you this functionality based on ES2016 syntax.
As a last FYI - AWS Lambda now supports .Net Core which provides this clean async syntax out of the box.
I would like to offer the following solution, which simply creates a nested function structure.
// start with the last action
var next = function() { context.succeed(); };
// for every new function, pass it the old one
next = (function(param1, param2, next) {
return function() { serviceCall(param1, param2, next); };
})("x", "y", next);
What this does is to copy all of the variables for the function call you want to make, then nests them inside the previous call. You'll want to schedule your events backwards. This is really just the same as making a pyramid of callbacks, but works when you don't know ahead of time the structure or quantity of function calls. You have to wrap the function in a closure so that the correct value is copied over.
In this way I am able to sequence AWS service calls such that they go 1-2-3 and end with closing the context. Presumably you could also structure it as a stack instead of this pseudo-recursion.
I found this article which seems to have the answer in native javascript.
Five patterns to help you tame asynchronis javascript.
By default Javascript is asynchronous.
So, everything that you have to do, it's not to use those libraries, you can, but there's simple ways to solve this. In this code, I sent the email, with the data that comes from the event, but if you want, you just need to add more functions inside functions.
What is important is the place where your context.done(); is going to be, he is going to end your Lambda function. You need to put him in the end of the last function.
var AWS = require('aws-sdk');
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';
var ses = new AWS.SES({apiVersion: '2010-12-01'});
exports.handler = function(event, context) {
console.log(event.nome);
console.log(event.email);
console.log(event.mensagem);
nome = event.nome;
email = event.email;
mensagem = event.mensagem;
var to = ['email#company.com.br'];
var from = 'site#company.com.br';
// Send email
mensagem = ""+nome+"||"+email+"||"+mensagem+"";
console.log(mensagem);
ses.sendEmail( {
Source: from,
Destination: { ToAddresses: to },
Message: {
Subject: {
Data: 'Form contact our Site'
},
Body: {
Text: {
Data: mensagem,
}
}
}
},
function(err, data) {
if (err) {
console.log("ERROR="+err, err.stack);
context.done();
} else {
console.log("EMAIL SENT="+data);
context.done();
}
});
}
I'm using Mongoose with Node.js and have the following code that will call the callback after all the save() calls has finished. However, I feel that this is a very dirty way of doing it and would like to see the proper way to get this done.
function setup(callback) {
// Clear the DB and load fixtures
Account.remove({}, addFixtureData);
function addFixtureData() {
// Load the fixtures
fs.readFile('./fixtures/account.json', 'utf8', function(err, data) {
if (err) { throw err; }
var jsonData = JSON.parse(data);
var count = 0;
jsonData.forEach(function(json) {
count++;
var account = new Account(json);
account.save(function(err) {
if (err) { throw err; }
if (--count == 0 && callback) callback();
});
});
});
}
}
You can clean up the code a bit by using a library like async or Step.
Also, I've written a small module that handles loading fixtures for you, so you just do:
var fixtures = require('./mongoose-fixtures');
fixtures.load('./fixtures/account.json', function(err) {
//Fixtures loaded, you're ready to go
};
Github:
https://github.com/powmedia/mongoose-fixtures
It will also load a directory of fixture files, or objects.
I did a talk about common asyncronous patterns (serial and parallel) and ways to solve them:
https://github.com/masylum/i-love-async
I hope its useful.
I've recently created simpler abstraction called wait.for to call async functions in sync mode (based on Fibers). It's at an early stage but works. It is at:
https://github.com/luciotato/waitfor
Using wait.for, you can call any standard nodejs async function, as if it were a sync function, without blocking node's event loop. You can code sequentially when you need it.
using wait.for your code will be:
//in a fiber
function setup(callback) {
// Clear the DB and load fixtures
wait.for(Account.remove,{});
// Load the fixtures
var data = wait.for(fs.readFile,'./fixtures/account.json', 'utf8');
var jsonData = JSON.parse(data);
jsonData.forEach(function(json) {
var account = new Account(json);
wait.forMethod(account,'save');
}
callback();
}
That's actually the proper way of doing it, more or less. What you're doing there is a parallel loop. You can abstract it into it's own "async parallel foreach" function if you want (and many do), but that's really the only way of doing a parallel loop.
Depending on what you intended, one thing that could be done differently is the error handling. Because you're throwing, if there's a single error, that callback will never get executed (count won't be decremented). So it might be better to do:
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
And handle the error in the callback. It's better node-convention-wise.
I would also change another thing to save you the trouble of incrementing count on every iteration:
var jsonData = JSON.parse(data)
, count = jsonData.length;
jsonData.forEach(function(json) {
var account = new Account(json);
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
});
If you are already using underscore.js anywhere in your project, you can leverage the after method. You need to know how many async calls will be out there in advance, but aside from that it's a pretty elegant solution.