Before describing the issue in a very complex way, I would like to know how to execute multiple dependent awaits in a return Promise one after another without getting new data in my return Promise block. In other words, I just want my try-block to be executed as one statement.
const handler = (payload) => {
return new Promise(async (resolve, reject) => {
try {
const exists = await getRedis(payload)
if(exists === null) {
await setRedis(payload)
await write2Mongo(payload)
resolve()
} else {
resolve()
}
} catch (err) {
reject(err)
}
});
};
In concrete terms, it's about RabbitMQ ("amqplib": "^0.8.0"), where the payloads fly in. These I want to check first if they are known by the system. If not, I want to set them in Redis ("async-redis": "^2.0.0") and then write them to MongoDB ("mongoose": "^6.0.9"). Since I get a lot of messages from RabbitMQ, it works fine at first and then I get a "duplicate key error" from Mongo. This is because my first getRedis returns a null. While writing the data into Redis and MongoDB, a second message comes into my block and gets a "null" value from getRedis, because the message was not yet set via setRedis.
As I read, this is an antipattern with bad error handling. But the corresponding posts have unfortunately not solved my problem.
Can you please help me.
in senario that you describe, you want a queue that you can process it in series
let payloads = [];
const handler = payload => payloads.push(payload);
;(async function insertDistincPayloads() {
for (let i=0; i < payloads.length; i++) {
const exists = await getRedis(payload)
if(exists === null) {
await setRedis(payload)
await write2Mongo(payload)
}
}
payloads = []
setTimeout(insertDistincPayloads, 100); // loop continuously with a small delay
})();
sorry for my bad english :-)
I need to retrieve information from a Firestore Document when another document is created. When I try to do this I get hit with an error about the function not being async. It has been so long since I used javascript I am basically a novice again and have no idea how to fix this.
ok, so I am using Firebase Cloud Functions and the function in question is a Firestore .onCreate() trigger.
When the function is triggered I set a sender variable (which is the document ID from a different collection that I need to retrieve)
then I try to get the document as per the documentation.
The function ends up like this:
exports.pushFriendRequestNotification = functions.firestore.document('friends/{friendID}')
.onCreate((snap, context) => {
// when friend request is created
data = doc.data()//get request data
sender = data["sender"]//get request sender from data
const requestRef = db.collection('User').doc(sender);
const doc = await requestRef.get();//get user data of sender
if (!doc.exists) {
console.log('No such document!');
} else {
console.log('Document data:', doc.data());
}
});
when I run this in the emulator I get this error:
const doc = await requestRef.get();//get user data of sender
^^^^^
SyntaxError: await is only valid in async functions and the top level bodies of modules
I have absolutely no idea where to go from here.
Can anyone help me with this?
Thanks
The await keyword is valid only in an async function.
exports.pushFriendRequestNotification = functions.firestore.document('friends/{friendID}')
.onCreate(async (snap, context) => {
// ^^^^^
})
If you are (or need to) use synchronous function then you would have to use promise chaining.
exports.pushFriendRequestNotification = functions.firestore.document('friends/{friendID}')
.onCreate((snap, context) => {
return requestRef.get().then((snapshot) => {
if (snapshot.exists) { ... }
})
})
Apart from that, the order of variables/statements looks incorrect. With the current code (as in original question), you may end up getting an error: "Cannot access 'doc' before initialization" Try refactoring it like this:
exports.pushFriendRequestNotification = functions.firestore.document('friends/{friendID}')
.onCreate(async (snap, context) => {
// accessing data from newly created doc
const newDocData = snap.data()
// const sender = "" // ??
const requestRef = db.collection('User').doc(sender);
const doc = await requestRef.get();//get user data of sender
if (!doc.exists) {
console.log('No such document!');
} else {
console.log('Document data:', doc.data());
}
})
Where is the sender coming from? I've just commented it above but if the sender is present in new document then you can access it by: const sender = newDocData.sender
If your using await you have to specify that function is asynchronous. Otherwise it will throw error.
exports.pushFriendRequestNotification = functions.firestore.document('friends/{friendID}').onCreate(async (snap, context) => {
// when friend request is created
data = doc.data()//get request data
sender = data["sender"]//get request sender from data
const requestRef = db.collection('User').doc(sender);
const doc = await requestRef.get();//get user data of sender
if (!doc.exists) {
console.log('No such document!');
} else {
console.log('Document data:', doc.data());
}
});
Yet some of your references is unknown to us. Maybe this code is not completed.
The main point is you need to understand when you can access async/await or Promise
All await methods must be inside an async block or be handled in an async manor using .then() promises
in this case, the parent function is on this line .onCreate((snap, context) => {
simply inserting an async at the start of the variables will upgrade the arrow function to an async arrow function
.onCreate(async (snap, context) => {
Im on a big problem, I need to do an application that gets trello content as soon as i can but i dont know why my for isnt working as it should do. when i the output of this should be the card id and when the function is called it should show the member. I dont know why, but when the 'miembro' function is called, it delays and it is shown after the second id, so its delayed a lot and i need them to show one under the other. I appreciate a quick answer, thank you!
const trelloKB = require("trello-kb");
const fetch = require('node-fetch');
// Replace this by the application key of your Trello account
var appKey = '51501902fff527d305686a29d6d61cfa';
// Replace this by a valid authorization token
var authToken = '9828f5f03073ae52ffdae77bdf49c939df8a315b169cb81aeb42a3d43d0f9e21';
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
function miembros (id){
fetch('https://api.trello.com/1/cards/'+id+'/members?key=51501902fff527d305686a29d6d61cfa&token=9828f5f03073ae52ffdae77bdf49c939df8a315b169cb81aeb42a3d43d0f9e21&fields=fullName', {
method: 'GET'
})
.then(response => {
setTimeout(function() {
return(response.text());
}, 3000);
})
.then(text => console.log(text))
.catch(err => console.error(err));
}
trelloKB.get(appKey, authToken, '33CP31Sf').then(
function (cards) {
// Print the title of each card
var ms = 3000;
for(i=0; i<2; i++){
var card = cards[0];
var id = card.id;
var titleCard = card.title;
console.log(id);
miembros(id);
}
},
);
I think you should learn about synchronous and asynchronous concept
You need to use async-await and return a promise in your function miembros()
read this async await and promise.
this is my example
const cards =[{id:1,title:'kimiwo'},{id:2,title:'namae wa'},{id:3,title:'udiiiin'}];
yourname(cards);
async function yourname (cards) {
for(let card of cards){
console.log(`id :${card.id},text:${card.title}`);
let result = await(await miembros(card.id)).text();
console.log(result);
}
}
function miembros(id){
return fetch('https://api.trello.com/1/cards/'+id+'/members?key=51501902fff527d305686a29d6d61cfa&token=9828f5f03073ae52ffdae77bdf49c939df8a315b169cb81aeb42a3d43d0f9e21&fields=fullName')
}
you can see the result here
*Edit
fetch returns promise, so you can just return fetch and wrap your function with async-await
It’s delayed because you need to use a sync await or .then
You need to get the first Id first, then do a .then to get the second ID through the function call.
Also, you shouldn’t show you API keys, they’re supposed to be private lol
I'm trying to update a tool that was created a while ago which uses nodejs (I am not a JS developer, so I'm trying to piece the code together) and am getting stuck at the last hurdle.
The new functionality will take in a swagger .json definition, compare the endpoints against the matching API Gateway on the AWS Service, using the 'aws-sdk' SDK for JS and then updates the Gateway accordingly.
The code runs fine on a small definition file (about 15 endpoints) but as soon as I give it a bigger one, I start getting tons of TooManyRequestsException errors.
I understand that this is due to my calls to the API Gateway service being too quick and a delay / pause is needed. This is where I am stuck
I have tried adding;
a delay() to each promise being returned
running a setTimeout() in each promise
adding a delay to the Promise.all and Promise.mapSeries
Currently my code loops through each endpoint within the definition and then adds the response of each promise to a promise array:
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath));
Once the loop is finished I run this:
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
I have tried the same with a mapSeries (no luck).
It looks like the functions within the (getMethodResponse promise) are run immediately and hence, no matter what type of delay I add they all still just execute. My suspicious is that the I need to make (getMethodResponse) return a function and then use mapSeries but I cant get this to work either.
Code I tried:
Wrapped the getMethodResponse in this:
return function(value){}
Then added this after the loop (and within the loop - no difference):
Promise.mapSeries(function (promises) {
return 'a'();
}).then(function (results) {
console.log('result', results);
});
Also tried many other suggestions:
Here
Here
Any suggestions please?
EDIT
As request, some additional code to try pin-point the issue.
The code currently working with a small set of endpoints (within the Swagger file):
module.exports = (apiName, externalUrl) => {
return getSwaggerFromHttp(externalUrl)
.then((swagger) => {
let paths = swagger.paths;
let resourcePath = '';
let resourceMethod = '';
let promises = [];
_.each(paths, function (value, key) {
resourcePath = key;
_.each(value, function (value, key) {
resourceMethod = key;
let statusList = [];
_.each(value.responses, function (value, key) {
if (key >= 200 && key <= 204) {
statusList.push(key)
}
});
_.each(statusList, function (value, key) { //Only for 200-201 range
//Working with small set
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath))
});
});
});
//Working with small set
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
})
.catch((err) => {
winston.error(err);
});
};
I have since tried adding this in place of the return Promise.all():
Promise.map(promises, function() {
// Promise.map awaits for returned promises as well.
console.log('X');
},{concurrency: 5})
.then(function() {
return console.log("y");
});
Results of this spits out something like this (it's the same for each endpoint, there are many):
Error: TooManyRequestsException: Too Many Requests
X
Error: TooManyRequestsException: Too Many Requests
X
Error: TooManyRequestsException: Too Many Requests
The AWS SDK is being called 3 times within each promise, the functions of which are (get initiated from the getMethodResponse() function):
apigateway.getRestApisAsync()
return apigateway.getResourcesAsync(resourceParams)
apigateway.getMethodAsync(params, function (err, data) {}
The typical AWS SDK documentation state that this is typical behaviour for when too many consecutive calls are made (too fast). I've had a similar issue in the past which was resolved by simply adding a .delay(500) into the code being called;
Something like:
return apigateway.updateModelAsync(updateModelParams)
.tap(() => logger.verbose(`Updated model ${updatedModel.name}`))
.tap(() => bar.tick())
.delay(500)
EDIT #2
I thought in the name of thorough-ness, to include my entire .js file.
'use strict';
const AWS = require('aws-sdk');
let apigateway, lambda;
const Promise = require('bluebird');
const R = require('ramda');
const logger = require('../logger');
const config = require('../config/default');
const helpers = require('../library/helpers');
const winston = require('winston');
const request = require('request');
const _ = require('lodash');
const region = 'ap-southeast-2';
const methodLib = require('../aws/methods');
const emitter = require('../library/emitter');
emitter.on('updateRegion', (region) => {
region = region;
AWS.config.update({ region: region });
apigateway = new AWS.APIGateway({ apiVersion: '2015-07-09' });
Promise.promisifyAll(apigateway);
});
function getSwaggerFromHttp(externalUrl) {
return new Promise((resolve, reject) => {
request.get({
url: externalUrl,
header: {
"content-type": "application/json"
}
}, (err, res, body) => {
if (err) {
winston.error(err);
reject(err);
}
let result = JSON.parse(body);
resolve(result);
})
});
}
/*
Deletes a method response
*/
function deleteMethodResponse(httpMethod, resourceId, restApiId, statusCode, resourcePath) {
let methodResponseParams = {
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: restApiId,
statusCode: statusCode
};
return apigateway.deleteMethodResponseAsync(methodResponseParams)
.delay(1200)
.tap(() => logger.verbose(`Method response ${statusCode} deleted for path: ${resourcePath}`))
.error((e) => {
return console.log(`Error deleting Method Response ${httpMethod} not found on resource path: ${resourcePath} (resourceId: ${resourceId})`); // an error occurred
logger.error('Error: ' + e.stack)
});
}
/*
Deletes an integration response
*/
function deleteIntegrationResponse(httpMethod, resourceId, restApiId, statusCode, resourcePath) {
let methodResponseParams = {
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: restApiId,
statusCode: statusCode
};
return apigateway.deleteIntegrationResponseAsync(methodResponseParams)
.delay(1200)
.tap(() => logger.verbose(`Integration response ${statusCode} deleted for path ${resourcePath}`))
.error((e) => {
return console.log(`Error deleting Integration Response ${httpMethod} not found on resource path: ${resourcePath} (resourceId: ${resourceId})`); // an error occurred
logger.error('Error: ' + e.stack)
});
}
/*
Get Resource
*/
function getMethodResponse(httpMethod, statusCode, apiName, resourcePath) {
let params = {
httpMethod: httpMethod.toUpperCase(),
resourceId: '',
restApiId: ''
}
return getResourceDetails(apiName, resourcePath)
.error((e) => {
logger.unimportant('Error: ' + e.stack)
})
.then((result) => {
//Only run the comparrison of models if the resourceId (from the url passed in) is found within the AWS Gateway
if (result) {
params.resourceId = result.resourceId
params.restApiId = result.apiId
var awsMethodResponses = [];
try {
apigateway.getMethodAsync(params, function (err, data) {
if (err) {
if (err.statusCode == 404) {
return console.log(`Method ${params.httpMethod} not found on resource path: ${resourcePath} (resourceId: ${params.resourceId})`); // an error occurred
}
console.log(err, err.stack); // an error occurred
}
else {
if (data) {
_.each(data.methodResponses, function (value, key) {
if (key >= 200 && key <= 204) {
awsMethodResponses.push(key)
}
});
awsMethodResponses = _.pull(awsMethodResponses, statusCode); //List of items not found within the Gateway - to be removed.
_.each(awsMethodResponses, function (value, key) {
if (data.methodResponses[value].responseModels) {
var existingModel = data.methodResponses[value].responseModels['application/json']; //Check if there is currently a model attached to the resource / method about to be deleted
methodLib.updateResponseAssociation(params.httpMethod, params.resourceId, params.restApiId, statusCode, existingModel); //Associate this model to the same resource / method, under the new response status
}
deleteMethodResponse(params.httpMethod, params.resourceId, params.restApiId, value, resourcePath)
.delay(1200)
.done();
deleteIntegrationResponse(params.httpMethod, params.resourceId, params.restApiId, value, resourcePath)
.delay(1200)
.done();
})
}
}
})
.catch(err => {
console.log(`Error: ${err}`);
});
}
catch (e) {
console.log(`getMethodAsync failed, Error: ${e}`);
}
}
})
};
function getResourceDetails(apiName, resourcePath) {
let resourceExpr = new RegExp(resourcePath + '$', 'i');
let result = {
apiId: '',
resourceId: '',
path: ''
}
return helpers.apiByName(apiName, AWS.config.region)
.delay(1200)
.then(apiId => {
result.apiId = apiId;
let resourceParams = {
restApiId: apiId,
limit: config.awsGetResourceLimit,
};
return apigateway.getResourcesAsync(resourceParams)
})
.then(R.prop('items'))
.filter(R.pipe(R.prop('path'), R.test(resourceExpr)))
.tap(helpers.handleNotFound('resource'))
.then(R.head)
.then([R.prop('path'), R.prop('id')])
.then(returnedObj => {
if (returnedObj.id) {
result.path = returnedObj.path;
result.resourceId = returnedObj.id;
logger.unimportant(`ApiId: ${result.apiId} | ResourceId: ${result.resourceId} | Path: ${result.path}`);
return result;
}
})
.catch(err => {
console.log(`Error: ${err} on API: ${apiName} Resource: ${resourcePath}`);
});
};
function delay(t) {
return new Promise(function(resolve) {
setTimeout(resolve, t)
});
}
module.exports = (apiName, externalUrl) => {
return getSwaggerFromHttp(externalUrl)
.then((swagger) => {
let paths = swagger.paths;
let resourcePath = '';
let resourceMethod = '';
let promises = [];
_.each(paths, function (value, key) {
resourcePath = key;
_.each(value, function (value, key) {
resourceMethod = key;
let statusList = [];
_.each(value.responses, function (value, key) {
if (key >= 200 && key <= 204) {
statusList.push(key)
}
});
_.each(statusList, function (value, key) { //Only for 200-201 range
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath))
});
});
});
//Working with small set
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
})
.catch((err) => {
winston.error(err);
});
};
You apparently have a misunderstanding about what Promise.all() and Promise.map() do.
All Promise.all() does is keep track of a whole array of promises to tell you when the async operations they represent are all done (or one returns an error). When you pass it an array of promises (as you are doing), ALL those async operations have already been started in parallel. So, if you're trying to limit how many async operations are in flight at the same time, it's already too late at that point. So, Promise.all() by itself won't help you control how many are running at once in any way.
I've also noticed since, that it seems this line promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath)) is actually executing promises and not simply adding them to the array. Seems like the last Promise.all() doesn't actually do much.
Yep, when you execute promises.push(getMethodResponse()), you are calling getMethodResponse() immediately right then. That starts the async operation immediately. That function then returns a promise and Promise.all() will monitor that promise (along with all the other ones you put in the array) to tell you when they are all done. That's all Promise.all() does. It monitors operations you've already started. To keep the max number of requests in flight at the same time below some threshold, you have to NOT START the async operations all at once like you are doing. Promise.all() does not do that for you.
For Bluebird's Promise.map() to help you at all, you have to pass it an array of DATA, not promises. When you pass it an array of promises that represent async operations that you've already started, it can do no more than Promise.all() can do. But, if you pass it an array of data and a callback function that can then initiate an async operation for each element of data in the array, THEN it can help you when you use the concurrency option.
Your code is pretty complex so I will illustrate with a simple web scraper that wants to read a large list of URLs, but for memory considerations, only process 20 at a time.
const rp = require('request-promise');
let urls = [...]; // large array of URLs to process
Promise.map(urls, function(url) {
return rp(url).then(function(data) {
// process scraped data here
return someValue;
});
}, {concurrency: 20}).then(function(results) {
// process array of results here
}).catch(function(err) {
// error here
});
In this example, hopefully you can see that an array of data items are being passed into Promise.map() (not an array of promises). This, then allows Promise.map() to manage how/when the array is processed and, in this case, it will use the concurrency: 20 setting to make sure that no more than 20 requests are in flight at the same time.
Your effort to use Promise.map() was passing an array of promises, which does not help you since the promises represent async operations that have already been started:
Promise.map(promises, function() {
...
});
Then, in addition, you really need to figure out what exactly causes the TooManyRequestsException error by either reading documentation on the target API that exhibits this or by doing a whole bunch of testing because there can be a variety of things that might cause this and without knowing exactly what you need to control, it just takes a lot of wild guesses to try to figure out what might work. The most common things that an API might detect are:
Simultaneous requests from the same account or source.
Requests per unit of time from the same account or source (such as request per second).
The concurrency operation in Promise.map() will easily help you with the first option, but will not necessarily help you with the second option as you can limit to a low number of simultaneous requests and still exceed a requests per second limit. The second needs some actual time control. Inserting delay() statements will sometimes work, but even that is not a very direct method of managing it and will either lead to inconsistent control (something that works sometimes, but not other times) or sub-optimal control (limiting yourself to something far below what you can actually use).
To manage to a request per second limit, you need some actual time control with a rate limiting library or actual rate limiting logic in your own code.
Here's an example of a scheme for limiting the number of requests per second you are making: How to Manage Requests to Stay Below Rate Limiting.
tl;dr I need to test that my method adds a row to a spreadsheet on successful load of a Google spreadsheet.
saveDataToGoogleSpreadSheet(conversationData){
return new Promise((resolve, reject) => {
Spreadsheet.load(this.getGoogleAPISettings(), (err, spreadsheet) => {
if (err) {
return reject(err);
}
return spreadsheet.receive((receivedError, rows, info) => {
if (receivedError) {
return reject(receivedError);
}
const rowData = this.getSpreadsheetRowData(conversationData, info.totalRows);
spreadsheet.add(rowData);
return spreadsheet.send(sendError => (sendError ? reject(sendError) : resolve()));
});
});
});
}
I tested the case one and case two of the function (the two first errors) but I couldn't do it for the last one, the case of success where we an add a row to a spreadsheet.
I need some help with the structure of the test, or a hint on how could my test be.
Edit: how the previous tests were made
it('should add a row to a Google Spreadsheet', (done) => {
nock('https://spreadsheets.google.com')
.post('/feeds/cells/1ZOd7Sysc-JNa-D5AHb7ZJkwBRMBGaeKpzIwEl7B8RbQ/1/private/full/batch')
.replyWithError({ message: 'abcde' });
api.saveDataToGoogleSpreadSheet({ data: 'some data' })
.then(() => done(new Error('should not have made the call')))
.catch((err) => {
expect(err).to.equal('Error Reading Spreadsheet');
done();
});
}).timeout(4000);
It is hard to tell what is the problem with the test from the little code you have, and no background on what the various objects are and how they come to be, so I will just assume that Spreadsheet is an object that is created through a library you require, and other than that, all other objects are created by the module. i.e. I assume you somewhere have a line resembling something like this:
const Spreadsheet = require('some-google-docs-spreadsheet-lib');
That means one problem is finding out how to control the Spreadsheet object so we can stub out its behavior.
Just to start you out, you might get some good pointers on general code and test structure for easy testing from these two answers, as they cover the two most relevant techniques: dependency injection and exploiting link seams.
Mocking Redis Constructor with Sinon
How to test an ES6 class that needs jquery?
For all I know, you might already utilize one of these techniques, as you say you have been able to test the two error situations. But maybe you have not really been unit testing and done the actual network calls to the service instead (which is more of an integration test)? Anyway, I'll assume no more than what I wrote above and show you how to do the testing using proxyquire:
const assert = require('assert');
const dummy = ()=> {};
const SpreadSheetStubLibrary = { load: dummy };
const MyClassToTest = proxyquire('../src/my-module', {
'some-google-docs-spreadsheet-lib': SpreadSheetStubLibrary
})
const config = {};
const conversationData = {};
let stubSetup;
let spreadsheet;
let myObj;
function setupStubs() {
stubSetup = stubSpreadsheetLoadFunction();
spreadsheet = stubSetup.spreadsheet;
SpreadSheetStubLibrary.load = stubSetup.load;
myObj = new MyClassToTest(config);
conversationData = {};
};
function createSpreadsheetStubObj(){
return {
receive: sinon.stub(),
add: sinon.stub(),
send: sinon.stub()
}
}
function stubSpreadsheetLoadFunction(){
const spreadsheet = createSpreadsheetStubObj();
return {
load: (settings, cb) => cb(null, spreadsheet),
spreadSheetStubObj: spreadsheet
};
}
it('should add a row to the spreadsheet on successful load', () => {
// Arrange
setupStubs();
const rowData = { foo: 1, bar: 2};
spreadsheet.receive.yields(); // calls any callback given
myObj.getSpreadsheetRowData = () => rowData; // see lines below
// if you want to use the real getSpreadsheetRowData, uncomment these lines
//const rows = []; // not used in the method?
//const info = { totalRows : 100 };
//spreadsheet.receive.yields(null, rows, info);
// Act
return myObj.saveDataToGoogleSpreadSheet(conversationData).then(()=>{
// Assert
assert(spreadsheet.add.calledOnce);
assert(spreadsheet.add.calledWith(rowData));
});
});
it('should add a row to the spreadsheet on successful load', () => {
// reuse the above
});
See the Sinon docs for the stub API.
Disclosure: I am part of the Sinon maintainer team.