I found quite unusual behavior of Promise.all().finally() - look like it returning data before map() was applied to this.
1. Data received from database.
2. Making call to Google Maps API inside map(), applied to the data retrieved from database and adding to the object property "Distance" with result from Google API call.
3. Return data in Promise.all() - data received without new property.
I can't get how this even possible?
public static get = async (req: Request, res: Response) => {
const latitude = req.query.lat;
const longitude = req.query.long;
const pool = await new sql.ConnectionPool(CommonConstants.connectionString).connect();
const request = pool.request();
const result = await request.execute('SuppliersSP');
sql.close();
const rows = result.recordset.map(async (supplier) => {
const data = { origin: [latitude, longitude], destination: [supplier.Latitude, supplier.Longitude] };
const distance = await GetDistance(data) || 0;
Object.defineProperty(supplier, 'Distance', {
enumerable: true,
configurable: true,
writable: true,
value: distance
});
return supplier;
})
Promise.all(rows).finally(() => {
res.setHeader('Access-Control-Allow-Origin', '*');
res.status(200).json(rows);
});
}
Here is GetDistance function:
import { Constants } from "./constants";
const https = require('https');
export function GetDistance(coords) {
const { origin, destination } = coords;
return new Promise((resolve, reject) => {
https.get(`${Constants.GoogleMapsUrl}?origins=${origin[0]},${origin[1]}
&destinations=${destination[0]},${destination[1]}
&key=${Constants.GoogleMapsApiKey}`, (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end', () => {
const distance = JSON.parse(data);
resolve(distance.rows[0].elements[0].distance.value);
});
}).on("error", (err) => {
reject("Error: " + err.message);
});
});
}
It was solved by a change .finally() to .then().
Related
I am writing an API endpoint which searches Google Book's API to add book titles to the notebooks in my MongoDB Database based on their Id's. I have it working properly, but sometimes my endpoint's response array returns missing the last item. My console logs seem to indicate my endpoint response is being sent before the Https request has finished fetching all the data. I need to solve the endpoint sometimes returning an incomplete array. I think I may need to include an async await somewhere but I'm not sure where. Any help is much appreciated!
Here is my code:
//GET a user's books with notes
const getNotebooks = async (req, res) => {
...
const books = await Book.find({ userId }).sort({createdAt: -1}) //gets all notebooks as array
let results = []
for (let i = 0; i < books.length; i++){
console.log(i + " - " + books[i].gId)
const item = {
id: books[i].userId,
gId: books[i].gId,
title: '',
}
const request = https.request(apiUrl + books[i].gId, async (response) => {
let data = '';
response.on('data', (chunk) => {
data = data + chunk.toString();
});
response.on('end', () => {
const body = JSON.parse(data);
item.title = body.volumeInfo.title
results.push(item)
if (i == books.length - 1){
res.status(200).json(results)
}
});
})
request.on('error', (error) => {
console.log('An error', error);
res.status(400).json({error: error})
});
request.end()
}}
You can use async-await with Promises only, since Node's core https module does not have any build-in promise support, you will have to convert it first into the promise format, then you can use async-await with it, i am not familiar with the standard https module but i will make an example with your code
const getNotebooks = async (req, res) => {
//...
const books = await Book.find({ userId }).sort({createdAt: -1}) //gets all notebooks as array
let results = []
for (let i = 0; i < books.length; i++){
console.log(i + " - " + books[i].gId)
const item = {
id: books[i].userId,
gId: books[i].gId,
title: '',
}
const options = apiUrl + books[i].gId
// now you can use await to this function which return a promise
await makeRequest(options, res, item, results, books )
}
}
// Making this function into a promise outside of the scope from your getNotebooks function.
function makeRequest(options, res, item, results, books) {
return new Promise((resolve, rejects) => {
const request = https.request(options, (response) => {
let data = '';
response.on('data', (chunk) => {
data = data + chunk.toString();
});
response.on('end', () => {
const body = JSON.parse(data);
resolve(body);
item.title = body.volumeInfo.title
results.push(item)
if (i == books.length - 1){
res.status(200).json(results)
}
});
});
request.on('error', (error) => {
console.log('An error', error);
rejects(error);
res.status(400).json({error: error})
});
request.end()
});
};
Alternatively you can also use a more modern way to request data, for example with fetch which is now available in node, or even Axios library, if you don't want any dependency, fetch should be the way to go, an example if there is no mistake and assuming is a GET method request:
const getNotebooks = async (req, res) => {
//...
try {
const books = await Book.find({ userId }).sort({createdAt: -1}) //gets all notebooks as array
let results = []
for (let i = 0; i < books.length; i++) {
console.log(i + " - " + books[i].gId)
const item = {
id: books[i].userId,
gId: books[i].gId,
title: '',
}
// using fetch on node js is now possible without any package to install
const response = await fetch(apiUrl + books[i].gId)
const data = await response.json();
item.title = data.volumeInfo.title
results.push(item)
if (i == books.length - 1){
res.status(200).json(results)
}
}
} catch (error) {
res.status(400).json({error: error})
};
};
As you can see you would write 2 line code instead of 20+ line.
I receive an array of images and want to get the resolution (width, height) and store them properly to the db. How can I push the resolution const resolution correctly into the array of the respective file?
const FilesStorage = require("../models/filesStorageModel");
const sizeOf = require('image-size');
const url = require('url')
const https = require('https')
exports.uploadFiles = async (req, res) => {
const filesArray = req.files
try {
for (const file of filesArray) {
const imgUrl = file.location
const options = url.parse(imgUrl)
https.get(options, (res) => {
const chunks = []
res.on('data', (chunk) => {
chunks.push(chunk)
}).on('end', () => {
const buffer = Buffer.concat(chunks)
const resolution = sizeOf(buffer)
console.log(resolution)
})
})
const uploadFiles = new FilesStorage({
name: file.originalname,
altTag: "alt",
format: file.mimetype,
filePath: file.location,
sizeKB: (file.size / 1000),
height: "resolution.height", // How?
width: "resolution.width", // How?
})
await uploadFiles.save()
}
return res.status(200).json({
success: true,
message: 'Files successfully uploaded',
})
} catch(err) {
return res.status(500).json({
success: false,
message: `Something wen't wrong`,
})
}
}
Here's the answer you're looking for:
First, we need to consider variable scopes. When we look at the resolution constant you have defined in your handler, we can see the scope makes it local to that handler and not the parent code block.
To make it accessible in other areas of the code block, we move it above the handler scope but keep it inside your for loop so it's file-specific. With this change, we can now insert the resolution into your database upload call.
Hope this helps!
const FilesStorage = require("../models/filesStorageModel");
const sizeOf = require('image-size');
const url = require('url')
const https = require('https')
exports.uploadFiles = async(req, res) => {
const filesArray = req.files
try {
for (const file of filesArray) {
const imgUrl = file.location
const options = url.parse(imgUrl)
//Added variable outside arrow function scope
var resolution;
https.get(options, (res) => {
const chunks = []
res.on('data', (chunk) => {
chunks.push(chunk)
}).on('end', () => {
const buffer = Buffer.concat(chunks)
//Set value to variable outside function scope
resolution = sizeOf(buffer)
console.log(resolution)
})
})
const uploadFiles = new FilesStorage({
name: file.originalname,
altTag: "alt",
format: file.mimetype,
filePath: file.location,
sizeKB: (file.size / 1000),
//Get value by accessing variable above
height: resolution.height,
width: resolution.width,
})
await uploadFiles.save()
}
return res.status(200).json({
success: true,
message: 'Files successfully uploaded',
})
} catch (err) {
return res.status(500).json({
success: false,
message: `Something wen't wrong`,
})
}
}
I need to get some data from DynamoDb, using the scan() method. I have implemented some basic pagination by calling my function recursively n number of times to get the correct page.
Currently, I call my function and inside the scan() callback, if the data can be send back, I use the handler callback to return the data.
CURRENT CODE
const AWS = require('aws-sdk')
const docClient = new AWS.DynamoDB.DocumentClient()
const TABLE_NAME = process.env.TABLE_NAME
const DEFAULT_PAGE_SIZE = 500
const DEFAULT_PAGE_NUMBER = 1
const self = {
handler: (event, context, callback) => {
const {pageNumber, pageSize} = event.queryStringParameters ? event.queryStringParameters : {pageNumber: DEFAULT_PAGE_NUMBER, pageSize: DEFAULT_PAGE_SIZE}
const params = {
TableName: ORGANISATION_TYPES_TABLE_NAME,
Limit: pageSize ? pageSize : DEFAULT_PAGE_SIZE
}
return self.scan(params, pageNumber, 1, callback)
},
scan: (params, pageNumber, pageCount, callback) => {
docClient.scan(params, (err, data) => {
if (err) {
callback(null, {
statusCode: 500,
body: JSON.stringify(err)
})
};
if (data.LastEvaluatedKey && pageCount < pageNumber) {
pageCount += 1
params.ExclusiveStartKey = data.LastEvaluatedKey
self.scan(params, pageNumber, pageCount, callback)
} else {
callback(null, {
statusCode: 200,
body: JSON.stringify(data)
})
}
})
}
}
module.exports = self
The above code does work, allowing me to specify a pageSize and pageNumber query parameter.
However, I want to Promisify self.scan.
I tried the following, but it results in the response being undefined
DESIRED CODE
const AWS = require('aws-sdk')
const docClient = new AWS.DynamoDB.DocumentClient()
const ORGANISATION_TYPES_TABLE_NAME = process.env.ORGANISATION_TYPES_TABLE_NAME
const DEFAULT_PAGE_SIZE = 500
const DEFAULT_PAGE_NUMBER = 1
const self = {
handler: (event, context, callback) => {
const {pageNumber, pageSize} = event.queryStringParameters ? event.queryStringParameters : {pageNumber: DEFAULT_PAGE_NUMBER, pageSize: DEFAULT_PAGE_SIZE}
const params = {
TableName: ORGANISATION_TYPES_TABLE_NAME,
Limit: pageSize ? pageSize : DEFAULT_PAGE_SIZE
}
return self.scan(params, pageNumber, 1).then((response) => {
callback(null, {
statusCode: 200,
body: JSON.stringify(response)
})
}).catch((err) => {
callback(null, {
statusCode: 500,
body: JSON.stringify(err)
})
})
},
scan: (params, pageNumber, pageCount) => {
return new Promise((resolve, reject) => {
docClient.scan(params, (err, data) => {
if (err) {
reject(err)
};
if (data.LastEvaluatedKey && pageCount < pageNumber) {
pageCount += 1
params.ExclusiveStartKey = data.LastEvaluatedKey
self.scan(params, pageNumber, pageCount, callback)
} else {
resolve(data)
}
})
})
}
}
module.exports = self
I also tried just doing return Promise.resolve(data) inside the docClient.scan() callback, but that doesn't work either. It's as if promises cannot be resolved inside a callback?
I have recently helped someone with this problem, there's actually quite an elegant solution that we hit upon that uses the hasNextPage property on the response you get from the SDK. The key is to have your recursive function pass an array that holds your results through the recursive calls and just concat until you run out of pages and then just return the array.
const scan = async params => {
function scanRec(promise, xs) {
return promise
.then(async result => {
const response = result.$response;
const items = xs.concat(result.Items);
response.hasNextPage() ? scanRec(response.nextPage().promise(), items) : items
})
}
return scanRec(docClient.query(params).promise(), []);
}
You'd then use the function in the normal way:
const params = { /** params **/ };
scan(params).then(x => {
// ...
})
I have a koa router I need to call a api where will async return result. This means I cannot get my result immediately, the api will call my callback url when it's ok. But now I have to use it like a sync api which means I have to wait until the callback url is called.
My router like this:
router.post("/voice", async (ctx, next) => {
// call a API here
const params = {
data: "xxx",
callback_url: "http//myhost/ret_callback",
};
const req = new Request("http://xxx/api", {
method: "POST",
body: JSON.stringify(params),
});
const resp = await fetch(req);
const data = await resp.json();
// data here is not the result I want, this api just return a task id, this api will call my url back
const taskid = data.taskid;
// now I want to wait here until I got "ret_callback"
// .... wait .... wait
// "ret_callback" is called now
// get the answer in "ret_callback"
ctx.body = {
result: "ret_callback result here",
}
})
my callback url like this:
router.post("/ret_callback", async (ctx, next) => {
const params = ctx.request.body;
// taskid will tell me this answer to which question
const taskid = params.taskid;
// this is exactly what I want
const result = params.text;
ctx.body = {
code: 0,
message: "success",
};
})
So how can I make this aync api act like a sync api?
Just pass a resolve() to another function. For example, you can do it this way:
// use a map to save a lot of resolve()
const taskMap = new Map();
router.post("/voice", async (ctx, next) => {
// call a API here
const params = {
data: "xxx",
callback_url: "http//myhost/ret_callback",
};
const req = new Request("http://xxx/api", {
method: "POST",
body: JSON.stringify(params),
});
const resp = await fetch(req);
const data = await resp.json();
const result = await waitForCallback(data.taskid);
ctx.body = {
result,
} })
const waitForCallback = (taskId) => {
return new Promise((resolve, reject) => {
const task = {};
task.id = taskId;
task.onComplete = (data) => {
resolve(data);
};
task.onError = () => {
reject();
};
taskMap.set(task.id, task);
});
};
router.post("/ret_callback", async (ctx, next) => {
const params = ctx.request.body;
// taskid will tell me this answer to which question
const taskid = params.taskid;
// this is exactly what I want
const result = params.text;
// here you continue the waiting response
taskMap.get(taskid).onComplete(result);
// not forget to clean rubbish
taskMap.delete(taskid);
ctx.body = {
code: 0,
message: "success",
}; })
I didn't test it but I think it will work.
function getMovieTitles(substr) {
let movies = [];
let fdata = (page, search, totalPage) => {
let mpath = {
host: "jsonmock.hackerrank.com",
path: "/api/movies/search/?Title=" + search + "&page=" + page,
};
let raw = '';
https.get(mpath, (res) => {
res.on("data", (chunk) => {
raw += chunk;
});
res.on("end", () => {
tdata = JSON.parse(raw);
t = tdata;
totalPage(t);
});
});
}
fdata(1, substr, (t) => {
i = 1;
mdata = [];
for (i = 1; i <= parseInt(t.total_pages); i++) {
fdata(i, substr, (t) => {
t.data.forEach((v, index, arrs) => {
movies.push(v.Title);
if (index === arrs.length - 1) {
movies.sort();
if (parseInt(t.page) === parseInt(t.total_pages)) {
movies.forEach(v => {
console.log(v)
})
}
}
});
});
}
});
}
getMovieTitles("tom")
Okay so first of all, this should not be a "goal" for you. NodeJS works better as ASync.
However, let us assume that you still want it for some reason, so take a look at sync-request package on npm (there is a huge note on there that you should not this in production.
But, I hope you mean on how to make this API simpler (as in one call kinda thingy). You still need .next or await but it will be be one call anyway.
If that is the case, please comment on this answer I can write you a possible method I use myself.
How about this ?
router.post("/voice", async (ctx, next) => {
const params = {
data: "xxx",
callback_url: "http//myhost/ret_callback",
};
const req = new Request("http://xxx/api", {
method: "POST",
body: JSON.stringify(params),
});
const resp = await fetch(req);
const data = await resp.json();
// data here is not the result I want, this api just return a task id, this api will call my url back
const taskid = data.taskid;
let response = null;
try{
response = await new Promise((resolve,reject)=>{
//call your ret_callback and when it finish call resolve(with response) and if it fails, just reject(with error);
});
}catch(err){
//errors
}
// get the answer in "ret_callback"
ctx.body = {
result: "ret_callback result here",
}
});
i'm testing await functions but i have an "undefined" result when i check the return of the async function, my express function is like this (i removed unnecesary code, just to show how i am using the async
const getAppsConsumptionSum = async (msisdn, startPeriod, endPeriod) => {
var urlTigoPlus = 'http://...';
var args = {
requestConfig: {
timeout: config.get('localServer.remoteTimeout')
}
};
remoteApi = await restClient.get(url, args,
async (data, response) => {
if (response.statusCode === 200) {
sumatoria = await group(data.arrayofdata).by('subapplication').reduce(async function(id, entries) {
return {
appname: id,
mb: (entries.map(getBytes).reduce(add)) / 1048576
};
});
return sumatoria;
} else {
next(utils.error(503));
}
}
);
};
exports.dataAppsConsumption = async function(req, resp, next) {
let prepaidQuery = 'select ...';
const resultPrepaid = await clientDseDev.execute(prepaidQuery)
.then(async resultPrepaid => {
sumatoria = await getAppsConsumptionSum(variable1, startPeriod, endPeriod);
console.log('this variable shows undefined ' + sumatoria)
//i tried also with this
getAppsConsumptionSum(variable1, startPeriod, endPeriod).then((sumatoria) => {
console.log('this variable shows undefined ' + sumatoria)
});
})
.catch((err) => {
console.log(err)
});
};
thanks all for your help, #jfriend00 solution was right, also i had to add a return before the call to the axios function
return axios.get(url, options
)
.then(async function (response) {
if (response.status === 200){
group(response.data.arrayofdata).by('subapplication').reduce(function(id, entries) {
return {
appname: id,
mb: (entries.map(getBytes).reduce(add)) / 1048576
};
});
return sumatoria;