problem with making api request in node.js and getting the response in/to function caller - node.js

Ive spent a bit of time trying to understand this. I hope the answer is obvious and just show my lack of experience
My goal is to send API request to steam for various IDs of game mods and find the time_updated for each one, to put these all into an array and then find out which one most the most recently updated
I have got the code below, but its not quite doing what I want, I think I am just getting muddled over timings
My plan was to have a few different values in arrMODID = [], and to loop through each one, get the time_updated, push that to an array and for const result = await myfunction(); to be able to access the data in the modinfoArray
however that is returning an array with just [{test},{}] in it and is being fired before the function has put any data into the array
can anyone give me a shove in the right direction please
thank you
import request from 'request';
const myfunction = async function(x, y) {
var arrMODID = ["2016338122"];
var modinfoArray = []
var timeUpdated
for (const element of arrMODID) {
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
},
function(error, response, body){
var response = JSON.parse(body);
var myResponse = response.response.publishedfiledetails
myResponse.forEach(function(arrayItem) {
//console.log(arrayItem.time_updated)
timeUpdated = arrayItem.time_updated
//console.log(timeUpdated)
modinfoArray.push({"2016338122":arrayItem.time_updated})
console.log(modinfoArray) // only this log returns the added items
})
});
}
return ["test", modinfoArray];
};
// Start function
const start = async function(a, b) {
const result = await myfunction();
console.log(result); // this returns the empty array
}
// Call start
start();

You need to use an http request library that supports promises so you can await that inside your function. You cannot successfully mix promises and asynchronous operations like request.post() that uses plain callbacks because you can manage the control flow in a promise-like way with plain callbacks.
I'd suggest using the got() library. Also, the request() library has been deprecated and is not recommended for new code. If you absolutely wanted to stay with the request() library, you could use the request-promise module instead, but keep in mind that the request() library is in maintenance mode only (no new feature development) whereas this list of alternatives are all being actively developed.
Here's a runnable implementation using the got() library:
import got from 'got';
const myfunction = async function() {
const arrMODID = ["2016338122"];
const modinfoArray = [];
for (const element of arrMODID) {
const response = await got.post({
headers: { 'content-type': 'application/x-www-form-urlencoded' },
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
}).json();
const myResponse = response.response.publishedfiledetails;
for (const arrayItem of myResponse) {
modinfoArray.push({ "2016338122": arrayItem.time_updated });
}
}
return ["test", modinfoArray];
};
// Start function
const start = async function() {
const result = await myfunction();
console.log(result);
return result;
}
// Call start
start().then(result => {
console.log("done");
}).catch(err => {
console.log(err);
});

Related

How to call a function that waits for return of API call (Node.js)

While learning Node.js and after some trial and error I have this working code that makes API calls to retrieve user records where each API call is dependent on the result of the previous API call.
const axios = require('axios')
var getDataById = async (config) => {
var response = await axios(config);
var userById = {};
userById['userId'] = response.data["userId"];
return(userById);
};
(async () => {
var a = []; // array for storing user data
var userId = '0001';
var done = false; // false until user '0010' is reached
while (!done) {
url = "https://someurl.com/" + userId;
var axiosconfig = {
method: 'get',
url: url,
headers: {
'Authorization': 'Bearer SECRET_TOKEN'
}
};
var userDataById = await getDataById(axiosconfig);
a.push(userDataById);
userId = userDataById['userId'];
if (userId == '0010') { done = true }
}
})()
How can I call this code from elsewhere in my program in such a way that
I can pass arguments to it...for example 'userId'
I can return 'a' to the calling function
the calling function waits for 'a' to be returned before continuing
TBH, I don't quite get how this works.
;(async () => {})()
Is there a better way of coding this?
You need to create an async function and then await it.
The function definition will look like this
async function AddUser(userId)
You can return any variable and you call it like this:
await addUser('002')
await keyword make sure that the calling function waits for 'a' to be returned before continuing.
For your last question:
(async () => {})()
Is an arrow function.
See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions
Using async keyword + arrow function is a trick to allow you to create an async context and being able to use await. The two last () are used to direct call this function.

Node.js backend return response before all the API calls within the endpoints are made

I have a GET endpoint, which basically makes some API calls to the Spoonacular API. Essentially, I make two API calls within the endpoint.
The first API call gets the list of recipe ID's for the specific ingredients
The second API calls gets the metadata for each of the recipe ID's.
After the first API call I store all the Id's in an array (recipeArray), and I want to make the second api call for each ID in my array (function recipeTest does this).
When I try to do this and then return my response to the front end, it always returns a response before completing all the API calls in the second step.
Here, is my code. The first API calls works just fine, but the second API call (recipeTest function), is where it messed up. Before that function finishes making all the API calls to the Spoonacular API, my endpoint returns an empty Array (res.send(toSend)). So, I was just wondering if there is any way around this?
Thank you so much in advance, I really appreciate it!
module.exports = (app) => {
app.get('/api/search', async (req, res) => {
console.log("endpoint working");
let ingredientList = "apples,+eggs,+bacon"; // needs to be given from the front end
let ingredientSearchUrl = `https://api.spoonacular.com/recipes/findByIngredients?ingredients=${ingredientList}&number=1&ignorePantry=true&apiKey=${keys.spoonacularKey}`;
try {
const ingredientSearchResult = await axios({
method: 'get',
url: ingredientSearchUrl
});
var recipeArray = ingredientSearchResult.data.map(info => {
return info.id;
});
} catch (err) {
console.log("error in finding recipe ID ", err);
}
let toSend = [];
try {
const check = await recipeTest(recipeArray, toSend);
} catch (err) {
console.log("error in finding recipe information ", err);
}
res.send(toSend);
});
}
const recipeTest = async (recipeArray, toSend) => {
return Promise.all(
_.forEach(recipeArray, async (recipeId) => {
let recipeInfoUrl = `https://api.spoonacular.com/recipes/${recipeId}/information?includeNutrition=false&apiKey=${keys.spoonacularKey}`;
let recipeInfo = {};
const recipeData = await axios({
method: 'get',
url: recipeInfoUrl
});
// console.log("recipeInfo search working", recipeData.data);
recipeInfo['id'] = recipeData.data.id;
recipeInfo['title'] = recipeData.data.title;
recipeInfo['time'] = recipeData.data.readyInMinutes;
recipeInfo['recipeUrl'] = recipeData.data.sourceUrl;
recipeInfo['imageUrl'] = recipeData.data.image;
// console.log('recipe info dict', recipeInfo);
toSend.push(recipeInfo);
console.log('toSend inside', toSend);
})
);
}
_.forEach return collection itself and not all your async handlers.
Use recipeArray.map to get an array of async functions to let Promise.all do its work:
Promise.all(
recipeArray.map(x => async (recipeId) => {

How to make multiple http requests from a Google Cloud Function (Cheerio, Node.js)

MY PROBLEM:
I'm building a web-scraper with Cheerio, Node.js, and Google Cloud Functions.
The problem is I need to make multiple requests, then write data from each request to a Firestore database before calling response.send() and thereby terminating the function.
My code requires two loops: the first loop is with urls from my db, with each one making a separate request. The second loop is with Cheerio using .each to scrape multiple rows of table data from the DOM and make a separate write for each row.
WHAT I'VE TRIED:
I've tried pushing each request to an array of promises and then waiting for all the promises to resolve with promises.all() before calling res.send(), but I'm still a little shaky on promises and not sure that is the right approach. (I have gotten the code to work for smaller datasets that way, but still inconsistently.)
I also tried creating each request as a new promise and using async/await to await each function call from the forEach loop to allow time for each request and write to fully finish so I could call res.send() afterward, but I found out that forEach doesn't support Async/await.
I tried to get around that with the p-iteration module but because its not actually forEach but rather a method on the query (doc.forEach()) I don't think it works like that.
So here's my code.
NOTE:
As mentioned, this is not everything I tried (I removed my promise attempts), but this should show what I am trying to accomplish.
export const getCurrentLogs = functions.https.onRequest((req, response) => {
//First, I make a query from my db to get the urls
// that I want the webscraper to loop through.
const ref = scheduleRef.get()
.then((snapshot) => {
snapshot.docs.forEach((doc) => {
const scheduleGame = doc.data()
const boxScoreUrl = scheduleGame.boxScoreURL
//Inside the forEach I call the request
// as a function with the url passed in
updatePlayerLogs("https://" + boxScoreUrl + "/");
});
})
.catch(err => {
console.log('Error getting schedule', err);
});
function updatePlayerLogs (url){
//Here I'm not sure on how to set these options
// to make sure the request stays open but I have tried
// lots of different things.
const options = {
uri: url,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
request(options)
.then(($) => {
//Below I loop through some table data
// on the dom with cheerio. Every loop
// in here needs to be written to firebase individually.
$('.stats-rows').find('tbody').children('tr').each(function(i, element){
const playerPage = $(element).children('td').eq(0).find('a').attr('href');
const pts = replaceDash($(element).children('td').eq(1).text());
const reb = replaceDash($(element).children('td').eq(2).text());
const ast = replaceDash($(element).children('td').eq(3).text());
const fg = replaceDash($(element).children('td').eq(4).text());
const _3pt = replaceDash($(element).children('td').eq(5).text());
const stl = replaceDash($(element).children('td').eq(9).text());
const blk = replaceDash($(element).children('td').eq(10).text());
const to = replaceDash($(element).children('td').eq(11).text());
const currentLog = {
'pts': + pts,
'reb': + reb,
'ast': + ast,
'fg': fgPer,
'3pt': + _3ptMade,
'stl': + stl,
'blk': + blk,
'to': + to
}
//here is the write
playersRef.doc(playerPage).update({
'currentLog': currentLog
})
.catch(error =>
console.error("Error adding document: ", error + " : " + url)
);
});
})
.catch((err) => {
console.log(err);
});
};
//Here I call response.send() to finish the function.
// I have tried doing this lots of different ways but
// whatever I try the response is being sent before all
// docs are written.
response.send("finished writing logs")
});
Everything I have tried either results in a deadline exceeded error (possibly because of quota limits which I have looked into but I don't think I should be exceeding) Or some unexplained error where the code doesn't finish executing but shows me nothing in the logs.
Please help, is there a way to use async/await in this scenario that I am not understanding? Is there a way to use promises to make this elegant?
Many thanks,
Maybe have a look at something like this. It uses Bluebird promises and the request-promise library
const Promise = require('bluebird');
var rp = require('request-promise');
const urlList = ['http://www.google.com', 'http://example.com']
async function getList() {
await Promise.map(urlList, (url, index, length) => {
return rp(url)
.then((response) => {
console.log(`${'\n\n\n'}${url}:${'\n'}${response}`);
return;
}).catch(async (err) => {
console.log(err);
return;
})
}, {
concurrency: 10
}); //end Promise.map
}
getList();

NodeJs delay each promise within Promise.all()

I'm trying to update a tool that was created a while ago which uses nodejs (I am not a JS developer, so I'm trying to piece the code together) and am getting stuck at the last hurdle.
The new functionality will take in a swagger .json definition, compare the endpoints against the matching API Gateway on the AWS Service, using the 'aws-sdk' SDK for JS and then updates the Gateway accordingly.
The code runs fine on a small definition file (about 15 endpoints) but as soon as I give it a bigger one, I start getting tons of TooManyRequestsException errors.
I understand that this is due to my calls to the API Gateway service being too quick and a delay / pause is needed. This is where I am stuck
I have tried adding;
a delay() to each promise being returned
running a setTimeout() in each promise
adding a delay to the Promise.all and Promise.mapSeries
Currently my code loops through each endpoint within the definition and then adds the response of each promise to a promise array:
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath));
Once the loop is finished I run this:
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
I have tried the same with a mapSeries (no luck).
It looks like the functions within the (getMethodResponse promise) are run immediately and hence, no matter what type of delay I add they all still just execute. My suspicious is that the I need to make (getMethodResponse) return a function and then use mapSeries but I cant get this to work either.
Code I tried:
Wrapped the getMethodResponse in this:
return function(value){}
Then added this after the loop (and within the loop - no difference):
Promise.mapSeries(function (promises) {
return 'a'();
}).then(function (results) {
console.log('result', results);
});
Also tried many other suggestions:
Here
Here
Any suggestions please?
EDIT
As request, some additional code to try pin-point the issue.
The code currently working with a small set of endpoints (within the Swagger file):
module.exports = (apiName, externalUrl) => {
return getSwaggerFromHttp(externalUrl)
.then((swagger) => {
let paths = swagger.paths;
let resourcePath = '';
let resourceMethod = '';
let promises = [];
_.each(paths, function (value, key) {
resourcePath = key;
_.each(value, function (value, key) {
resourceMethod = key;
let statusList = [];
_.each(value.responses, function (value, key) {
if (key >= 200 && key <= 204) {
statusList.push(key)
}
});
_.each(statusList, function (value, key) { //Only for 200-201 range
//Working with small set
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath))
});
});
});
//Working with small set
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
})
.catch((err) => {
winston.error(err);
});
};
I have since tried adding this in place of the return Promise.all():
Promise.map(promises, function() {
// Promise.map awaits for returned promises as well.
console.log('X');
},{concurrency: 5})
.then(function() {
return console.log("y");
});
Results of this spits out something like this (it's the same for each endpoint, there are many):
Error: TooManyRequestsException: Too Many Requests
X
Error: TooManyRequestsException: Too Many Requests
X
Error: TooManyRequestsException: Too Many Requests
The AWS SDK is being called 3 times within each promise, the functions of which are (get initiated from the getMethodResponse() function):
apigateway.getRestApisAsync()
return apigateway.getResourcesAsync(resourceParams)
apigateway.getMethodAsync(params, function (err, data) {}
The typical AWS SDK documentation state that this is typical behaviour for when too many consecutive calls are made (too fast). I've had a similar issue in the past which was resolved by simply adding a .delay(500) into the code being called;
Something like:
return apigateway.updateModelAsync(updateModelParams)
.tap(() => logger.verbose(`Updated model ${updatedModel.name}`))
.tap(() => bar.tick())
.delay(500)
EDIT #2
I thought in the name of thorough-ness, to include my entire .js file.
'use strict';
const AWS = require('aws-sdk');
let apigateway, lambda;
const Promise = require('bluebird');
const R = require('ramda');
const logger = require('../logger');
const config = require('../config/default');
const helpers = require('../library/helpers');
const winston = require('winston');
const request = require('request');
const _ = require('lodash');
const region = 'ap-southeast-2';
const methodLib = require('../aws/methods');
const emitter = require('../library/emitter');
emitter.on('updateRegion', (region) => {
region = region;
AWS.config.update({ region: region });
apigateway = new AWS.APIGateway({ apiVersion: '2015-07-09' });
Promise.promisifyAll(apigateway);
});
function getSwaggerFromHttp(externalUrl) {
return new Promise((resolve, reject) => {
request.get({
url: externalUrl,
header: {
"content-type": "application/json"
}
}, (err, res, body) => {
if (err) {
winston.error(err);
reject(err);
}
let result = JSON.parse(body);
resolve(result);
})
});
}
/*
Deletes a method response
*/
function deleteMethodResponse(httpMethod, resourceId, restApiId, statusCode, resourcePath) {
let methodResponseParams = {
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: restApiId,
statusCode: statusCode
};
return apigateway.deleteMethodResponseAsync(methodResponseParams)
.delay(1200)
.tap(() => logger.verbose(`Method response ${statusCode} deleted for path: ${resourcePath}`))
.error((e) => {
return console.log(`Error deleting Method Response ${httpMethod} not found on resource path: ${resourcePath} (resourceId: ${resourceId})`); // an error occurred
logger.error('Error: ' + e.stack)
});
}
/*
Deletes an integration response
*/
function deleteIntegrationResponse(httpMethod, resourceId, restApiId, statusCode, resourcePath) {
let methodResponseParams = {
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: restApiId,
statusCode: statusCode
};
return apigateway.deleteIntegrationResponseAsync(methodResponseParams)
.delay(1200)
.tap(() => logger.verbose(`Integration response ${statusCode} deleted for path ${resourcePath}`))
.error((e) => {
return console.log(`Error deleting Integration Response ${httpMethod} not found on resource path: ${resourcePath} (resourceId: ${resourceId})`); // an error occurred
logger.error('Error: ' + e.stack)
});
}
/*
Get Resource
*/
function getMethodResponse(httpMethod, statusCode, apiName, resourcePath) {
let params = {
httpMethod: httpMethod.toUpperCase(),
resourceId: '',
restApiId: ''
}
return getResourceDetails(apiName, resourcePath)
.error((e) => {
logger.unimportant('Error: ' + e.stack)
})
.then((result) => {
//Only run the comparrison of models if the resourceId (from the url passed in) is found within the AWS Gateway
if (result) {
params.resourceId = result.resourceId
params.restApiId = result.apiId
var awsMethodResponses = [];
try {
apigateway.getMethodAsync(params, function (err, data) {
if (err) {
if (err.statusCode == 404) {
return console.log(`Method ${params.httpMethod} not found on resource path: ${resourcePath} (resourceId: ${params.resourceId})`); // an error occurred
}
console.log(err, err.stack); // an error occurred
}
else {
if (data) {
_.each(data.methodResponses, function (value, key) {
if (key >= 200 && key <= 204) {
awsMethodResponses.push(key)
}
});
awsMethodResponses = _.pull(awsMethodResponses, statusCode); //List of items not found within the Gateway - to be removed.
_.each(awsMethodResponses, function (value, key) {
if (data.methodResponses[value].responseModels) {
var existingModel = data.methodResponses[value].responseModels['application/json']; //Check if there is currently a model attached to the resource / method about to be deleted
methodLib.updateResponseAssociation(params.httpMethod, params.resourceId, params.restApiId, statusCode, existingModel); //Associate this model to the same resource / method, under the new response status
}
deleteMethodResponse(params.httpMethod, params.resourceId, params.restApiId, value, resourcePath)
.delay(1200)
.done();
deleteIntegrationResponse(params.httpMethod, params.resourceId, params.restApiId, value, resourcePath)
.delay(1200)
.done();
})
}
}
})
.catch(err => {
console.log(`Error: ${err}`);
});
}
catch (e) {
console.log(`getMethodAsync failed, Error: ${e}`);
}
}
})
};
function getResourceDetails(apiName, resourcePath) {
let resourceExpr = new RegExp(resourcePath + '$', 'i');
let result = {
apiId: '',
resourceId: '',
path: ''
}
return helpers.apiByName(apiName, AWS.config.region)
.delay(1200)
.then(apiId => {
result.apiId = apiId;
let resourceParams = {
restApiId: apiId,
limit: config.awsGetResourceLimit,
};
return apigateway.getResourcesAsync(resourceParams)
})
.then(R.prop('items'))
.filter(R.pipe(R.prop('path'), R.test(resourceExpr)))
.tap(helpers.handleNotFound('resource'))
.then(R.head)
.then([R.prop('path'), R.prop('id')])
.then(returnedObj => {
if (returnedObj.id) {
result.path = returnedObj.path;
result.resourceId = returnedObj.id;
logger.unimportant(`ApiId: ${result.apiId} | ResourceId: ${result.resourceId} | Path: ${result.path}`);
return result;
}
})
.catch(err => {
console.log(`Error: ${err} on API: ${apiName} Resource: ${resourcePath}`);
});
};
function delay(t) {
return new Promise(function(resolve) {
setTimeout(resolve, t)
});
}
module.exports = (apiName, externalUrl) => {
return getSwaggerFromHttp(externalUrl)
.then((swagger) => {
let paths = swagger.paths;
let resourcePath = '';
let resourceMethod = '';
let promises = [];
_.each(paths, function (value, key) {
resourcePath = key;
_.each(value, function (value, key) {
resourceMethod = key;
let statusList = [];
_.each(value.responses, function (value, key) {
if (key >= 200 && key <= 204) {
statusList.push(key)
}
});
_.each(statusList, function (value, key) { //Only for 200-201 range
promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath))
});
});
});
//Working with small set
return Promise.all(promises)
.catch((err) => {
winston.error(err);
})
})
.catch((err) => {
winston.error(err);
});
};
You apparently have a misunderstanding about what Promise.all() and Promise.map() do.
All Promise.all() does is keep track of a whole array of promises to tell you when the async operations they represent are all done (or one returns an error). When you pass it an array of promises (as you are doing), ALL those async operations have already been started in parallel. So, if you're trying to limit how many async operations are in flight at the same time, it's already too late at that point. So, Promise.all() by itself won't help you control how many are running at once in any way.
I've also noticed since, that it seems this line promises.push(getMethodResponse(resourceMethod, value, apiName, resourcePath)) is actually executing promises and not simply adding them to the array. Seems like the last Promise.all() doesn't actually do much.
Yep, when you execute promises.push(getMethodResponse()), you are calling getMethodResponse() immediately right then. That starts the async operation immediately. That function then returns a promise and Promise.all() will monitor that promise (along with all the other ones you put in the array) to tell you when they are all done. That's all Promise.all() does. It monitors operations you've already started. To keep the max number of requests in flight at the same time below some threshold, you have to NOT START the async operations all at once like you are doing. Promise.all() does not do that for you.
For Bluebird's Promise.map() to help you at all, you have to pass it an array of DATA, not promises. When you pass it an array of promises that represent async operations that you've already started, it can do no more than Promise.all() can do. But, if you pass it an array of data and a callback function that can then initiate an async operation for each element of data in the array, THEN it can help you when you use the concurrency option.
Your code is pretty complex so I will illustrate with a simple web scraper that wants to read a large list of URLs, but for memory considerations, only process 20 at a time.
const rp = require('request-promise');
let urls = [...]; // large array of URLs to process
Promise.map(urls, function(url) {
return rp(url).then(function(data) {
// process scraped data here
return someValue;
});
}, {concurrency: 20}).then(function(results) {
// process array of results here
}).catch(function(err) {
// error here
});
In this example, hopefully you can see that an array of data items are being passed into Promise.map() (not an array of promises). This, then allows Promise.map() to manage how/when the array is processed and, in this case, it will use the concurrency: 20 setting to make sure that no more than 20 requests are in flight at the same time.
Your effort to use Promise.map() was passing an array of promises, which does not help you since the promises represent async operations that have already been started:
Promise.map(promises, function() {
...
});
Then, in addition, you really need to figure out what exactly causes the TooManyRequestsException error by either reading documentation on the target API that exhibits this or by doing a whole bunch of testing because there can be a variety of things that might cause this and without knowing exactly what you need to control, it just takes a lot of wild guesses to try to figure out what might work. The most common things that an API might detect are:
Simultaneous requests from the same account or source.
Requests per unit of time from the same account or source (such as request per second).
The concurrency operation in Promise.map() will easily help you with the first option, but will not necessarily help you with the second option as you can limit to a low number of simultaneous requests and still exceed a requests per second limit. The second needs some actual time control. Inserting delay() statements will sometimes work, but even that is not a very direct method of managing it and will either lead to inconsistent control (something that works sometimes, but not other times) or sub-optimal control (limiting yourself to something far below what you can actually use).
To manage to a request per second limit, you need some actual time control with a rate limiting library or actual rate limiting logic in your own code.
Here's an example of a scheme for limiting the number of requests per second you are making: How to Manage Requests to Stay Below Rate Limiting.

Request JSON within a loop

I'm having trouble to surf correctly between objects that are been returned by an API request.
Basically, I have an Array (tickerArr with 25 elements that I get from another request) which I use for a forEach loop for another request.
Here's what the code looks like:
var request = require('request');
var async = require('async');
var fs = require('fs')
var getTickerList = require('./getTickerList').getTickerList;
var unixTimeStamp = new Date().toISOString();
async function getTickerInfo() {
var tickerArr = await getTickerList;
var arrLength = tickerArr.length;
var init = 0;
console.log(`${arrLength} to process to the batchfile...`);
var getTickerInfo = new Promise((resolve, reject) => {
async.forEach(tickerArr, (item, callback) => {
request({
url:`https://api.kraken.com/0/public/Ticker?pair=${item}`,
json: true
}, (error, response, body) => {
var tickerInfo = JSON.stringify(body.result)
fs.appendFile(`./modules/batches/${unixTimeStamp}batch.json`, tickerInfo, (err) => {
if (err) throw err;
init ++;
var progress = Math.round(init / arrLength * 100);
console.log(`[${progress}%] - ${item} successfully added!`);
});
resolve();
})
});
});
}
getTickerInfo()
Unfortunately, and even if the request works correctly, the objects being returned have a specific path:
Error: []
Result:
Pair(x):
a: [1, 2, 3]
b: [1, 2, 3] ect...
You can find an exact example of the information being returned from the request here:
{
"error": [],
"result": {
"XXBTZEUR": {
"a": ["2212.31000", "3", "3.000"],
"b": ["2212.21100", "1", "1.000"],
"c": ["2212.31000", "0.15800000"],
"v": ["4999.06419498", "9993.55448658"],
"p": ["2206.04624", "2181.36028"],
"t": [15065, 29524],
"l": ["2167.00000", "2122.92000"],
"h": ["2239.00000", "2239.00000"],
"o": "2184.99000"
}
}
}
The only problem is: I would like to 'transform' the object I get from the request, to another object (just arranging stuff around, most of all: put the 'Pair(x)' attribute as a value for a name key) but since I don't know the pair in advance (the values in my tickerArray are enough to make the request but do not apparently correspond to reference the object), I can't access the information contained within result to manipulate it.
Anybody have any idea?
Thanks in advance!
Your question has multiple levels and maybe it's best to break up the task into its constituent parts first.
Calling the Kraken REST API and unwrapping the response
Creating small, composable functions
Transforming an object into a new layout
Writing the results to a file
0. Preface
Since you want to work with await, the first thing to know is that this is syntactical sugar for "wait for promise resolution". async is the keyword that enables await semantics inside a function.
So given this promise-returning function:
function foo() {
return Promise.resolve('some value');
}
this:
async function worker() {
var fooResult = await foo();
console.log(fooResult);
}
worker();
and this:
foo().then(function worker(fooResult) {
console.log(fooResult);
});
are effectively the same thing. (As always, it's a little more complicated than that, but that's the gist of it.)
Since we are working with promises here, the most sensible thing to do is to use libraries that also use promises. request has the request-promise counter-part, so let's use that.
var request = require('request-promise');
1. Calling the Kraken REST API and unwrapping the response
Making an HTTP request and breaking up the response (err and result parts) works the same for all API endpoints, so we can write a function that handles this task.
function getFromApi(endpoint, params) {
return request({
url: "https://api.kraken.com/0/public/" + endpoint,
qs: params,
json: true
}).then(function (data) {
if (data.error && data.error.length) {
throw new Error(`API error xyz for ${endpoint} with params ...`);
}
return data.result;
});
}
Users of this function will not have to deal with err at all and have access to result directly.
2. Creating small, composable functions
Our getFromApi() function returns a promise, therefore we can re-use it in wrappers for various API endpoints:
function getTickerList() {
return getFromApi("TickerList"); // ...or something like that
}
function getSingleTicker(pair) {
return getFromApi("Ticker", {pair: pair}).then(transformTicker);
}
function getTickerInfo(tickerList) {
return Promise.all(tickerList.map(getSingleTicker));
}
Note that, like in getFromApi(), we can use Promise#then to modify the overall output of an operation.
getTickerInfo() accepts an array of ticker names. It uses Array#map to run all API requests in parallel, and Promise#all to allow awaiting the overall result. The fairly complex operation of fetching and transforming multiple results in parallel composes into a pretty straight-forward one-liner.
3. Transforming an object into a new layout
transformTicker() is meant to accept an object in form {"XXBTZEUR": {"a": [...], ... }} and return a transformed variant of it.
function transformTicker(ticker) {
var result = {};
// ...massage ticker into desired form
Object.keys(ticker).forEach( pair => {
Object.keys(ticker[pair]).forEach( k => {
result[k] = { /* ... */ };
});
});
return result;
}
4. Writing the results to a file
Appending to a JSON file does not work. JSON can only be read and written as a whole. Let's fetch a list of ticker names, the associated tickers and write the results to a file.
async function worker() {
var tickerList = await getTickerList();
var tickerInfo = await getTickerInfo(tickerList);
var filename = `./modules/batches/${new Date().toISOString()}batch.json`;
fs.writeFile(filename, JSON.stringify(tickerInfo), (err) => {
if (err) throw err;
console.log("Done");
});
}
worker();
You can switch use a promisified version (see Bluebird#promisifyAll) of the fs module, since the plain version breaks the nice async/await semantics again by requiring a continuation callback.
With a promisified fs module, the worker could look like this:
async function worker() {
var tickerList = await getTickerList();
var tickerInfo = await getTickerInfo(tickerList);
var filename = `./modules/batches/${new Date().toISOString()}batch.json`;
await fs.writeFileAsync(filename, JSON.stringify(tickerInfo));
console.log("Done");
}

Resources