Firebase cloud function: http function returns null - node.js

Here is what I am trying to do.
I am introducing functionality to enable users to search for local restaurants.
I created a HTTP cloud function, so that when the client delivers a keyword, the function will call an external API to search for the keyword, fetch the responses, and deliver the results.
In doing #2, I need to make two separate url requests and merge the results.
When I checked, the function does call the API, fetch the results and merge them without any issue. However, for some reason, it only returns null to the client.
Below is the code: could someone take a look and advise me on where I went wrong?
exports.restaurantSearch = functions.https.onCall((data,context)=>{
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API"+k;
const url2 = "another_url_to_call_the_external_API"+k;
const url_array = [ url1, url2 ];
const result_array = [];
const info_array = [];
url_array.forEach(url=>{
return fetch(url, {headers: {"Authorization": "API_KEY"}})
.then(response=>{
return response.json()
})
.then(res=>{
result_array.push(res.documents);
if (result_array.length===2) {
const new_result_array_2 = [...new Set((result_array))];
new_result_array_2.forEach(nra=>{
info_array.push([nra.place_name,nra.address_name])
})
//info_array is not null at this point, but the below code only return null when checked from the client
return info_array;
}
})
.catch(error=>{
console.log(error)
return 'error';
})
})
});
Thanks a lot in advance!

You should use Promise.all() instead of running each promise (fetch request) separately in a forEach loop. Also I don't see the function returning anything if result_array.length is not 2. I can see there are only 2 requests that you are making but it's good to handle all possible cases so try adding a return statement if the condition is not satisfied. Try refactoring your code to this (I've used an async function):
exports.restaurantSearch = functions.https.onCall(async (data, context) => {
// Do note the async ^^^^^
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API" + k;
const url2 = "another_url_to_call_the_external_API" + k;
const url_array = [url1, url2];
const responses = await Promise.all(url_array.map((url) => fetch(url, { headers: { "Authorization": "API_KEY" } })))
const responses_array = await Promise.all(responses.map((response) => response.json()))
console.log(responses_array)
const result_array: any[] = responses_array.map((res) => res.documents)
// Although this if statement is redundant if you will be running exactly 2 promises
if (result_array.length === 2) {
const new_result_array_2 = [...new Set((result_array))];
const info_array = new_result_array_2.map(({place_name, address_name}) => ({place_name, address_name}))
return {data: info_array}
}
return {error: "Array length incorrect"}
});
If you'll be running 2 promises only, other option would be:
// Directly adding promises in Promise.all() instead of using map
const [res1, res2] = await Promise.all([fetch("url1"), fetch("url2")])
const [data1, data2] = await Promise.all([res1.json(), res2.json()])
Also check Fetch multiple links inside of forEach loop

Related

Facing issue with scooping in node JS

This code is showing empty object ( {} )
// declared at top
let mainData = {};
let trainStations = {};
let routes = {};
let trainNo = {};
data["data"].forEach(async (element) => {
const response2 = await fetch(
`https://india-rail.herokuapp.com/trains/getRoute?trainNo=${element["train_base"]["train_no"]}`
);
const data2 = await response2.json();
data2["data"].forEach((ele) => {
routes[ele["source_stn_code"]] = true;
});
trainNo[element["train_base"]["train_no"]] = routes;
});
console.log(trainNo);
if i do this then i will give response with data
data["data"].forEach(async (element) => {
const response2 = await fetch(
`https://india-rail.herokuapp.com/trains/getRoute?trainNo=${element["train_base"]["train_no"]}`
);
const data2 = await response2.json();
data2["data"].forEach((ele) => {
routes[ele["source_stn_code"]] = true;
});
trainNo[element["train_base"]["train_no"]] = routes;
console.log(trainNo);
});
maybe there is some scooping issue please kindly help me to solve this problem :)
Please refer here and also check this.
As a short note, using await inside a forEach() loop will give unexpected results. This is because the forEach() does not wait until the promise to settled (either fulfilled or rejected).
A simple solution for this could be using either the traditional for loop or the for..of loop.
for(let element of data["data"]){
const response2 = await fetch(
`https://india-rail.herokuapp.com/trains/getRoute?trainNo=${element["train_base"]["train_no"]}`
);
const data2 = await response2.json();
data2["data"].forEach((ele) => {
routes[ele["source_stn_code"]] = true;
});
trainNo[element["train_base"]["train_no"]] = routes;
}
console.log(trainNo);
NOTE: Make sure to wrap the above for..of loop inside an async function because the await keyword is allowed inside a function only when the function is defined with async keyword.

Asynchronous function in Node.js API not working as intended

As an exercise, I'm creating a simple API that allows users to provide a search term to retrieve links to appropriate news articles across a collection of resources. The relevent function and the route handler that uses the function is as follows:
function GetArticles(searchTerm) {
const articles = [];
//Loop through each resource
resources.forEach(async resource => {
const result = await axios.get(resource.address);
const html = result.data;
//Use Cheerio: load the html document and create Cheerio selector/API
const $ = cheerio.load(html);
//Filter html to retrieve appropriate links
$(`a:contains(${searchTerm})`, html).each((i, el) => {
const title = $(el).text();
let url = $(el).attr('href');
articles.push(
{
title: title,
url: url,
source: resource.name
}
);
})
})
return articles; //Empty array is returned
}
And the route handler that uses the function:
app.get('/news/:searchTerm', async (req, res) => {
const searchTerm = req.params.searchTerm;
const articles = await GetArticles(searchTerm);
res.json(articles);
})
The problem I'm getting is that the returned "articles" array is empty. However, if I'm not "looping over each resource" as commented in the beginning of GetArticles, but instead perform the main logic on just a single "resource", "articles" is returned with the requested data and is not empty. In other words, if the function is the following:
async function GetArticles(searchTerm) {
const articles = [];
const result = await axios.get(resources[0].address);
const html = result.data;
const $ = cheerio.load(html);
$(`a:contains(${searchTerm})`, html).each((i, el) => {
const title = $(el).text();
let url = $(el).attr('href');
articles.push(
{
title: title,
url: url,
source: resources[0].name
}
);
})
return articles; //Populated array
}
Then "articles" is not empty, as intended.
I'm sure this has to do with how I'm dealing with the asynchronous nature of the code. I've tried refreshing my knowledge of asynchronous programming in JS but I still can't quite fix the function. Clearly, the "articles" array is being returned before it's populated, but how?
Could someone please help explain why my GetArticles function works with a single "resource" but not when looping over an array of "resources"?
Try this
function GetArticles(searchTerm) {
return Promise.all(resources.map(resource => axios.get(resource.address))
.then(responses => responses.flatMap(result => {
const html = result.data;
//Use Cheerio: load the html document and create Cheerio selector/API
const $ = cheerio.load(html);
let articles = []
//Filter html to retrieve appropriate links
$(`a:contains(${searchTerm})`, html).each((i, el) => {
const title = $(el).text();
let url = $(el).attr('href');
articles.push(
{
title: title,
url: url,
source: resource.name
}
);
})
return articles;
}))
}
The problem in your implementation was here
resources.forEach(async resource...
You have defined your function async but when result.foreach get executed and launch your async functions it doesn't wait.
So your array will always be empty.

How to mock variable responses based on the url when fetch(url) is called using the Jest testing framework?

I have a module that looks like follows:
calculate-average.js
const fetch = require('node-fetch')
const stats = require('stats-lite')
const BASE_URL = 'https://www.example.com/api'
const calculateAverage = async(numApiCalls) => {
const importantData = []
for (let i = 0; i < numApiCalls; i++) {
const url = `${BASE_URL}/${i}` // will make requests to https://www.example.com/api/0, https://www.example.com/api/1 and so on....
const res = await fetch(url)
const jsonRes = await res.json()
importantData.push(jsonRes.importantData)
}
return stats.mean(importantData)
}
module.exports = calculateAverage
I tried testing it along the following lines but I am clearly way off from the solution:
calculate-average.test.js
const calculateAverage = require('../calculate-average')
jest.mock(
'node-fetch',
() => {
return jest.fn(() => {})
}
)
test('Should calculate stats for liquidation open interest delatas', async() => {
const stats = await calculateAverage(100) // Should make 100 API calls.
console.log(stats)
})
What I need to do is the following:
Be able to specify custom varied responses for each API call. For example, I should be able to specify that a call to https://www.example.com/api/0 returns { importantData: 0 }, a call to https://www.example.com/api/1 returns { importantData: 1 } and so on...
If a request is made to a url that I have not specified a response for, a default response is provided. For example if a response is made to https://www.example.com/api/101, then a default response of { importantData: 1000 } is sent.
I would preferably like to do this only using Jest without depending on modules like mock-fetch and jest-mock-fetch. However, if the solution without using is way too complex, then I would be happy to use them. Just don't want to create unnecessary dependencies if not required.
Sure you can! You can use mock function mockResolvedValueOnce method to return a result for a specific call and mockResolvedValue to return the default result.
jest.mock('node-fetch', () => {
const generateResponse = (value) => {
return { json: () => ({ importantData: value }) };
};
return jest
.fn()
.mockResolvedValue(generateResponse(1000)) // default response
.mockResolvedValueOnce(generateResponse(0)) // response for first call
.mockResolvedValueOnce(generateResponse(1)) // response for second call
.mockResolvedValueOnce(generateResponse(2)); // response for third call
});
Note that we are returning an object with the json property so that it returns the json data when you call res.json() in calculate-average.js.
If you want to return a specific response based on the url parameter, you will have to mock the desired behaviour in the returned mock function for node-fetch. The following example will mock the returned value so that for URLs where the counter is greater than 100 it will return 1000. Otherwise, it will return the same value present in the url:
jest.mock('node-fetch', () => {
return jest.fn((url) => {
// Get and parse the URL parameter.
const value = parseInt(url.split('/').slice(-1)[0], 10);
return Promise.resolve({
json: () => ({ importantData: value > 100 ? 1000 : value })
});
});
});

node.js Get.Request & Pagination & Async

I'm having a tremendously tough time organizing the flow here as I'm self-taught so wondering if someone might be able to assist.
var channelIds = ['XYZ','ABC','QRS']
var playlistIds = [];
var videoIds = [];
ORDER OF PROCESS
1. Get All Playlist IDs: If returning Get Request JSON contains nextPageToken run Get Request again with that page before going to (2)
2. Get All Video IDs: If returning Get Request JSON contains nextPageToken run Get Request again with that page before going to (3)
3. Aggregate into Final Array: I need put all in an array such as:
var ArrFinal = [{channelId,playlistID,videoId},{channelId,playlistID,videoId},{channelId,playlistID,videoId}];
I don't necessarily need someone to write the whole thing. I'm trying to better understand the most efficient way to know when the previous step is done, but also handle the nextPageToken iteration.
i'm not familiar with the youtube api.
But what you basically need is a get function for each endpoint. This function should also care about the "nextPageToken".
Something like that: (not tested)
'use strict';
const Promise = require('bluebird');
const request = Promise.promisifyAll(require('request'));
const playlistEndpoint = '/youtube/v3/playlists';
const baseUrl = 'https://www.googleapis.com'
const channelIds = ['xy', 'ab', 'cd'];
const getPlaylist = async (channelId, pageToken, playlists) => {
const url = `${baseUrl}${playlistEndpoint}`;
const qs = { 
channelId,
maxResults: 25,
pageToken
};
try {
const playlistRequest = await request.getAsync({ url, qs });
const nextPageToken = playlistRequest.body.nextPageToken;
// if we already had items, combine with the new ones
const items = playlists ? playlists.concat(playlistRequest.body.items) : playlistRequest.body.items;
if (nextPageToken) {
// if token, do the same again and pass results to function
return getPlaylist(channelId, nextPageToken, items);
}
// if no token we are finished
return items;
}
catch (e) {
console.log(e.message);
}
};
const getVideos = async (playlistId, pageToken, videos) => {
// pretty much the same as above
}
function awesome(channelIds) {
const fancyArray = [];
await Promise.map(channelIds, async (channelId) => {
const playlists = await getPlaylist(channelId);
const videos = await Promise.map(playlists, async (playlistId) => {
const videos = await getVideos(playlistId);
videos.forEach(videoId => {
fancyArray.push({ channelId, playlistId, videoId })
})
});
});
return fancyArray;
}
awesome(channelIds)
// UPDATE
This may be a lot concurrent requests, you can limit them by using
Promise.map(items, item => { somefunction() }, { concurrency: 5 });

Fetch with absolute url prefix

Most of the times I prefix fetch or node-fetch with an http://localhost (to make it an absolute url).
import fetch from 'node-fetch';
fetch('http://localhost/whatever')
Is there any way of avoiding the localhost part, other than simply placing localhost in a variable?
const baseUrl = 'http://localhost';
fetch(`${baseUrl}/whatever`)
Very related to Superagent with absolute url prefix
TL;DR: fetch-absolute does exactly that.
Detailed:
You can create one abstraction layer on top of fetch.
function fetchAbsolute(fetch) {
return baseUrl => (url, ...otherParams) => url.startsWith('/') ? fetch(baseUrl + url, ...otherParams) : fetch(url, ...otherParams)
}
Or you can simply use fetch-absolute.
const fetch = require('node-fetch');
const fetchAbsolute = require('fetch-absolute');
const fetchApi = fetchAbsolute(fetch)('http://localhost:3030');
it('should should display "It works!"', async () => {
const response = await fetchApi('/');
const json = await response.json();
expect(json).to.eql({ msg: 'It works!' });
});
You can override the fetch function:
import origFetch from 'node-fetch';
const fetch = (url, ...params) => {
if (url.startsWith('/')) return origFetch('http://localhost' + url, ...params)
else return origFetch(url, ...params);
}
The other answer creates a function that returns a function that returns a function--that's not necessary; you just need to return a function.
function fetchAbsolute(fetch, base_url) {
return (url, ...params) => {
if (url.startsWith('/')) return fetch(base_url + url, ...params)
else return fetch(url, ...params);
}
}
const fetch = fetchAbsolute(origFetch, 'http://localhost');

Resources