I'm writing a RESTful API in typescript and I'm trying to use my already parsed data that is stored in redis with a specific key in another function. The problem I am having is that instead of receiving the actual data from redis, I just keep receiving a boolean value of true. I have tried googling a lot and reading the redis documentation, unfortunately to no avail. Does anyone here now how I can access the actual data so I can use it in another function? I suspect I am facing some kind of async problems here, but I am not entirely sure.
For example if I try to bind the response to a variable this will happen:
const key = something
const reply = client.mget(key);
console.log("This is the reply: " + reply);
This is the reply: true
Br,
Victor
EDIT:
So basically I'm sending the data from https.get into an object handler, which parses the data into my preferred form and then I stringify that object and send it into redis as a string, which looks like this:
client.set(redisInfo, JSON.stringify(objecthandler(newdata)));
My actual code for getting the data at the moment looks like this:
const getRedis = (rediskey: string, success: any, error: any) => {
client.mget(rediskey, function (err: Error, reply: any) {
if (!err) {
success(reply);
}
else {
error(err);
}
});
};
//My callback function that I'm trying to get to work so I can use the redis data in other functions
function getrediscallback(key: string) {
getRedis(key, function success(reply: string): string {
//This console.log actually returns the data that I want
console.log("this is the reply: " + reply);
return reply;
},
function error(err: Error) {
console.log("Something went wrong" + err);
});
}
So when I use the callback in another function, it will look something like this:
const redisdata = getrediscallback("getCars");
//This gives me the value of undefined
console.log(redisdata)
This means that the callback function actually gets the actual data, but it is never reached later on when I use the callback function in another function.
const redis = require('redis');
const client = redis.createClient(6379);
const bluebird = require("bluebird");
bluebird.promisifyAll(redis.RedisClient.prototype);
bluebird.promisifyAll(redis.Multi.prototype);
const redisdata = await client.getAsync('user:photos');
if (redisdata) {
console.log(`cache EXISTS`)
return res.json({ source: 'cache', data: JSON.parse(redisdata) })
}
import util from "util";
import redisClient from "../config/redisConfig";
let hgetall = util.promisify(redisClient.hgetall).bind(redisClient);
await hgetall("Key")
Related
I tried to make function for my project in the service. This service need to check is user exists in my database, but my function(this function is checking) inside the class return undefined.
This is my code:
const express = require("express");
const mongoDB = require('mongodb').MongoClient;
const url = "here I paste url to my databse(everything is good here)";
class CheckService {
isUserExists(username) {
mongoDB.connect(url, (error, connection) => {
if (error) {
console.log("Error", '\n', error);
throw error;
}
const query = {name: username};
const db = connection.db("users");
const result = db.collection("users").find(query).toArray(
function findUser(error, result) {
if (error) {
throw error;
}
const arr = [];
if (result.value === arr.value) {
console.log(false);
connection.close();
return false;
} else {
console.log(true);
console.log(arr);
console.log(result);
connection.close();
return true;
}
});
console.log(result);
});
}
}
module.exports = new CheckService();
I imported my service to another service:
const checkService = require('./check.service');
After this, I invoked my function from my service like this:
console.log('function:',checkService.isUserExists(username));
I expected good result, but function doesn't return, that I want, unfortunately.
Please help!
There are two problems with your function
it doesn't return anything
toArray() is a promise, so your console.log probably just prints a promise.
Probably you're looking for something like this:
class CheckService {
async isUserExists(username) {
const connection = await mongoDB.connect(url);
const query = {name: username};
const db = connection.db("users");
const result = await db.collection("users").find(query).toArray();
connection.close();
return result.length !== 0;
}
}
You'd then invoke it with something like
await new CheckService().isUserExists(username);
Though, it's worth noting that you probably don't need toArray and instead could use findOne or even count() since all you care about is existence. I probably also wouldn't instantiate a new connection each time (but that's not super relevant)
2 things wrong here. Firstly the first comment is correct. You're only logging the result and not passing back to the caller. Secondly, a quick peek at the mongo docs shows that retrieval methods are promises. You need to make the function async and add awaits where needed.
Mongo:
https://www.mongodb.com/docs/drivers/node/current/fundamentals/crud/read-operations/retrieve/
Promises:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
Ive spent a bit of time trying to understand this. I hope the answer is obvious and just show my lack of experience
My goal is to send API request to steam for various IDs of game mods and find the time_updated for each one, to put these all into an array and then find out which one most the most recently updated
I have got the code below, but its not quite doing what I want, I think I am just getting muddled over timings
My plan was to have a few different values in arrMODID = [], and to loop through each one, get the time_updated, push that to an array and for const result = await myfunction(); to be able to access the data in the modinfoArray
however that is returning an array with just [{test},{}] in it and is being fired before the function has put any data into the array
can anyone give me a shove in the right direction please
thank you
import request from 'request';
const myfunction = async function(x, y) {
var arrMODID = ["2016338122"];
var modinfoArray = []
var timeUpdated
for (const element of arrMODID) {
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
},
function(error, response, body){
var response = JSON.parse(body);
var myResponse = response.response.publishedfiledetails
myResponse.forEach(function(arrayItem) {
//console.log(arrayItem.time_updated)
timeUpdated = arrayItem.time_updated
//console.log(timeUpdated)
modinfoArray.push({"2016338122":arrayItem.time_updated})
console.log(modinfoArray) // only this log returns the added items
})
});
}
return ["test", modinfoArray];
};
// Start function
const start = async function(a, b) {
const result = await myfunction();
console.log(result); // this returns the empty array
}
// Call start
start();
You need to use an http request library that supports promises so you can await that inside your function. You cannot successfully mix promises and asynchronous operations like request.post() that uses plain callbacks because you can manage the control flow in a promise-like way with plain callbacks.
I'd suggest using the got() library. Also, the request() library has been deprecated and is not recommended for new code. If you absolutely wanted to stay with the request() library, you could use the request-promise module instead, but keep in mind that the request() library is in maintenance mode only (no new feature development) whereas this list of alternatives are all being actively developed.
Here's a runnable implementation using the got() library:
import got from 'got';
const myfunction = async function() {
const arrMODID = ["2016338122"];
const modinfoArray = [];
for (const element of arrMODID) {
const response = await got.post({
headers: { 'content-type': 'application/x-www-form-urlencoded' },
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
}).json();
const myResponse = response.response.publishedfiledetails;
for (const arrayItem of myResponse) {
modinfoArray.push({ "2016338122": arrayItem.time_updated });
}
}
return ["test", modinfoArray];
};
// Start function
const start = async function() {
const result = await myfunction();
console.log(result);
return result;
}
// Call start
start().then(result => {
console.log("done");
}).catch(err => {
console.log(err);
});
I'm currently trying to practice using an API with Twitter API.
I'm using Twit package to connect to twitters API but when I try to do a get request I get
Promise { pending }
I have tried using Async-Await but I'm not sure what I'm doing wrong here.
Here is my code:
const Twit = require('twit');
const twitterAPI = require('../secrets');
//Twit uses OAuth to stablish connection with twitter
let T = new Twit({
consumer_key: twitterAPI.apiKey,
consumer_secret: twitterAPI.apiSecretKey,
access_token: twitterAPI.accessToken,
access_token_secret: twitterAPI.accessTokenSecret
})
const getUsersTweets = async (userName) => {
let params = { screen_name: userName, count: 1 }
const userTweets = await T.get('search/tweets', params, await function (err, data, response) {
if (err) {
return 'There was an Error', err.stack
}
return data
})
return userTweets
}
console.log(getUsersTweets('Rainbow6Game'));
Problem
The biggest assumption that is wrong with the sample code is that T.get is expected to eventually resolve with some data.
const userTweets = await T.get('search/tweets', params, await function (err, data, response) {
if (err) {
return 'There was an Error', err.stack
}
return data // 'data' returned from here is not necessarily going to be received in 'userTweets' variable
})
callback function provided as last argument in T.get function call doesn't have to be preceded by an 'await'.
'data' returned from callback function is not necessarily going to be received in 'userTweets' variable. It totally depends on how T.get is implemented and can not be controlled.
Reason
Thing to be noted here is that async / await works well with Promise returning functions which eventually get resolved with expected data, however, that is not guaranteed here
Relying on the result of asynchronous T.get function call will probably not work because it returns a Promise { pending } object immediately and will get resolved with no data. The best case scenario is that everything with your function will work but getUsersTweets function will return 'undefined'.
Solution
The best solution is to make sure that your getUsersTweets function returns a promise which eventually gets resolved with correct data. Following changes are suggested:
const getUsersTweets = (userName) => {
return new Promise ((resolve, reject) => {
let params = { screen_name: userName, count: 1 }
T.get('search/tweets', params, function (err, data, response) {
if (err) {
reject(err);
}
resolve(data);
})
}
}
The above function is now guaranteed to return expected data and can be used in the following way:
const printTweets = async () => {
const tweets = await getUsersTweet(userName);
console.log(tweets);
}
printTweets();
From what I can see on your code, getUserTweets is an async function, so it will eventually return a promise. I'm assuming you will use this value on another function, so you will need to use it inside an async function and use await, otherwise you will always get a promise.
const logTweets = async (username) => {
try {
const userTweets = await getUsersTweets(username);
// Do something with the tweets
console.log(userTweets);
catch (err) {
// catch any error
console.log(err);
}
}
If logging is all you want and you wrapped it inside a function in which you console.log it, you can call that function directly:
logTweets('someUsername');
MY PROBLEM:
I'm building a web-scraper with Cheerio, Node.js, and Google Cloud Functions.
The problem is I need to make multiple requests, then write data from each request to a Firestore database before calling response.send() and thereby terminating the function.
My code requires two loops: the first loop is with urls from my db, with each one making a separate request. The second loop is with Cheerio using .each to scrape multiple rows of table data from the DOM and make a separate write for each row.
WHAT I'VE TRIED:
I've tried pushing each request to an array of promises and then waiting for all the promises to resolve with promises.all() before calling res.send(), but I'm still a little shaky on promises and not sure that is the right approach. (I have gotten the code to work for smaller datasets that way, but still inconsistently.)
I also tried creating each request as a new promise and using async/await to await each function call from the forEach loop to allow time for each request and write to fully finish so I could call res.send() afterward, but I found out that forEach doesn't support Async/await.
I tried to get around that with the p-iteration module but because its not actually forEach but rather a method on the query (doc.forEach()) I don't think it works like that.
So here's my code.
NOTE:
As mentioned, this is not everything I tried (I removed my promise attempts), but this should show what I am trying to accomplish.
export const getCurrentLogs = functions.https.onRequest((req, response) => {
//First, I make a query from my db to get the urls
// that I want the webscraper to loop through.
const ref = scheduleRef.get()
.then((snapshot) => {
snapshot.docs.forEach((doc) => {
const scheduleGame = doc.data()
const boxScoreUrl = scheduleGame.boxScoreURL
//Inside the forEach I call the request
// as a function with the url passed in
updatePlayerLogs("https://" + boxScoreUrl + "/");
});
})
.catch(err => {
console.log('Error getting schedule', err);
});
function updatePlayerLogs (url){
//Here I'm not sure on how to set these options
// to make sure the request stays open but I have tried
// lots of different things.
const options = {
uri: url,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
request(options)
.then(($) => {
//Below I loop through some table data
// on the dom with cheerio. Every loop
// in here needs to be written to firebase individually.
$('.stats-rows').find('tbody').children('tr').each(function(i, element){
const playerPage = $(element).children('td').eq(0).find('a').attr('href');
const pts = replaceDash($(element).children('td').eq(1).text());
const reb = replaceDash($(element).children('td').eq(2).text());
const ast = replaceDash($(element).children('td').eq(3).text());
const fg = replaceDash($(element).children('td').eq(4).text());
const _3pt = replaceDash($(element).children('td').eq(5).text());
const stl = replaceDash($(element).children('td').eq(9).text());
const blk = replaceDash($(element).children('td').eq(10).text());
const to = replaceDash($(element).children('td').eq(11).text());
const currentLog = {
'pts': + pts,
'reb': + reb,
'ast': + ast,
'fg': fgPer,
'3pt': + _3ptMade,
'stl': + stl,
'blk': + blk,
'to': + to
}
//here is the write
playersRef.doc(playerPage).update({
'currentLog': currentLog
})
.catch(error =>
console.error("Error adding document: ", error + " : " + url)
);
});
})
.catch((err) => {
console.log(err);
});
};
//Here I call response.send() to finish the function.
// I have tried doing this lots of different ways but
// whatever I try the response is being sent before all
// docs are written.
response.send("finished writing logs")
});
Everything I have tried either results in a deadline exceeded error (possibly because of quota limits which I have looked into but I don't think I should be exceeding) Or some unexplained error where the code doesn't finish executing but shows me nothing in the logs.
Please help, is there a way to use async/await in this scenario that I am not understanding? Is there a way to use promises to make this elegant?
Many thanks,
Maybe have a look at something like this. It uses Bluebird promises and the request-promise library
const Promise = require('bluebird');
var rp = require('request-promise');
const urlList = ['http://www.google.com', 'http://example.com']
async function getList() {
await Promise.map(urlList, (url, index, length) => {
return rp(url)
.then((response) => {
console.log(`${'\n\n\n'}${url}:${'\n'}${response}`);
return;
}).catch(async (err) => {
console.log(err);
return;
})
}, {
concurrency: 10
}); //end Promise.map
}
getList();
I'm new to Node.js, Mongoose, and testing in this environment. I have the following schema declared in a separate file.
Issue = mongoose.model("Issue", {
identifier: String,
date: String,
url: String,
name: String,
thumbnailURL: String
});
Then I have this method which simply returns all of the Issue instances in the MongoDB collection.
function issues(request, response) {
response.setHeader('Content-Type', 'text/json');
Issue.find().sort('date').exec(function(error, items) {
if (error) {
response.send(403, {"status": "error", "error:": exception});
}
else {
response.send(200, {"issues": items});
}
});
}
I've gotten this far through experimentation, and now I want to test it, but I've run into a problem. How do I go about testing it, without setting up a MongoDB connection, etc. I know that I can set all that stuff up, but that's an integration test. I want to write unit tests to test things like:
Does the function set the content type correctly
Does the function sort by the date field
Does the function return a 403 when an error occurs?
... and so on
I'm curious to see how I could refactor my existing code to make it more unit testable. I've tried maybe creating a second function that's called through, accepting the response and Item schema objects as parameters, but it doesn't feel right. Anyone have any better suggestions?
Mongoose model (your Issue) returns a new instance of the Query object. The new query instance has access to the exec method through the prototype. (mongoose 3.8~)
If you want to return an error you can write:
sinon.stub(mongoose.Query.prototype, "exec").yields({ name: "MongoError" }, null);
Using mocha with chaijs and sinonjs in my node code something like this method works for me:
var should = require('chai').should(),
sinon = require('sinon'),
mongoose = require('mongoose');
it('#issues() handles mongoosejs errors with 403 response code and a JSON error message', function (done) {
// mock request
var request = {};
// mock response
var response = {};
response.setHeader = function(header) {};
response.send = function (responseCode, jsonObject) {
responseCode.should.equal(403);
jsonObject.stats.should.equal('error');
// add a test for "error:": exception
done();
}
var mockFind = {
sort: function(sortOrder) {
return this;
},
exec: function (callback) {
callback('Error');
}
}
// stub the mongoose find() and return mock find
mongoose.Model.find = sinon.stub().returns(mockFind);
// run function
issues(request, response);
});
I'm not sure how to test the Content-Type, and I haven't tested this code myself, but I'm happy to help out if it doesn't work. It seems to make sense to me. Basically we just created a callback so we could move the response.send out of the actual custom logic, then we can test via that callback. Let me know if it doesn't work or make sense. You can use the links that the other guys posted to prevent having to connect to the db.
Issue = mongoose.model("Issue", {
identifier: String,
date: String,
url: String,
name: String,
thumbnailURL: String
});
function issues(callback, request, response) {
Issue.find().sort('number').exec(function(error, items) {
if (error) {
callback(403, {"status": "error", "error:": exception});
}
else {
callback(200, {"issues": items});
}
});
}
//Note: probably don't need to make a whole new `sender` function - depends on your taste
function sender(statusCode, obj) {
response.setHeader('Content-Type', 'text/json');
response.send(statusCode, obj);
}
//Then, when you want to implement issues
issues(sender, request, response);
//The tests - will depend on your style and test framework, but you get the idea
function test() {
//The 200 response
issues(function(code, obj) {
assert(code === 200);
assert(obj.issues === ??); //some testable value, or just do a truthy test
//Here, you can also compare the obj.issues item to the proper date-sorted value
});
//The error response
issues(function(code, obj) {
assert(code === 403);
assert(code.status === 'error');
});
}
A good place to start would be:
Investigate the concepts around stubs and mocks, and test doubles.
Check out Sinon.js which is the Mocking framework of choice for Node.JS