I'm working on an interview project for which I have to add an endpoint that lets me POST an array of products (listings) and it should create them (MongoDB + Mongoose) or update them accordingly. The problem is I'm clearly not dealing with Promises properly and I'm getting a timeout on my test.
Here's the spec:
it.only('should create listings or update them if they already exist, incrementing the quantity with the difference ' +
'between the current sold_quantity and the initial_sold_quantity', (done) => {
var iPhone = { ... };
var samsung = { ... };
request(app).post('/listings/myEndpoint').send([iPhone, samsung]).expect(200, {
created: 1,
updated: 1
}, done);
});
exports.myEndpoint = (req, res) => {
var listings = req.body;
var created, updated = 0;
listings.forEach(reqListing => {
Listing.findOne({ listing_id: reqListing.listing_id })
.then(foundListing => {
if (!foundListing) {
var newListing = reqListing;
newListing.initial_sold_quantity = newListing.sold_quantity;
Listing.create(newListing);
created++;
} else {
var newQuantity = reqListing.sold_quantity - foundListing._doc.initial_sold_quantity;
if (foundListing._doc.quantity != newQuantity) {
foundListing._doc.quantity = newQuantity;
foundListing.save();
updated++;
}
}
});
return {
created: created,
updated: updated
};
});
};
THINGS I'VE TRIED:
Giving it more time. I tried changing the default timeout for Mocha tests but it doesn't really matter if it's 2 seconds or 20, it'll still timeout.
Isolating the update vs the creation. Really doesn't matter either if I'm only updating a product or if I'm only creating one, it'll still timeout.
Removing the logic. As far as I've checked it doesn't really matter what happens inside the if/else blocks because it'll still give me a timeout. So even if the code looks like this:
exports.myEndpoint = (req, res) => {
var listings = req.body;
var created, updated = 0;
listings.forEach(reqListing => {
Listing.findOne({ listing_id: reqListing.listing_id })
.then(foundListing => {
if (!foundListing) {
console.log("creating");
} else {
console.log("updating");
}
});
return {
created: created,
updated: updated
};
});
};
it'll still timeout.
After asking some questions in the Nodeiflux Discord server I managed to find a solution, maybe not the prettiest because it doesn't make use of async/await but I'm not supposed to change the style of the project too much so I'll leave it without async/await.
First, the silly problem which came after fixing the post's question:
var created = 0, updated = 0;
instead of not initializing created.
Second, making use of forEach with Promises inside didn't make too much sense because it would discard whatever was returning inside so I put the return outside the forEach clause and changed the forEach iteration for a map instead. I also made use of Promise.all() to get all promises to resolve before returning:
exports.upsert = (req, res) => {
var listings = req.body;
var created = 0, updated = 0;
var allowedArrayLength = 50;
return Promise.all(listings.map(reqListing =>
Listing.findOne({
listing_id: reqListing.listing_id
})
.then(foundListing => {
if (!foundListing) {
createListing(reqListing);
created++;
} else {
var prevQuantity = foundListing.quantity;
if (updateListing(reqListing, foundListing).quantity > prevQuantity) {
updated++;
}
}
})
)).then(() => ({ created: created, updated: updated }));
};
Related
I'm making a website which tracks kill statistics in a game but I can only access 50 key value pairs per request, so to make sure that my data is accurate I want to make a request about every 30 seconds.
I feel like I may have gone wrong at some stage in the implementation. Like, maybe there is a way to make requests that doesn't involve using
express.get('/route/', (req, res) => { //code })
syntax and I just don't know about it. In short, what I want is for the database to be updated every 30 seconds without having to refresh the browser. I've tried wrapping my get request in a function and putting it in set interval but it still doesn't run.
const express = require('express');
const zkillRouter = express.Router();
const axios = require('axios');
const zkillDbInit = require('../utils/zkillTableInit');
const sqlite3 = require('sqlite3').verbose();
zkillDbInit();
let db = new sqlite3.Database('zkill.db');
setInterval(() => {
zkillRouter.get('/', (req, res)=>{
axios.get('https://zkillboard.com/api/kills/w-space/', {headers: {
'accept-encoding':'gzip',
'user-agent': 'me',
'connection': 'close'
}})
.then(response => {
// handle success
const wormholeData = (response.data)
res.status(200);
//probably not the best way to do this but it's fine for now.
for (let i = 0; i < Object.keys(wormholeData).length; i++) {
const currentZKillId = Object.keys(wormholeData)[i]
const currentHash = Object.values(wormholeData)[i]
let values = {
$zkill_id: currentZKillId,
$hash: currentHash
};
//puts it in the database
db.run(`INSERT INTO zkill (zkill_id, hash) VALUES ($zkill_id, $hash)`,
values
, function(err){
if(err){
throw(err)
}
})
}
})
})
}, 1000)
module.exports = zkillRouter;
One thing I have considered is that I don't necessarily need this functionality to be part of the same program. All I need is the database, so if I have to I could run this code separately in say, the terminal as just a little node program that makes requests to the api and updates the database. I don't think that would be ideal but if it's the only way to do what I want then I'll consider it.
clarification: the .get() method is called on zkillRouter which is an instance of express.Router() declared on line two. This in turn links back to my app.js file through an apiRouter, so the full route is localhost:5000/api/zkill/. That was a big part of the problem, I didn't know you could call axios.get without specifying a route, so I was stuck on this for a while.
I fixed it myself.
(edit 4)
I was using setInterval wrong.
I just got rid of the router statement as it wasn't needed.
I definitely need to tweak the interval so that I don't get so many sql errors for violating the unique constraint. Every 5 minutes should be enough I think.
Don't throw an error.
function myfunction(){
axios.get('https://zkillboard.com/api/kills/w-space/', {headers: {
'accept-encoding':'gzip',
'user-agent': 'me lol',
'connection': 'close'
}})
.then(response => {
// handle success
const wormholeData = (response.data)
for (let i = 0; i < Object.keys(wormholeData).length; i++) {
const currentZKillId = Object.keys(wormholeData)[i]
const currentHash = Object.values(wormholeData)[i]
let values = {
$zkill_id: currentZKillId,
$hash: currentHash
};
db.run(`INSERT INTO zkill (zkill_id, hash) VALUES ($zkill_id, $hash)`,
values
, function(err){
if(err){
return console.log(i);
}
})
}
})
}
setInterval(myfunction, 1000 * 30)
Here, I am currently learning how to use MongoDB with NodeJS, but here is 1h or 2h blocked on internet looking now for a solution with 2 errors that I have and impossible to find the problem here is my code (Sorry, I am French, in the console.log, the sentences are in fr):
var MongoClient = require('mongodb').MongoClient
const mongodbUrl = "mongodb://localhost:27017/"
let client = new MongoClient(mongodbUrl, {useNewUrlParser: true})
let essai
var randomstring = '';
test()
function test(){
var chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXTZabcdefghiklmnopqrstuvwxyz-*/+.,?;:!§ù%$*£¤¨^=+})°]à#ç^_\è`-|(['{#é~&²";
var string_length = 14;
randomstring = '';
for (var i=0; i<string_length; i++) {
var rnum = Math.floor(Math.random() * chars.length);
randomstring += chars.substring(rnum,rnum+1);
}
//console.log(randomstring)
CheckBDD(randomstring)
//setTimeout(1000)
}
function CheckBDD(randomstring){
client.connect((err, client) => {
if (err) throw err
//console.log("Connexion à la bdd avec succès")
const dbName = "wordlist"
let db = client.db("wordlist")
var collection = 'wordlist';
db.collection('wordlist').findOne({wordlist: randomstring}, (err, essai) => {
if (err) throw err
//console.log(essai)
if (essai == null)
{
db.collection('wordlist').insertOne( { wordlist: randomstring} );
console.log("Valeur inconnue:" + randomstring)
} else {
console.log("Valeur déjà connue:" + randomstring)
}
test()
});
});
}
Errors:
the options [servers] is not supported
the options [caseTranslate] is not supported
From what I found, it will come from the connection options but... There you go. I hope you can help me thank you very much in advance,
Owzlaa
I just started having the exact same "errors" appearing in some of my database requests. It isn't all of the time though. I had exactly the same call in two different places in my code: one inside my server's connection callback and another in a module used inside of a route controller. The one inside my server's connection callback did not trigger the error whereas the other one did.
Doing a console log on the client object showed exactly the same options. So what was the difference? The server connection callback was only a "read operation" whereas the one in the other module eventually calls a method to perform an update on the collection itself.
I modified the code to have a conditional in it so when one case is true it modifies the collection and when the other case is true it does not. Guess what? The second case, where there's no modification to the collection, does not trigger the error. It would seem that doing some sort of write causes this thing to trigger.
I am not sure I want to call it an error. It's more like a warning because it is non-blocking (at least in my case). It also doesn't seem to trigger any of my error handling.
I still don't have a solution for making it go away other than perhaps we have to make copies of the database object... I haven't tried it yet though.
// Express code : NOTE - client is the MongoClient exported from somewhere else
let db;
client.connect()
.then(c => {
if (c) {
db = c.db(DATABASE_NAME);
req.client = c;
return verifiers.verifyForFiles(db);
} else {
req.errStatus = 500;
throw new Error('Database error');
}
})
.then(checkResult => {
const { hasCollection, hasSchema } = checkResult;
if (!hasCollection) {
/** triggers errors */
return collections.createFilesCollection(db);
} else {
if (hasSchema) {
/** does not trigger errors */
return;
} else {
/** triggers errors */
return collections.updateFilesCollection(db);
}
}
})
.then(() => next())
.catch(next);
// more Express stuff
Experts,
It seems that yelp recently changed their REST API to limit the amount of requests you can make per second. I've tried using setTimeout and various sleep functions with no success. I believe it has to do with setTimeout although. I only get a few responses back and a slew of TOO_Many_Requests_Per_Second. Also, I'm using the Node.js Fusion API Client. Any help would be appreciated. Thanks in advance.
Here is the code below as I'm getting the Yelp URL from my Parse Server, and I want to get the Yelp Business Name response:
'use strict';
var Parse = require('parse/node');
Parse.initialize("ServerName");
Parse.serverURL = 'ParseServerURL';
const yelp = require('yelp-fusion');
const client = yelp.client('Key');
var object;
var Business = Parse.Object.extend("Business");
var query = new Parse.Query(Business);
query.notEqualTo("YelpURL", "Bus");
query.find({
success: function(results) {
for (var i = 0; i < results.length; i++) {
object = results[i];
//I belive a setTimeout block needs to come somewhere in here. Tried many places but with no success.
client.business(object.get('YelpURL')).then(response => {
console.log(response.jsonBody.name);
}).catch(e => {
console.log(e);
});
}
},
error: function(error) {
alert("Error" + error.code + " " + error.message);
}
});
Use query each, which will iterate over each object and perform the requests in a sequence rather than all more or less at once:
query.each(
function(object) {
return client.business(object.get('YelpURL')).then(response => {
console.log(response.jsonBody.name);
});
}
).catch( e => {
res.json('error');
});
One cool thing about this is that it'll automatically propagate the error from client.bussiness() call if there is one to the catch block at the bottom. It will iterate over the objects one at a time, and since we "return" the results of the client.business() call, it's not going to move on to the next object until you've gotten the response. query.each() will also iterate over every object in a collection that meets your query criteria, so you don't have to worry about limits.
Im not quite sure if this is what your looking for, but you can retrieve up to 50 records per request, in the example below will return 20 business names within that zip code, or you can tweak it a little to return all that data for those businesses, does this help:
app.get('/:id', (req, res) => {
let zipcode = req.params.id;
let names = [];
let searchRequest = {
term: 'Business', // or for ex. food
limit: 20, //set the number of responses you want up to 50
radius: 20000, // 20 miles
location: zipcode
};
client.search(searchRequest)
.then(response => {
response.jsonBody.businesses.map(elem => {
names.push(elem.name);
})
res.json(names); // business names only
//or
//res.json(response.jsonBody.businesses) //all details included with business name
}).catch(e => {
res.json('error');
});
})
I want to call function from my object in express route. This function should call mongoose query, then run next, next etc. - all needed operations.
Here is my example route:
var MailSender = require('../../libs/mailer');
router.get('/mailer/test', function (req, res, next) {
MailSender.getPending();
});
And mailer file:
(here include all required)
module.exports = {
currentMails : {},
getPending : function() {
MailObj.find()
.limit(10)
.exec(this.blockPending);
},
blockPending : function(err, mail) {
currentMails = {};
mail.forEach(function(data) {
let mailId = mongoose.Types.ObjectId(data._id);
currentMails[mailId] = data;
});
MailObj.update({ _id: { $in: Object.keys(currentMails) } }, { block: 1 }, {multi: true}, function() {
// Doesn't work
this.myNextFunc();
});
},
myNextFunc : function() {
console.log("Yep!");
}
}
getPending - it works great and call blackPending with query results.
blockPending - works greats, I can prepare ids and update records
But... myNextFunc() doesn't work and I can't call any object function from this scope (console says that they are undefined). I know, that I make something wrong but... what?
I'ld like to encapsule related functions in such objects and run inside as callbacks. What am I doing wrong?
As far as you are using Mongoose, why don't you take profit of it, and you update each mail in the loop?? It is less efficient, but maybe as a first approach it deserves:
var numUpdates = 0;
mail.forEach(function(data) {
data.block = 1;
data.save(function(err, mailSaved) {
//check error
if(numUpdates ++ >= mail.length) {
this.myNextFunc();
}
})
});
I'm using CouchDB 1.5 and trying to fix some values in documents with a rather simple request. I simply get a document, modify a value in it and then put it back immediately. Given that my database has low usage, I don't expect this simple operation to produce a conflict. And yet, the 85 documents belonging to 85 different users all fail to update with conflict errors, with no apparent reasons.
Here's the code I'm using:
var _ = require('lodash');
var PouchDB = require('pouchdb');
var couchdbUrl = 'https://USER:PASS#DOMAIN.COM';
var usersDb = new PouchDB(`${couchdbUrl}/_users`, {
skip_setup: true
});
usersDb.query('faulty_users/object_username', {
include_docs: true
})
.then((userDocs) => {
userDocs.rows
.forEach(function(userDoc) {
userDb = new PouchDB(`${couchdbUrl}/user%2F${userDoc.doc.hoodieId}`);
userDb.get('accountvalues/default', {
conflicts: true
})
.then((doc) => {
console.log(doc._id, doc._rev, doc._conflicts);
doc.values.accountValues.username = userDoc.doc.name.replace(/^user\//, '');
userDb.put(doc)
.catch((e) => {
console.log(userDoc.doc.hoodieId, e);
});
});
});
});
I've read and read again the documentation of PouchDB and I can't find what I'm doing wrong. I hope the error will be pretty obvious to someone here :-)