I have a NodeJs consumer sending data to Elasticsearch at a rate of 2k TPS more or less.
I need to store the Requests as I receive them, and if there is any response, later on, I need to update the info on the request with some data from the response. The thing is that due to the hight TPS, I'm having a lot of issues where the response arrives in Elasticsearch before the request etc.. and this creates a Version conflict on the _doc. this is the part of my node code that does the upsert. I need some help to optimizing this code. Thanks a lot in advance.
sendToElasticSearch(index, type, id, body, cb) {
out('starting sendToElasticSearch()');
var me = this;
me.client.exists({
index: index,
type: type,
id: id
}, function(err, exists) {
if (err) {
cb(err)
} else {
if (exists === true) {
out('exists. doing update.');
// update existing document
me.client.update({
index: index,
type: type,
id: id,
body: body
}, function(err, resp) {
if (err) {
cb(err);
} else {
cb(null, resp);
}
});
} else {
out('adding new document');
// add new document
me.client.create({
index: index,
type: type,
id: id,
body: body
}, function(err, resp) {
if (err) {
cb(err);
} else {
cb(null, resp);
}
});
}
}
});
}
sendToElasticSearch(index, type, id, body, cb) {
var self = this;
function onDone (err, exists) {
if (err)
return cb(err);
var do = exists ? 'update' : 'create';
self[do]({index, type, id, body}, cb);
}
self.client.exists({index, type, id}, onDone);
}
Related
I have a PUT request that I'm trying to have hit the backend, but for some reason, it never reaches it. What's odd is the if(req.body.bidderId){} hits no problem, but not the if(req.body.watchingGroup){}
The watching angular service uses identical code to the bidderId so I don't know what's different between the two where only one would reach the endpoint? Whats wrong with the addToWatchList call? I did testing and both console.log statements in code block return the correct value. So the data is ready to be passes, but is never received.
console.log("MADE IT TO LISTINGS BACKEND");
never outputs for watchingGroup scenario
watching.service.ts
addToWatchList(id: string, watchingGroup: string[]) {
const watching: Watching = {
id: id,
watchingGroup: watchingGroup
};
console.log("Here are the values being passed to backend");
console.log(watching.id);
console.log(watching.watchingGroup);
console.log(watching);
return this.http.put(`http://localhost:3000/api/listings/${watching.id}`, watching,
);
}
app.js
app.put("/api/listings/:id", (req, res) => {
console.log("MADE IT TO LISTINGS BACKEND");
if (req.body.biddingGroup) {
console.log("bidding has been received");
Post.findByIdAndUpdate(
{ _id: req.params.id },
{
currentBid: req.body.currentBid,
lastBidTimeStamp: Date.now(),
bidderId: req.body.bidderId,
auctionEndDateTime: req.body.auctionEndDateTime,
biddingGroup: req.body.biddingGroup,
lastBidTimeStamp: req.body.lastBidTimeStamp
},
function(err, docs) {
if (err) res.json(err);
else {
console.log(docs);
}
}
);
}
if (req.body.watchingGroup) {
console.log("watching has been received");
Post.findByIdAndUpdate(
{ _id: req.params.id },
{
watchingGroup: req.body.watchingGroup
},
function(err, docs) {
if (err) res.json(err);
else {
console.log(docs);
}
}
);
}
});
addToWatchList
addToWatchList(
auctionId: string,
watchingGroup: string[]
) {
this.watchItStatus = true;
this.userId = localStorage.getItem("userId: ");
var unique = watchingGroup.filter(function(elem, index, self) {
return index === self.indexOf(elem);
});
this.uniqueResult = unique;
watchingGroup.push(this.userId);
this.watchListService.addToWatchList(auctionId, this.uniqueResult);
}
As i suspected you're not subscribing to it. It's weird but you need to subscribe to it.
this.watchListService.addToWatchList(auctionId, this.uniqueResult).subscribe(
(res) => {
// Handle success response
console.log("SUCCESS");
},
(err) => {
// Handle error response
console.log("ERROR");
}
);
I am trying to retrieve data at multiple API endpoints simultaneously and aggregate the result to be sent back to the client as one response.
However, when trying to return data from async.parallel, I get an undefined result. I think this is because the result code is being executed before it is returned, but I am not sure.
I would like to pass an array of ids to the server, get their metadata, combine their metadata into one object, and send a single object back to the client.
function doSomething() {
for(let i=0; i<req.map.length; i++) {
stack[i] = (callback) => {
let data = subroute(req, res, req.map[i])
callback(null, data)
}
}
async.parallel(stack, (err, result) => {
if(err) {
// Handle error
} else {
console.log(result) // Logs undefined
res.json([result]) // Would like to send this result back to the client
}
})
}
function subroute(req, res, num) {
request({
url: 'https://example.com/api/endpoint/' + num
},
(error, response, body) => {
if(error) {
res.json({ "error": error })
} else {
let i = {
name: body.name,
age: body.age,
}
return i
}
})
}
How can I accumulate the results of many API endpoint responses on the server and send the back as one response to the client?
Change
let data = subroute(req, res, req.map[i])
callback(null, data)
to
subroute(req, res, req.map[i], callback)
Subsequently, change subroute to receive the callback and return the return
function subroute(req, res, num, cb) {
request({
url: 'https://example.com/api/endpoint/' + num
},
(error, response, body) => {
if(error) {
res.json({ "error": error })
cb(<whatever error you would want to pass>);
} else {
let i = {
name: body.name,
age: body.age,
}
cb(null, i);
}
})
}
By doing let data = subroute(req, res, req.map[i]) you are assigning the result of subroute call to data. Notice subroute does not have a return statement in the function body so data will always be undefined.
According to the docs I should get a data structure with the item as it was prior to the deletion (in case there was no error)
I do check there was no error but I get an empty object for data:
docClient.delete(params, (err, data) => {
if (err) {
console.error('Error tring to delete item:' + err);
callback(err, null); // error
} else if (!data.Items || data.Items.length == 0) {
console.info(JSON.stringify(data));
callback(null, null); // no items
} else {
console.info(JSON.stringify(data));
callback(null, data.Items[0].ExposeStartTimestamp);
}
});
Both prints empty json: {}
In order for the deleted data to appear in the response, the request should contain the attribute ReturnValues with value ALL_OLD.
var params = {
TableName: 'TableName',
Key: {
HouseId: houseId
},
ReturnValues: 'ALL_OLD'
};
I am a little confused with the req, res parameters for node and the best way to handle these when I'm using asynchronous calls. A function I currently have is supposed to add an Item to my DB, update some DB models accordingly, and then send a response saying the updates were successful. However, the functions that this function asynchronously calls can send an erroneous response if an error happens. If this happens, I get the error Can't set headers after they are sent, since I am trying to call res.send twice. Would appreciate someone's help in figuring out a more optimal way to handle errors. Thanks!
the main function:
item.author = req.user._id;
item.description = req.query.description;
item.rating = req.query.rating;
item.save()
.then(result => {
// update user's items
UserController.updateItems(req.user._id, result._id);
ItemController.updateItemRating(req.query.outingId, req.query.rating, res);
res.send(result);
})
.catch(error => {
res.send(error);
});
updateItemRating:
export const updateItemRating = (itemId, rating, res) => {
Item.findOne({ _id: itemId }).exec((err, item) => {
if (item === undefined || item === null) {
return res.status(404).send('Item not found; check item ID');
}
Item.findOneAndUpdate(
{ _id: itemId },
{ $set: { rating: rating },
},
(error, item) => {
if (error) {
res.status(404).send('Error updating item with new rating');
}
});
});
};
updateItems:
export const updateItems = (userId, itemId) => {
User.findOneAndUpdate(
{ _id: userId },
{ $push: { items: [itemId] } },
(err, user) => {
console.log('user' + user);
if (err) {
console.log('got an error in updateItems');
}
});
};
Function call to updateItems and updateItemRating both are asynchronous . Response send is getting called multiple time , and also not sure which method send is getting called first . To fix you problem I can suggest you to apply below technique :
Callback : You can pass callback as argument which will do res.send and same callback you can call on error or success condition .
UserController.updateItems(req.user._id, result._id,function(status,message){res.status(status).send(message);});
You can update item rating method like :
export const updateItemRating = (itemId, rating, callback) => {
Item.findOne({ _id: itemId }).exec((err, item) => {
if (item === undefined || item === null) {
callback(404,'Item not found; check item ID');
}
Item.findOneAndUpdate(
{ _id: itemId },
{ $set: { rating: rating },
},
(error, item) => {
if (error) {
callback(404,'Error updating item with new rating');
}else{
callback(200);
}
});
});
};
Async Module : You can synchronize you method call using this module.
Instead of having your update functions send the result, you should throw() an error, that way your outer function will catch the error and you can use that to return it.
Another way to think about this is that your outer function is handling the success case with a res.send(), so it should also be responsible for the res.send of the error case.
The less your database layer knows about the caller the easier it will be to reuse our code.
Create a custom error type to encapsulate the 404:
function NotFoundError(message) {
this.message = (message || "");
}
NotFoundError.prototype = new Error();
Then use that in your inner function:
export const updateItemRating = (itemId, rating, res) => {
Item.findOne({ _id: itemId }).exec((err, item) => {
if (item === undefined || item === null) {
throw new NotFoundError('Item not found; check item ID');
}
Item.findOneAndUpdate(
{ _id: itemId },
{ $set: { rating: rating },
},
(error, item) => {
if (error) {
throw new NotFoundError('Error updating item with new rating');
}
});
});
};
And your main becomes:
item.save()
.then(result => {
// update user's items
UserController.updateItems(req.user._id, result._id);
ItemController.updateItemRating(req.query.outingId, req.query.rating, res);
res.send(result);
})
.catch(error => {
if (error instanceof NotFoundError) {
res.status(404).send(error.message);
}
else {
res.send(error);
}
});
I have an array of documents with unique _id and I want to insert them to my database. Some of them already in db, and for those I want to update an array property (push in array an item). All of this I need to make asyncronuosly, so after all inserted/updated I want to write response back (with callback) to client than all ok or write an error. After googling on subject I've found this solution with async module I've tried to implement it for my case. Now my code looks like this:
function processUsers(arr, listName, callback) {
var users = global.db.collection('vkusers');
var q = async.queue(function(task, cb) {
console.log('upsert butch');
users.insert(task.doc, function(err, doc) {
if (err) {
users.update({
_id : task.doc._id
}, {
$addToSet : {
mLists : listName
}
}, function(error, result){ console.log(error); console.log(result); });
}
});
}, arr.length);
for ( var doc in arr) {
q.push({
doc : arr[doc]
}, function(err) {
if (err)
callback(err, null);
})
}
q.drain = function() {
// this is the queue's callback, called when the queue is empty,
// i.e. when all your documents have been processed.
console.log('drain');
callback(null, { result: "success", upserted: arr.length });
}
}
Callback has signature callback(error, result), arr - my array of documents. I've tested it and with database everything is OK, i am getting the right result. But callback, and q.drain never fired!
You need to call async.queue's callback (cb in your code) when your insert/update is complete. Something like this:
var q = async.queue(function(task, cb) {
console.log('upsert butch');
users.insert(task.doc, function(err, doc) {
if (err) {
users.update({
_id : task.doc._id
}, {
$addToSet : {
mLists : listName
}
}, function(error, result) {
console.log(error);
console.log(result);
cb(error); // Update finished; call cb and pass in "error" so that it can bubble up if it exists
});
} else {
cb(); // Insert succeeded; call cb
}
});
}, arr.length);