I am connected to a websocket using node.js ws module.
Each time i get response with event called "add_items" i want to save it to mLab cloud database.
Websocket response looks like this
{"event":"add_items","data":[
{"id":["13635936204"],
"m":"name1",
"p": 14.00}
{"id":["13635936220"],
"m":"name2",
"p": 50.00}
]}
What would be the best way to connect to database and save new data ?
Currently i use .forEach method to loop through array , than create objects for each new item and save it to databse using db.collection.insert() method. But the problem is that i might get over 50 items per sec and sometimes i get error "mongodb connection timed out". Is it possible to use mongoose module to simplify this process ?
You can use MongoDB's insertMany:
db.collection.insertMany(websocketEvent.data);
Where websocketEvent is the response received via Websocket. Check insertMany syntax and info here.
UPDATE:
Maybe you can loop through the array calling updateOne and decide to update or insert using the arrayFilters field, something like this:
websocketEvent.data.forEach(event => {
db.collection.updateOne(
{ _id: ObjectID(event.id[0])},
{ $set: { m: event.m, p: event.p }},
{
upsert: true,
arrayFilters: [ <filterdocument1>, ... ]
}
)
}
Check the docs for info on the arrayFilters field for updateOne.
Related
I have an array of objects and I want to store them in a collection using only one I/O operation if it's possible. If any document already exists in the collection I want to replace it, or insert it otherwise.
These are the solutions that I found, but doesn't work exactly as I want:
insertMany(): this doesn't replace the document that already exists, but throws exception instead (This is what I found in the Mongodb documentation, but I don't know if it's the same as mongoose).
update() or updateMany() with upsert = true: this doesn't help me as well, because here I have to do the same updates to all the to stored documents.
There is no replaceMany() in mongodb or mongoose.
Is there anyone how knows any optimal way to do replaceMany using mongoose and node.js
There is bulkWrite (https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/), which makes it possible to execute multiple operations at once. In your case, you can use it to perform multiple replaceOne operations with upsert. The code below shows how you can do it with Mongoose:
// Assuming *data* is an array of documents that you want to insert (or replace)
const bulkData = data.map(item => (
{
replaceOne: {
upsert: true,
filter: {
// Filter specification. You must provide a field that
// identifies *item*
},
replacement: item
}
}
));
db.bulkWrite(bulkData);
You need to query like this:
db.getCollection('hotspot').update({
/Your Condition/
}, {
$set: {
"New Key": "Value"
}
}, {
multi: true,
upsert: true
});
It fulfils your requirements..!!!
I have a collection of about 1M documents. Each document has internalNumber property and I need to get all internalNumbers in my node.js code.
Previously I was using
db.docs.distinct("internalNumber")
or
collection.distinct('internalNumber', {}, {},(err, result) => { /* ... */ })
in Node.
But with the growth of the collection I started to get the error: distinct is too big, 16m cap.
Now I want to use aggregation. It consumes a lot of memory and it is slow, but it is OK since I need to do it only once at the script startup. I've tried following in Robo 3T GUI tool:
db.docs.aggregate([{$group: {_id: '$internalNumber'} }]);
It works, and I wanted to use it in node.js code the following way:
collection.aggregate([{$group: {_id: '$internalNumber'} }],
(err, docs) => { /* ... * });
But in Node I get an error: "MongoError: aggregation result exceeds maximum document size (16MB) at Function.MongoError.create".
Please help to overcome that limit.
The problem is that the native driver differs from how the shell method is working by default in that the "shell" is actually returning a "cursor" object where the native driver needs this option "explicitly".
Without a "cursor", .aggregate() returns a single BSON document as an array of documents, so we turn it into a cursor to avoid the limitation:
let cursor = collection.aggregate(
[{ "$group": { "_id": "$internalNumber" } }],
{ "cursor": { "batchSize": 500 } }
);
cursor.toArray((err,docs) => {
// work with resuls
});
Then you can use regular methods like .toArray() to make the results a JavaScript array which on the 'client' does not share the same limitations, or other methods for iterating a "cursor".
For Casbah users:
val pipeline = ...
collection.aggregate(pipeline, AggregationOptions(batchSize = 500, outputMode = AggregationOptions.CURSOR)
I'm trying a very simple CRUD API with the MEAN stack. I entered several documents into a mongolab sandbox db. I could do a GET on all documents and they would all be returned. Then I tested GET by ID:
router.route('/api/v1/resources/:resource_id')
// get the resource with that id (accessed at GET http://localhost:8080/api/resources/:resource_id)
.get(function(req, res) {
Resource.findById(req.params.resource_id, function(err, resources) {
if (err)
res.send(err);
res.json(resources);
});
});
And it simply wouldn't work. I kept getting null. But the document existed. I could see it when I would do a GET on all documents.
Then I got another document to return, finally.
The difference? The record that returned had an id in this format:
{
"_id": { "$oid":"1234567890abc"}
}
But records that did not return had this:
{
"_id": "1234567890abc"
}
Can Anyone explain this to me? I entered the data with Postman and I didn't do anything different between the entries.
What is $oid?
What creates the $oid?
Why does that nesting matter for mongoose's findById()?
Thanks!
$oid is from Strict MongoDB Extended JSON.
All your queries to MongoDB database that contains _id conditions should wrap _id in ObjectId function like the following:
db.resources.findOne({_id: ObjectId("507c35dd8fada716c89d0013")};
MongoLab provides UI for querying to MongoDB via JSON syntax. But you can't use ObjectId in JSON due specification restrictions.
So, MongoLab uses Strict MongoDB Extended JSON for alias ObjectId() -> $oid.
{"_id": {"$oid":"507c35dd8fada716c89d0013"})
Same $oid you see in the output because MongoLab UI uses JSON also.
Mongoose automatically converts a string _id to MongoDB query so you don't need doing it manually. The following queries are equivalent:
Resource.findById("507c35dd8fada716c89d0013");
Resource.findById(new mongoose.Types.ObjectId("507c35dd8fada716c89d0013"));
I have a database which have approx 600000 records.
I 'm using the sails.js but
when I'm fetching the data with waterline approach It takes very long time to fetch 600000 records (approx 17sec) and It has limited query i.e It doesn't have Accessing Join Tables. so I join take the result of two query and then filter the data that's by It take lot's of time.
So I decided to use MongoDB with sails instead of waterline and I'm wondering if I can somehow use the Blueprint API without being linked to a Waterline model.
How to use the MongoDB instead of the waterline?
If you want to use the Sails models api, you can overwrite the blueprint methods in the controller.
So to overwrite, say, User Model, create the following functions in UserController.js:
find, create, update, destroy
find will override 'get api/v1/user'
create will override 'post api/v1/user'
update will override 'put api/v1/user'
destroy will override 'delete api/v1/user'
Once inside the controller, you can run a native query on Mongo like so:
In UserControllelr.js
find: function (req, res) {
var packet = req.params.all();
// packet now has all url and posted parameters
User.native(function (err, UserCollection) {
if(err) {
//handle error
}
else {
// here you can run any native mongo Query using UserCollection
// eg:
UserCollection.aggregate(
{"$match": {"gender": "Male"} },
{"$group": { "_id": "$socialNetwork", "count": {"$sum":1} } },
function (err, results) {
//do stuff with results
})
}
})
}
Hope this helps.
I want to push to the starting of array while updating(a single document). I am using findOneAndUpdate but it seems like mongoose doesn't support $position operator. I can achieve this by using the native driver by Model.collection.update
{
'$push': {
post_IDs: {
'$each': [articles],
'$position': 0
}
}
}
but the native driver doesn't return the document updated. That's why I can't use it here. Is there any way to push to the start of the array while receiving the updated document in the callback - apart from using find() followed by save()?
Mongoose does not support the new operators directly, but the underlying driver dependency should be recent enough if your mongoose is a recent release.
You get the underlying "node native" driver functions by using the .collection accessor on the model:
Model.collection.findAndModify(
{ field: "value" },
[],
{
"$push": {
"post_IDs": {
"$each": [articles],
"$position": 0
}
}
},
{ new: true},
function(err,doc) {
}
);
The method there is the .findAndModify() from the native driver. The syntax is a little different. First is "query" then a "sort" array, then the "update" document. Also the options are set to return the "new" document, which is what the mongoose methods default to, but this one does not.