I have a database which have approx 600000 records.
I 'm using the sails.js but
when I'm fetching the data with waterline approach It takes very long time to fetch 600000 records (approx 17sec) and It has limited query i.e It doesn't have Accessing Join Tables. so I join take the result of two query and then filter the data that's by It take lot's of time.
So I decided to use MongoDB with sails instead of waterline and I'm wondering if I can somehow use the Blueprint API without being linked to a Waterline model.
How to use the MongoDB instead of the waterline?
If you want to use the Sails models api, you can overwrite the blueprint methods in the controller.
So to overwrite, say, User Model, create the following functions in UserController.js:
find, create, update, destroy
find will override 'get api/v1/user'
create will override 'post api/v1/user'
update will override 'put api/v1/user'
destroy will override 'delete api/v1/user'
Once inside the controller, you can run a native query on Mongo like so:
In UserControllelr.js
find: function (req, res) {
var packet = req.params.all();
// packet now has all url and posted parameters
User.native(function (err, UserCollection) {
if(err) {
//handle error
}
else {
// here you can run any native mongo Query using UserCollection
// eg:
UserCollection.aggregate(
{"$match": {"gender": "Male"} },
{"$group": { "_id": "$socialNetwork", "count": {"$sum":1} } },
function (err, results) {
//do stuff with results
})
}
})
}
Hope this helps.
Related
I am using Loopback for my project. I created a Model 'test' and a remote-method 'createOrder' for the test model.
'use strict';
module.exports = function (Test) {
/**
*
* #param {number} amount
* #param {Function(Error)} callback
*/
Test.createOrder = function (amount, callback) {
// Add log to the logs collection
callback(null);
};
};
I am using MongoDB as datasource.
{
"db": {
"name": "db",
"connector": "memory"
},
"paymentDS": {
"host": "localhost",
"port": 27017,
"url": "",
"database": "test",
"password": "",
"name": "testDS",
"user": "",
"useNewUrlParser": true,
"connector": "mongodb"
}
}
I want to insert logs into logs collection from the createOrder remote-method defined in test model.
How can I do that?
If you have model for your Logs collection, you can do:
Test.createOrder = function(amount, callback) {
// ...
Test.app.models.Log.create(/*...*/)
// ...
};
Otherwise, you could access logs collection directly from db connector:
Test.createOrder = function(amount, callback) {
// ...
Test.app.datasources.db.collection("logs").insert(/**/);
// ...
};
In order to create custom CRUD operations, you can use loopback remote methods implementation. Every model in loopback is inherited with Persistent class, you can use its default methods to perform CRUD operations from remote methods. Check my examples below,
Create Operation
Below example will create a document in MongoDB and also insert a record in Log collection.
Test.createOrder = function (amount, callback) {
// Inserting object into database
Test.create({amount: amount, /*data to insert*/}, function(err, data) {
// Adding logs to the logs collection
Test.app.models.Log.insert({ message: "order created", /*data to insert*/}, function(err, data) {
callback(null);
});
});
};
There are more handy methods in the persistent model which can be used to perform create operations in different cases like upsert, findOrCreate, save etc.
Read Operation
Below example will retrieve several documents from MongoDB on the basis of filters applied and also insert a record in Log collection.
Test.findOrders = function (amount, callback) {
// Finding objects from database
Test.find(/** filters - Optional **/ { where: {}, limit: 10, fields: [], include: [], order: "", skip : 1}, function(err, data) {
console.log(data);
// Adding logs to the logs collection
Test.app.models.Log.insert({ message: "orders finded", /*data to insert*/}, function(err, logData) {
callback(data);
});
});
};
Here filters are completely optional. where is used to specify where conditions, you can use its sub-filters (like gt, lt, nin, etc) to get more condition-specific records. fields can use to select specific columns from the collection, include can be used to join different collections on the basis of relations. Remaining filters can be understandable from their names. For more information on filters, you can check the link mentioned here.
There are also available many useful methods in the persistent model which can be used to perform read operations in different cases like find, findOne etc.
Update Operation
Below example will update documents in MongoDB and also insert a record in Log collection.
Test.updateOrder = function (amount, callback) {
// Updating object into database
Test.updateAttribute(/* where filter */{id: 1}, {amount: 10, /*data to update*/}, function(err, data) {
// Adding logs to the logs collection
Test.app.models.Log.insert({ message: "order updated", /*data to insert*/}, function(err, data) {
callback(null);
});
});
};
You can every sub filter of where filter to perform the update operation. Again, there are many handy methods are available in the persistent model which can be used in several different scenarios (like updateAttribute, updateAttributes, createUpdates, bulkUpdate etc).
Delete Operation
Below example will delete a document from MongoDB and also insert a record in Log collection.
Test.deleteOrder = function (amount, callback) {
// Deleting object from database
Test.destroyById(123, function(err, data) {
// Adding logs to the logs collection
Test.app.models.Log.insert({ message: "order deleted", /*data to insert*/}, function(err, logData) {
callback(data);
});
});
};
Persistent Model also contains a different type of methods to perform delete operation based on different scenarios (like destroyAll, destroyById).
Official documentation on CRUD operations
You can also find complete documentation on the CRUD operations in loopback from the below-mentioned URLs:
https://loopback.io/doc/en/lb3/Querying-data.html
https://loopback.io/doc/en/lb3/Creating-updating-and-deleting-data.html#deleting-data
Below link contains all available methods of the persistent model which can be useful to apply several BLs,
http://apidocs.loopback.io/loopback/#persistedmodel
There are also several methods available to perform bulk operations in the above link.
Logging
There ar already several extensions are available which can be connected easily with loopback to perform logging operations some of them are listed below:
loopback-component-logger (link),
This can be useful to log operations in console or in database.
I am creating a webapp for users to query data from mongodb. I have front end using react and backend with nodejs hooking up mongodb.
I have included all relevant matching conditions under aggregate function. However I would like to have the effect of commenting out certain key-value pair, (by this I mean nullifying the effect of the query field condition). For example,
router.get('/getmongo', function(req, res) {
MongoClient.connect(process.env.DB_CONN, function(err, db) {
if (err) throw err;
db.aggregate([
{
$match:{
'employmentStatus':/employed|unemployed/i,
'gender':{$in:["male","female"]},
'age':{$gte:20, $lte:30}
}
},
{ "$group": {
"_id": "$employmentStatus",
"count": { "$sum": 1 }
}}
]).limit(4).toArray(function(err, docs) {
if (err) throw err;
res.send(JSON.stringify(docs))
db.close();
})
})
Let say on front end, the user does not select the option of age, which means there is no specification from the user's end on the age criteria. Usually for this case, users would comment out or remove the whole line of age query on MongoDB GUI, leaving only employmentStatus and gender as the only two criteria to query. I was looking into mongoAPI but can't find any useful tool to replicate that effect. Any idea or suggestion?
(I am using Mongo v2.3.3 by the way)
Instead of adding conditions in the Query at MongoDB side, you should build your query conditionally based on user input.
Whenever user selects the columns, build the $match clause using those columns and values. In that way, it will be quite dynamic.
Example PseudoCode:
var queryObject = {};
if(age.selected)
queryObject.add(ageCondition)
if(employmentStatus.selected)
queryObject.add(employmentStatusCondition)
if(gender.selected)
queryObject.add(genderCondition)
.....
......
You can build some logic like this in Node.js.
In SQL, I can update a field with inline calculation like this-
UPDATE user SET count=count+1 WHERE id=userID
It requires only one database operation.
But, in waterline, my only option now is to query the user, change the value, and save back into the database.
User.findOne(userID).exec(function(err, user) {
if(user) {
user.count++;
user.save(function(err) {
if(!err) console.log("success");
});
}
});
It requires double database access. How can I reduce the overhead?
You can execute native querys with waterline so I would go that route.
User.query("query string here", function (err,result){//Handle errors and result here});
http://sailsjs.org/documentation/reference/waterline-orm/models/query
So I have a model that is for a recipe where it has a relation of 'ingredients' and that is just an array of ObjectIds. When I run the following query on mongo shell it works fine and returns all my data.
Example model :
{
"name": "...",
"_id": ObjectId("530ca903746515c0161e6b9f"),
"ingredients": [
ObjectId("53069363ff7447a81a3a7a1d"),
ObjectId("53069363ff7447a81a3a7a17")
]
}
Query:
db.drink.find({"ingredients":{"$in":[ObjectId("53069364ff7447a81a3a7a87"), ObjectId("530fb948c1a3ff480d58e43c")]}});
Using sails.js though their waterline orm, they don't really have a way to query this though or at least through any possible google search that I can find. So trying to use the native driver I have something like the following -
var ings = new Array();
for (var i in req.body) {
ings.push(new ObjectID(req.body[i].toString()));
}
Drink.native(function(err, collection){
if(err){
return res.send(err, 500);
} else {
collection.find({"ingredients":{"$in":ings}}).toArray(function(err, data){
console.log(data);
});
}
});
The thing is the data array returned in the callback is always empty. If I check the 'ings' array it is an array of objectids so I am not sure why it won't return any data. If I remove the json object in the 'find' function it does return everything. Anyone have any idea how to make this query return data when using sails.js?
The code above actually is working it was an internal data issue on the mongodb where IDs were not matching between relations. When the relations were rebuilt all is working as it should be.
In my application im using node.js with redis database.How can i save many json objects in redis.
db.save({
description:'sdsd',userId:'324324',url:'http://www.abc.com/abc.html',appId:'123456'
}, function (err, res) {
if(err){
console.log(err);
return;
}
else{
console.log(res);
}
});
In couch db we can save the json objects again and again as document by using the above code.How to do this in redis.From their documentation i came to know below code save the json objects
client.hmset("hosts", "mjr", "1", "another", "23", "home", "1234");
Again i want to save the other json object in same "hosts" like below
client.hmset("hosts", "mjr", "2", "another", "33", "home", "1235");
How can i do this.
Redis storage model is different from CouchDB. In Redis, everything gets accessed by its key, so it all depends how you plan to retrieve your data.
So if you'd like to be able to retrieve data by userId, use this as the key.
redis.set('324324', {
description:'sdsd',
url:'http://www.abc.com/abc.html',
appId:'123456'
});
But if you need to retrieve a piece of data using more than one piece of the data, then redis may not be suitable.
In some cases, you may be able to use some tricks, so that to be able to query on both userId and appId, you could use 324324:123456 as the key, and query using
GET 324324:*
to get all apps for one user
or
GET *:123456
to get all users for a given app.