I have around 5,00,000 records in my database. I'm using DataTables to build an admin panel to manage the records.
I have Node.js as backed with MongoDB.
I have used this library - https://www.npmjs.com/package/datatables-query
So far on page load I have successfully loaded results a shown in the below Image.
Whenever I type something in the search box, I get 500 error as shown in the screenshot.
What could be the problem here?
Is DataTable a good option for showing grid with huge amount of data or is there any better option considering Node.js, Express and MongoDB combo?
Here is my server side code.
app.post('/getUsersData',function(req, res) {
var Model = require('./models/user'),
datatablesQuery = require('datatables-query'),
params = req.body,
query = datatablesQuery(Model);
query.run(params).then(function (users) {
var data = JSON.stringify(users);
// var data = JSON.stringify(users);
res.end(data);
}, function (err) {
res.status(500).json(err);
});
});
I have a table in MongoDB named User with 3 columns
1) Name
2) Email
3) Password
$(document).ready(function() {
var table = $('#datatable').DataTable({
// dom: 'Bfrtip',
processing: true,
serverSide: true,
order: [[1, 'asc']],
"aoColumnDefs": [ { "sClass": "hide_me", "aTargets": [ 0 ], visible: false } ], // first column in visible columns array gets class "hide_me"
ajax: {
url: "/getUsersData",
type: 'POST',
dataSrc: "data"
},
columns: [
{ data : "_id"},
{ data : "name" },
{ data : "email" },
{ data : "password" },
],
responsive: true
});
});
I would probably create an index on the data and search that index rather than searching the data itself. If you do however create a full text index you need to have a combination off all your cols in your collection and mongo allows only 1 full text index per collection.
As for alternatives, you could look into AWS's Elastic Search (which works just fine with MongoDB) or Sphinx Index (based on PostreSQL)
Edit:
I know this answer doesn't actually answer the question in the slightest but
I fear that the 500 error is not a memory issue in the application but rather on the DB (mongo is not like SQL, so don't design your applications like you would in SQL).
Some reading material, if you plan on changing db structure
http://rachbelaid.com/postgres-full-text-search-is-good-enough/
https://about.gitlab.com/2016/03/18/fast-search-using-postgresql-trigram-indexes/
The key here is figuring out what exactly went wrong on the DB or server side layer. The error message that you are alerting out is a generic message from the datatables library which points you to the following documentation: https://datatables.net/manual/tech-notes/7
The documentation linked to above says that this error will occur anytime a non-200 response is returned from the server. In this case you are explicitly throwing a 500 error in the error handler/callback of the datatables query call, but that doesn't give you/us the information we need!
The next step is to look at the response body in the network tab of your browser's console, since you are serializing the error as json and sending it along with your 500 response, or to console.log out the err in the error handler/callback in your node code. This will get you an understanding of what is actually going wrong, and with that information we can more accurately prescribe a solution. Post an update with whatever you log out please!
here is solution i tried to fix this issue:
goto ur jquery.datatable.min.js and then comment out the error message and you are good to go.
function K(a, b, c, d) {
// c = "DataTables warning: " + (a ? "table id=" + a.sTableId + " - " : "") + c;
// d && (c += ". For more information about this error, please see http://datatables.net/tn/" + d);
// if (b) E.console && console.log && console.log(c); else if (b = n.ext, b = b.sErrMode || b.errMode, a && r(a, null, "error", [a, d, c]), "alert" == b) alert(c); else {
// if ("throw" == b) throw Error(c);
// "function" ==
// typeof b && b(a, d, c)
// }
}
Related
I've started reading into creating Google Actions using Node.JS/Dialogflow/Firebase.
I have reached a big stumbling block, in trying to get a simple code running that would search a Firebase database for a certain value and then report back. For example from the JSON output, I would like to search for the applicationID and have the age passed back as the output.
I would be extremely grateful if someone can review my code and direct me in the right direction.
Table Structure
{
"groupA" : {
"applications" : {
"100" : {
"age" : 20,
"result" : "pass"
},
"200" : {
"age" : 25,
" result " : "pass"
},
"500" : {
"age" : 20,
" result " : "fail"
}
}
}
}
Node JS
return admin.database().ref('groupA').child('applications').orderByChild('applications').equalTo(500)
.once('value')
.then(acceptedApplicationsSnapshot => {
var id = acceptedApplicationsSnapshot.val().age;
var data = acceptedApplicationsSnapshot.val();
var theAge = acceptedApplicationsSnapshot.child("age").val();
agent.add('some random text' + theAge);
});
Within this example the value 500 should be searched with the age then given as the output.
You're ordering/filtering on property applications, but the value you're passing in is a key. So you'll want to use orderByKey instead of orderByChild:
return admin.database().ref('groupA').child('applications').orderByKey().equalTo(500)
A query may match multiple child nodes, so the above returns a snapshot with a list of results. Even when there's only one result, the query will return a list of one result:
{
"500": {
"age" : 20,
" result " : "fail"
}
}
This means that you'll need to loop over the results in your callback:
acceptedApplicationsSnapshot.forEach(appSnapshot => {
var theAge = appSnapshot.child("age").val();
agent.add('some random text' + theAge)
})
Since you know the exact key of the child node you want, you can also simply keep using child() to access the specific node:
return admin.database().ref('groupA').child('applications').child('500')
This will result in a snapshot of that specific node being returned, instead of the list-with-a-single-child-node of the previous snippet.
{
"age" : 20,
" result " : "fail"
}
And then you can use your existing code to retrieve the age from the snapshot.
To search for the applicationsID=500 and have the age passed back as the output, you can try this.
var db = admin.database();
var ref = db.ref("groupA").child("applications/500").once('value')
.then(function(dataSnapshot){
console.log(dataSnapshot.val().age);
});
More info here.
Edited: This should give you the age, in this case equals 20, for the parameter you pass in applications/500 in the code above.
Let me know if it helps.
In a Cosmos DB stored procedure, I'm using a inline sql query to try and retrieve the distinct count of a particular user id.
I'm using the SQL API for my account. I've run the below query in Query Explorer in my Cosmos DB account and I know that I should get a count of 10 (There are 10 unique user ids in my collection):
SELECT VALUE COUNT(1) FROM (SELECT DISTINCT c.UserId FROM root c) AS t
However when I run this in the Stored Procedure portal, I either get 0 records back or 18 records back (total number of documents). The code for my Stored Procedure is as follows:
function GetDistinctCount() {
var collection = getContext().getCollection();
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT VALUE COUNT(1) FROM (SELECT DISTINCT c.UserId FROM root c) AS t',
function(err, feed, options) {
if (err) throw err;
if (!feed || !feed.length) {
var response = getContext().getResponse();
var body = {code: 404, body: "no docs found"}
response.setBody(JSON.stringify(body));
} else {
var response = getContext().getResponse();
var body = {code: 200, body: feed[0]}
response.setBody(JSON.stringify(body));
}
}
)
}
After looking at various feedback forums and documentation, I don't think there's an elegant solution for me to do this as simply as it would be in normal SQL.
the UserId is my partition key which I'm passing through in my C# code and when I test it in the portal, so there's no additional parameters that I need to set when calling the Stored Proc. I'm calling this Stored Proc via C# and adding any further parameters will have an effect on my tests for that code, so I'm keen not to introduce any parameters if I can.
Your problem is caused by that you missed setting partition key for your stored procedure.
Please see the statements in the official document:
And this:
So,when you execute a stored procedure under a partitioned collection, you need to pass the partition key param. It's necessary! (Also this case explained this:Documentdb stored proc cross partition query)
Back to your question,you never pass any partition key, equals you pass an null value or "" value for partition key, so it outputs no data because you don't have any userId equals null or "".
My advice:
You could use normal Query SDK to execute your sql, and set the enableCrossPartitionQuery: true which allows you scan entire collection without setting partition key. Please refer to this tiny sample:Can't get simple CosmosDB query to work via Node.js - but works fine via Azure's Query Explorer
So I found a solution that returns the result I need. My stored procedure now looks like this:
function GetPaymentCount() {
var collection = getContext().getCollection();
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT DISTINCT VALUE(doc.UserId) from root doc' ,
{pageSize:-1 },
function(err, feed, options) {
if (err) throw err;
if (!feed || !feed.length) {
var response = getContext().getResponse();
var body = {code: 404, body: "no docs found"}
response.setBody(JSON.stringify(body));
} else {
var response = getContext().getResponse();
var body = {code: 200, body: JSON.stringify(feed.length)}
response.setBody(JSON.stringify(body));
}
}
)
}
Essentially, I changed the pageSize parameter to -1 which returned all the documents I knew would be returned in the result. I have a feeling that this will be more expensive in terms of RU/s cost, but it solves my case for now.
If anyone has more efficient alternatives, please comment and let me know.
Thanks for the help in advance.
I have the following in a mongodb collection
{
id: 12345,
userid: 9876,
CompName: PC-537,
LogonTime: 2015-07-1,
SessionName: Session1
}
What i am trying to do is Mapreduce the database down to where i can see how many sessions were allocated to each comp name, so instead of the comp name appearing multiple times in the collection, it will appear once, with the sessions associated with it appearing as values
Can someone please give me some pointers? I know how to mapreduce when using numbers (for example rainfall: 9mm) and having that tally up, but when its text how can i find reduce it so the PC's only appear once, and also either list all the sessions they had and the total sessions they had.
What I have tried so far is the following
var mapFunc = function() {
emit(this.CompName, this.SessionName);
};
var reduceFunc = function(keyComp, keySession) {
return Array.sum(keySession);
};
db.compcollection.mapReduce(
mapFunc,
reduceFunc,
{ out: "comp2" }
)
This gave me a result of the database reducing down to the session name not the CompName and also did not give a numeric count. So basically it gave me all the machines associated with a sessionname, not the reverse (which is what i want)
Can someone please help :)
In case anyone needs this in the future i have figured this out, see code below
var mapper = function() {
emit(this.CompName, 1);
};
var reducer = function(compname, counter) {
return Array.sum(count);
};
db.compcollection.mapReduce(mapper, reducer, { out: "comp15" }
)
I'm building an API using node express and mongodb, with mongoose.
I have a post resource that handles user posts, and would like to be able to perform various queries on the post resource.
For instance I have a functions as that returns all posts as follows:
// Gets a list of Posts
exports.index = function(req, res) {
console.log(req.query);
Post.findAsync()
.then(mUtil.responseWithResult(res))
.catch(mUtil.handleError(res));
};
I looking for a good way of processing any additional query params that might come with the request.
/posts will return all posts, but /posts?user=12 will return posts by user with id 12 and /posts?likes=12 will return posts with 12 or more likes.
How can I check for and apply the these query params to filter and return the results since they may or may not be present.
Thanks ;)
If user=12 means "users with id 12", how does likes=12 mean "likes greater than 12"? You need to be more descriptive with your queries. You can do that by passing an array of objects. Send your query in a way that can be interpreted like this:
var filters = [
{
param: "likes",
type: "greater"
value: 12
},
{
param: "user",
type: "equal",
value: "12"
}]
var query = Post.find();
filters.forEach(function(filter) {
if (filter.type === "equal") {
query.where(filter.param).equals(filter.value);
}
else if (filter.type === "greater") {
query.where(filter.param).gt(filter.value);
}
// etc,,,
})
query.exec(callback);
I am new to Node js and MongoDB (coming from an SQL background). I am trying to implement a function that removes all the records in a collection 'journeys' where the timestamp is older than 10 mins. This is what I have done so far:
connection.open(function(err, connection) {
var database = connection.db(DATABASE);
database.collection(JOURNEY_COLLECTION, {}, function(err, journeys) {
var now = new Date().getTime() / 1000 - 10 * 60;
journeys.remove({timestamp : ???}, function(err, result) {
if(err != null)
console.log('A refresh error occurred');
connection.close();
});
});
});
Am I on the right track? What should I be putting in place of the "???" ?
I'm sure someone'll post an answer that uses the example code, but I'm going to show you how to do it on a database level.
First, define a field that'll contain a Date. This can be either the creation time (method A) or when you want it removed (method B). I'll explain both, but method B is my favourite. It allows you to more easily modify the time to live. A field with the date set to null will not be removed.
Method A:
var someDocument = { "created": new Date() };
Method B:
var someDocument = { "expire": new Date(Date.now() + 10 * 60 * 1000) };
Add a TTL index on this field, like so:
Method A:
db.myCollection.ensureIndex( { "created": 1 }, { expireAfterSeconds: 600 } )
Method B:
db.myCollection.ensureIndex( { "expire": 1 }, { expireAfterSeconds: 0 } )
MongoDB runs a job every 60 seconds that looks for documents that should be removed. Due to this reason, you might get documents that are slightly stale.
(Note that I've named the field 'created' (method A) / 'expire' (method B), you can name them anything but I think this is more descriptive than just 'timestamp' for these examples )
You're on the right track.
Your "???" should be in the form {$lt:varName}
var tenMinutesOld = new Date()
tenMinutesOld.setMinutes(tenMinutesOld.getMinutes()-10)
journeys.remove({timestamp: {$lt:tenMinutesOld}}, function(err, result) {