I asked a question a few months ago, to which Meteor seems to have the answer.
Which, if any, of the NoSQL databases can provide stream of *changes* to a query result set?
How does Meteor receive updates to the results of a MongoDB query?
Thanks,
Chris.
You want query.observe() for this. Say you have a Posts collection with a tags field, and you want to get notified when a post with the important tag is added.
http://docs.meteor.com/#observe
// collection of posts that includes array of tags
var Posts = new Meteor.Collection('posts');
// DB cursor to find all posts with 'important' in the tags array.
var cursor = Posts.find({tags: 'important'});
// watch the cursor for changes
var handle = cursor.observe({
added: function (post) { ... }, // run when post is added
changed: function (post) { ... } // run when post is changed
removed: function (post) { ... } // run when post is removed
});
You can run this code on the client, if you want to do something in each browser when a post changes. Or you can run this on the server, if you want to say send an email to the team when an important post is added.
Note that added and removed refer to the query, not the document. If you have an existing post document and run
Posts.update(my_post_id, {$addToSet: {tags: 'important'}});
this will trigger the 'added' callback, since the post is getting added to the query result.
Currently, Meteor really works well with one instance/process. In such case all queries are going through this instance and it can broadcast it back to other clients. Additional, it polls MongoDB every 10s for changes to the database which were done by outside queries. They are plans for 1.0 to improve the scalability and hopefully allow multiple instances to inform each one about changes.
DerbyJS on the other hand is using Redis PubSub.
From the docs:
On the server, a collection with that name is created on a backend Mongo server. When you call methods on that collection on the server,
they translate directly into normal Mongo operations.
On the client, a Minimongo instance is created. Minimongo is essentially an in-memory, non-persistent implementation of Mongo in
pure JavaScript. It serves as a local cache that stores just the
subset of the database that this client is working with. Queries on
the client (find) are served directly out of this cache, without
talking to the server.
When you write to the database on the client (insert, update, remove),
the command is executed immediately on the client, and,
simultaneously, it's shipped up to the server and executed there too.
The livedata package is responsible for this.
That explains client to server
Server to client from what I can gather is the livedata and mongo-livedata packages.
https://github.com/meteor/meteor/tree/master/packages/mongo-livedata
https://github.com/meteor/meteor/tree/master/packages/livedata
Hope that helps.
Related
I have about 20 documents currently in my collection (and I'm planning to add many more probably in the 100s). I'm using the MongoDB Node.js clients collection.foreach() method to iterate through each one and based on the document records go to 3 different endpoints: two APIs (Walmart and Amazon) and one website scrape (name not relevant). Each document contains the relevant data to execute the requests and then I update the documents with the returned data.
The problem I'm encountering is the Walmart API and the website scrape will not return data toward the end of the iteration. Or at least my database is not getting updated. My assumption is that the foreach method is firing off a bunch of simultaneous requests and either I'm bumping up against some arbitrary limit of simultaneous requests allowed by the endpoint or the endpoints simply can't handle this many requests and ignore anything above and beyond its "request capacity." I've ran some of the documents that were not updating through the same code but in a different collection that contained just a single document and they did update so I don't think it's bad data inside the document.
I'm running this on Heroku (and locally for testing) using Node.js. Results are similar both on Heroku instance and locally.
If my assumption is correct I need a better way to structure this so that there is some separation between requests or maybe it only does x records on a single pass.
It sounds like you need to throttle your outgoing web requests. There's a fantastic node module for doing this called limiter. The code looks like this:
var RateLimiter = require('limiter').RateLimiter;
var limiter = new RateLimiter(1, 1000);
var throttledRequest = function() {
limiter.removeTokens(1, function() {
console.log('Only prints once per second');
});
};
throttledRequest();
throttledRequest();
throttledRequest();
I need a legacy java application to pull information from a meteor's collection.
Ideally, I would need a simple service where my app would be able to download the latest list of items prices. A scenario like going on (through an http GET):
www.mystore.com/listOfPrices
would return a json with an array
[{"item":"beer", price:"2.50"}, {"item":"water":, price:"1"}]
The problem is that I cannot make a meteor page printing the result "as is" because meteor assumes the client supports javascript. Note that I do plan to implement the java DDP client in a latter stage but here I would like to start with a very simple service.
Idea: I thought of doing my own Node.js request aside of the running meteor service in order to retrieve a snapshot of the collection. Then this request would be using a server based javascript DDP client in order to subscribe and filter to then return the collection once loaded as a jSON document (array).
Any idea on how to achieve this ?
Looks like you want to provide a REST interface. See the MeteorPedia page on REST for how to expose collection data. It might be as simple as
prices = new Mongo.Collection('prices');
// Add access points for `GET`, `POST`, `PUT`, `DELETE`
HTTP.publish({collection: prices}, function (data) {
// here you have access to this.userId, this.query, this.params
return prices.find({});
});
I'm trying to decide between two methods for inserting a new document to a collection from the client using Meteor.js. Call a Server Method or using the db API directly.
So, I can either access the db api directly on the client:
MyCollection.insert(doc)
Or, I can create a new Server Method (under the /server dir):
Meteor.methods({
createNew: function(doc) {
check(doc, etc)
var id = MyCollection.insert(doc);
return project_id;
}
});
And then call it from the client like this:
Meteor.call('createNew', doc, function(error, result){
// Carry on
});
Both work but as far as I can see from testing, I only benefit from latency compensation (the local cache updating and showing on the screen before the server responds) if I hit the db api directly, not if I use a Server Method, so my preference is for doing things this way. But I also get the impression the most secure approach is to use a Method on the server (mainly because Emily Stark gave it as an example in her video here) but then the db api is available on the client no matter what so why would a Server Method be better?
I've seen both approaches taken when reading source code elsewhere so I'm stumped.
Note. In both cases I have suitable Allow/Deny rules in place:
MyCollection.allow({
insert: function(userId, project){
return isAllowedTo.createDoc(userId, doc);
},
update: function(userId, doc){
return isAllowedTo.editDoc(userId, doc);
},
remove: function(userId, doc){
return isAllowedTo.removeDoc(userId, doc);
}
});
In short: Which is recommended and why?
The problem was that I had the method declarations under the /server folder, so they were not available to the client and this broke latency compensation (where the client creates stubs of these methods to simulate the action but in my case could not because it couldn't see them). After moving them out of this folder I am able to use Server Methods in a clean, safe and latency-compensated manner (even with all my Allow/Deny rules set to false - they do nothing and only apply to direct db api access from the client, not server).
In short: don't use the db api on the client or allow/deny rules on the server, forget they ever existed and just write Server Methods, make sure they're accessible to both client and server, and use these for crud instead.
I'm trying to stream some syslog data into Meteor collections via node.js. It's working fine, but the Meteor client polling cycle of ~10sec is too long of a cycle for my tastes - I'd like it be be ~1 second.
Client-side collection inserts via console are fast and all clients update instantly, as it's using DDP. But a direct MongoDB insert from the server side is subject to the polling cycle of the client(s).
So it appears that for now I'm relegated to using DDP to insert updates from my node daemon.
In the ddp-client package example, I'm able to see messages I've subscribed to, but I don't see how to actually send new messages into the Meteor collection via DDP and node.js, thereby updating all of the clients at once...
Any examples or guidance? I'd greatly appreciate it - as a newcomer to node and Meteor, I'm quickly hitting my limits.
Ok, I got it working after looking closely at some code and realizing I was totally over-thinking things. The protocol is actually pretty straight forward, RPC ish stuff.
I'm happy to report that it absolutely worked around the server-side insert delay (manual Mongo inserts were taking several seconds to poll/update the clients).
If you go through DDP, you get all the real-time(ish) goodness that you've come to know and love with Meteor :)
For posterity and to hopefully help drive some other folks to interesting use cases, here's the setup.
Use Case
I am spooling some custom syslog data into a node.js daemon. This daemon then parses and inserts the data into Mongo. The idea was to come up with a real-timey browser based reporting project for my first Meteor experiment.
All of it worked well, but because I was inserting into Mongo outside of Meteor proper, the clients had to poll every ~10 seconds. In another SO post #TimDog suggested I look at DDP for this, and his suggestion looks to have worked perfectly.
I've tested it on my system, and I can now instantly update all Meteor clients via a node.js async application.
Setup
The basic idea here is to use the DDP "call" method. It takes a list of parameters. On the Meteor server side, you export a Meteor method to consume these and do your MongoDB inserts. It's actually really simple:
Step 1: npm install ddp
Step 2: Go to your Meteor server code and do something like this, inside of Meteor.methods:
Meteor.methods({
'push': function(k,v) { // k,v will be passed in from the DDP client.
console.log("got a push request")
var d = {};
d[k] = parseInt(v);
Counts.insert(d, function(err,result){ // Now, simply use your Collection object to insert.
if(!err){
return result
}else{
return(err)
}
});
}
});
Now all we need to do is call this remote method from our node.js server, using the client library. Here's an example call, which is essentially a direct copy from the example.js calls, tweaked a bit to hook our new 'push' method that we've just exported:
ddpclient.call('push', ['hits', '1111'], function(err, result) {
console.log('called function, result: ' + result);
})
Running this code inserts via the Meteor server, which in turn instantly updates the clients that are connected to us :)
I'm sure my code above isn't perfect, so please chime in with suggestions. I'm pretty new to this whole ecosystem, so there's a lot of opportunity to learn here. But I do hope that this helps save some folks a bit of time. Now, back to focusing on making my templates shine with all this real-time data :)
According to this screencast its possible to simply call the meteor-methods declared by the collection. In your case the code would look like this:
ddpclient.call('/counts/insert', [{hits: 1111}], function(err, result) {
console.log('called function, result: ' + result);
})
First of all, i am very much a newbie to all the technologies mentioned in this post.
I am working on something, where I have sensors, which would send their reading via HTTP post. The sensor would send the sensed value periodically over http as XML.
I found a link here that explains how to create REST API.
Now in the link above it is quite clear untill the point, the author installs mongoDB. But after that point things get complex and the author didn't give an explanation what is happening in the code after.
What I am not able to figure out is,
How to create a database in node.js using mondgoDB and expose this database over http, for the sensors to send the readings.
How can I access this database in my URI's?
How can I access the date and time the data was added onto database.
I would really appreciate any help.
I am no node.js expert but the same rules apply across the board for this kind of stuff.
You must understand that your database will not be accessible directly. Instead it will be accesses from node.js. You can add a --rest option to MongoDBs startup which will start a self contained RESTlet within the mongod program, but this is probably not an awesome idea here.
As far as I can see your jkust confused about the layers, which is common in this scenario, so to explain:
Your sensors will POST data (I would probably change that to JSON format, it is more expressive and smaller than XML) out to your node.js server running on, i.e. 81.187.205.13
It will post to whatever destination your rest function to deal with this data is running, i.e. /someawesomecontroller/notsuchagoodfunction
That function (as described by the tutorial you linked) will then pick up this POST, parse it and use the default method within node.js (via the driver) to insert into MongoDB. You can see the guy who wrote that tutorial doing that in the later partsd, i.e.:
exports.findById = function(req, res) {
var id = req.params.id;
console.log('Retrieving wine: ' + id);
db.collection('wines', function(err, collection) {
collection.findOne({'_id':new BSON.ObjectID(id)}, function(err, item) {
res.send(item);
});
});
};
So really now all you need are some tutorials on how the MongoDB driver in node.js works, here is a nice starting place: Do you know any tutorial for mongoDB in nodeJS?
Hope it helps,