mongo change stream with load balancing
Can any one help as how we can achieve mango change stream with load balanceing server..?
we are working on micro service architecture facing issue on production when i go with load balanceing the same code is deployed over the 4 server when i perform any operation on single server the change stream trigger is fired form all the 4 server.
So what i want it should be trigger from same server where the operation are performed
Thanks in advance
Change stream is a database-level concept. Data is inserted/updated/deleted from the database, this produces change events. Any number of subscribers can subscribe to the change events and do whatever they want to do with the changes. Each subscriber is notified of every change event.
A change stream is not meant to inform the application that originated the change of the said change. This is redundant - the application already knows what it did.
Consider rephrasing your question to explain what you are trying to accomplish better.
Related
I'm using mongo watch() to subscribe to change stream events. I've noticed that the change stream events automatically stopped without throwing any specific error and become idle. Then have to restart the server to listen to the change stream again.
I'm not able to find out the specific reason for this strange behavior.
We are using Nodejs server. mongoose for db connection and watch.
If any one of you faced the same issue please guide me on that. We have 1 primary node and 2 secondary node cluster and hosted in mongodb atlas.
The collection.watch(...) method has to be called on the collection on every server restart. Common mistake is to call it once upon the creation of the collection. However, the database does not maintain reference to the result of this call as it does for other calls such as the collection.createIndexes(...).
Change streams only notify on data changes that have persisted to a majority of data-bearing members in the replica set. This ensures that notifications are triggered only by majority-committed changes that are durable in failure scenarios.
Change stream events stop working when a node fails in a replica set
I am trying to solve a problem where I am listening on a Mongo Changestream but in a production environment, we have multiple instances of the service running so in each instance I get the 'on change' event which I need to process.
But I want to process the change ONLY ONCE but here it is being processed multiple times(Number of instances of the service)
I need a way to ensure that each change Event is processed only once.
Thought of using Redis but then there is a chance of collision when two instances try to add the same on change event at the same time.
Please refer the diagram below for a rough idea of my architecture.
About once a minute, I need to cache all orderbooks from various cryptocurrency exchanges. There are hundreds of orderbooks, so this update function will likely never stop running.
My question is: If my server is constantly running this orderbook update function, will it block all other server functionality? Will users ever be able to interact with my server?
Do I need to create a separate service to perform the updating, or can Node somehow prioritize API requests and pause the caching function?
My question is: If my server is constantly running this orderbook
update function, will it block all other server functionality? Will
users ever be able to interact with my server?
If you are writing asynchronously, these actions will go into your eventloop and your node server would pick next event from eventloop while these actions are being performed. If you have too many events like this, your event queue would be long and user would face really slow response or may even get a timeout
Do I need to create a separate service to perform the updating, or can
Node somehow prioritize API requests and pause the caching function?
Node only consumes event from the event queue. There are no priorities.
From the design perspective, you should look for options which can reduce this write load like bulkCreate/edit or if you are using redis for cache, consider redis pipeline
This is a very open ended question much of which depends on your system. In general your server should be able to handle concurrent requests, but there are some things to watch out for.
Performance costs. If the operation to retrieve and store data requires too much computational power, then it will cause strain on all requests processed by the server.
Database connections. The server spends a lot of time waiting for database queries to complete. If you have one database connection for the entire application, and this connection is busy, they will have to wait until the database connection is free. You may want to look into database connection 'pooling'.
I currently have a volttron agent that periodicially downloads some data from the web in the form of a csv. I would like to use the DataPublisher example to take that csv data and push it to pubsub. However, from looking at the code, it seems like the DataPublisher is designed to run once, automatically, as soon as the agent starts up.
So my question them becomes, is there a way to start up the datapublisher from the original agent (which would itself have some sort of timer or loop)? I would then also need to stop it afterwards.
If I can't do this, my alternatives seem to be modifying the datapublisher to work on a schedule, or altering my other agent to publish the data from the csvs to pubsub.
Any assistance would be greatly appreciated.
The way I would do this is to expose an RPC method on the data publisher that either accepts a filename to publish. When this is called it would change the filename to publish and "start" the publishing of the data. The data publisher agent would always be running so there isn't a true restarting of the publisher.
The other agent (The one the downloaded the data?) would then just need to "kick off" the publishing through the rpc call.
This sounds like a very good feature that could be committed back to the VOLTTRON repository if you saw fit.
Is there anyone using the nice pinoccio from www.pinocc.io ?
I want to use it to post data into an apache couchdb using node.js. So I'm trying to poll data from the pinnocio API, but I'm a little lost:
schedule the polls
do long polls
do a completely different approach
Any ideas are welcome
Pitt
Sure. I wrote the Pinoccio API, here’s how you do it
https://gist.github.com/soldair/c11d6ae6f4bead140838
This example depends on the pinoccio npm module ~0.1.3 so make sure to npm install again to pick up the newest version.
you don't need to poll because pinoccio will send you changes as they happen if you have an open connection to either "stats" or "sync". if you want to poll you can but its not "real time".
sync gives you the current state + streams changes as they happen. so its perfect if you
only need to save the changes to your troop while your script is running. or show the current and last known state on a web page.
The solution that replicates every data point we store is stats. This is the example provided. Stats lets you read everything that has happened to a scout. Digital pins for example are the "digital" report. You can ask for data from a specific point in time or just from the current time (default). Changes to this "digital" report will continue streaming live as they happen, until the "end" time is reached, or if "tail" equals 0 in the options passed to stats.
hope this helps. i tested the script on my local couch and it worked well. you would need to modify it to copy more stats from each scout. I hope that soon you will be able to request multiple reports from multiple scouts in the same stream. i just have some bugs to sort out ;)
You need to look into 2 dimensions:
node.js talking to CouchDB. This is well understood and there are some questions you can find here.
Getting the data from the pinoccio. The API suggests that as long as the connection is open, you get data. So use a short timeout and a loop. You might want to run your own node.js instance for that.
Interesting fact: the CouchDB team seems to work on replacing their internal JS engine with node.js