I'm trying to work out how I can establish what I think needs to be a write-lock on a Mongo database during application startup.
I've got a setup whereby we have several (in this diagram just 2) Node API's that establish a connection to a Mongo replica set. Upon startup I want to be able to run some scripts against Mongo if it's an old version. For example:
MongoDB: v1 schema
Node App: expecting v3 schema
So during startup the Node App will run v2Upgrade.js and v3Upgrade.js or similar. However, I want to ensure that only 1 Node app can run this at any one time. So 2 questions:
Am I thinking about this in the right way?
How would I best create some sort of lock, so only 1 process runs these updates before the database is "ready"?
Related
I'm using mongo watch() to subscribe to change stream events. I've noticed that the change stream events automatically stopped without throwing any specific error and become idle. Then have to restart the server to listen to the change stream again.
I'm not able to find out the specific reason for this strange behavior.
We are using Nodejs server. mongoose for db connection and watch.
If any one of you faced the same issue please guide me on that. We have 1 primary node and 2 secondary node cluster and hosted in mongodb atlas.
The collection.watch(...) method has to be called on the collection on every server restart. Common mistake is to call it once upon the creation of the collection. However, the database does not maintain reference to the result of this call as it does for other calls such as the collection.createIndexes(...).
Change streams only notify on data changes that have persisted to a majority of data-bearing members in the replica set. This ensures that notifications are triggered only by majority-committed changes that are durable in failure scenarios.
Change stream events stop working when a node fails in a replica set
I am using moleculer micro services and postgres database with modules 'moleculer-db-adapter-sequelize' and 'Sequelize'. Every time i save any code moleculer service gets restarted at that time sequelize runs a DELETE query. How to stop it running that delete query?
As I can't add a comment, I ask this in the answer section.
We will need a bit more informations could you provide some sample of your code?
I suspect that you have a DELETE query in some part of your service lifecycle event.
I'm learning NodeJS and am trying to stick with the MVC architecture. I'm getting stuck on where to place those functions that update data from an outside source on a set loop, with a 30 second or so delay.
Example: I build an app that takes data from a API, Orders in this case, and stores it in a database. I can add orders to my database locally, and I want the orders database to be synchronized with the outside source mentioned previously, every 30 seconds.
My models directory will contain Order.js which includes an order schema and it will connect to MongoDB via Mongoose. My controller will have API endpoints for CRUD operations.
Where does the function go that refreshes the data from the server? In the controller? Then I would export that function so that I can set up the loop that updates the database in my app.js (or whatever I use to start the application)?
I recommend using something like node-cron to handle the setTimeout for you. It gives you the advantage of cron-like syntax to run your jobs on a schedule and will run while your node app is. I would put these jobs in a separate directory with node cron jobs. The individual node cron job can then import your MongoDB model. Your main application can then import index.js or something similar from the cronjobs dir which imports all your node cron jobs to bootstrap them on application startup.
I have been NodeJS as server side and MongoDB as our database. It really works great together.
Now I have added node-schedule library into our system , to call a function like a cron-job.
The process takes around hours to complete.
My issue is whenever cron is running , all users to my site gets No response fro server i.e database gets locked.
Stuck on the issue from a week , needs good solution to run cron , without affecting users using the site.
Typically you will want to write a worker and run the worker in a different entry point that is not part of your server. There are multiple ways you could achieve this.
1) Write a worker on another server to interact with your database
2) Write a service worker on another server that interacts with your api
3) Use the same server but setup a cronjob to execute the file that does the work at a specified time.
But you should not do this from the same entry point that your server is running on. You need a different execution file.
There is one thing you can do to run this where it will not bog down your server and that would be for your trigger for node-schedule to run a child process. https://nodejs.org/api/child_process.html
Is it possible to get a changelog of changes in mongodb , say from a given timestamp and then apply it at another instance of mongodb?
These 2 instances have same collection but changes to one is independent of the other.
Ideally the change log would be a transaction log of all the data changes that have happened from a given instance of time.
looks like the only way is to start the mongodb server in replica mode and get the oplog (as discussed with mongodb's solution architect - vigyan)
mongodb --replSet rs0
link to convert your standalone server to replica