I'm currently working on a app. This also includes some kind of group chats.
The users inside can make multiple votes, for example for kicking someone. The votes are all valid for 1 week. If all other users submit their opinion the vote gets deleted.So far so good.
I also want a logic, which deletes the vote automatically if it's expired.
So far I got the idea to store the expiration dates for the votes inside a database(MongoDB), sorted by their timestamp of expiration.
In NodeJs I'm always loading the vote with the smallest expiration date from the database.
Then I check how much time is left by subtracting the vote expiration date from the current Date
Date.now() - voteTmp;
Then I can set a timeout, which calls a function to delete the vote and automatically starts a new timeout for the next vote. Is it a problem to set a timeout with such a big number of seconds?
Do you have any better ideas?
Thank you:)
The node.js event loop is explained here:
when the event loop enters a given phase, it will perform any operations specific to that phase, then execute callbacks in that phase's queue until the queue has been exhausted or the maximum number of callbacks has executed.
On each iteration, the event loop checks for scheduled timers that satisfy the specified thresholds (delays) and executes their callbacks. Thus, the magnitude of delays for registered timers shouldn't matter.
However, in your scenario, there's a chance that you might accidentally register redundant or invalid timers (possibly after recovering from a crash). MongoDB supports (automatic) data expiration. You can instruct MongoDB to delete documents after a specified number of seconds has passed. That seems close enough to what you want to do.
Related
The title isn't accurate because based on what I have found in my research there doesn't seem to be a way to make a function atomic in nodejs, but I will lay out my problem to see if you people can come up with something that I have not been able to think about.
I am trying to setup a scheduler where I can set my appointment time slots say 1 hr long each and when someone makes an appointment I want to make sure that the time slot is not taken before scheduling it.
So for example I decide that I will be working from 9 am to 2 pm with a time slot of one hour. Then my schedule would be 9-10, 10-11, 11-12, 12-1, 1-2.
An appointment will come in with a start time of 11 and end time of 12. I need to make sure that slot isn't already taken.
I am using mongodb with nodejs and restify.
I understand that in my appointments collection I can set an index on a combination of values like start time and end time, as discussed here Creating Multifield Indexes in Mongoose / MongoDB.
But if I decide to change my time slot from 1 hour to say 1.5 hours then I will have scheduling conflicts as the start time and end time of entries in the database will not match up with the new interval
Currently I have a function which checks to make sure that the new appointment will not conflict but I am not sure if it will work out well when I have multiple requests coming in. This is a nodejs and restify app so basically an api with a mongodb that it talks to, to handle appointments.
I am running it with multiple workers, so I am worried that at a certain point two requests will come in at the same time, handled by two different workers for the same time slot. When my conflict checking function executes it will return saying that the slot is open for both of them since no appointment has been made yet and then there will be a scheduling conflict.
Any ideas on how to combat this, or is there something in the way javascript executes so that I shouldn't have to worry about it this? All input will be appreciated
Thanks!
I ended up using https://github.com/Automattic/kue, to queue my requests and added another endpoint where you can check the status of your request. So when you want to make an appointment your request ends up in the job queue, and you can then periodically check the status of your request. This way only one appointment request gets processed at a time so no concurrency issues.
I am saving a counter number in user storage.
I want to provide some content to the user which changes daily using this counter.
So every time the counter increases by 1 the content will change.
The problem is the timezone difference.
Is there anyway to run a function, daily which will increase this counter by 1. I could use setInterval() which is a part of the NodeJs library but that won't be an accurate "daily" update for all users.
User storage is only available to you as a developer when the Action is active. This data is not available once the Action is closed, so you wouldn't be able to asynchronously update the field. If you do want asynchronous access, I'd suggest using an external database and only storing the database row key in the user's userStorage. That way you can access the data and modify it whenever you want.
The setInterval method will run a function periodically, but may not work in the way you want. It only runs the function while the runtime is active. A lot of services will shut down a runtime after a period. Cloud Functions, for example, run sometimes but then will shut down when not used. Additonally, Cloud Functions can be run several times in parallel instances, executing a setInterval function several times in parallel. That would increment the counter more times than you want.
Using a dedicated Cron service would help reduce the number of simultaneous executions while also ensuring it runs when you want.
You are unable to directly access the user's timezone within the Action, meaning you won't be able to determine the end of a day. You can get the content to change every day, but it'll have some sort of offset. To get around this, you could have several cron jobs which run for different segments of users.
Using the conv.user.locale field, you can derive their language. en-US is generally going to be for American users, which generally are going to live in the US. While this could result in an odd behavior for traveling, you can then bucket users into a particular period of execution. Running the task overnight, either 1AM or 4AM they'll probably be unaware but know that it updates overnight.
You could use the location helper to get the user's location more precisely. This may be a bit unnecessary, but you could use that value to determine their timezone and then derive that user's "midnight" to put in the correct Cron bucket.
I am trying to figure out how to handle consistency in the database.
In scenario:
User A has an accounting document in the database include a balance field representing the amount of his current money. (supposed initially he has 100$)
My system has many methods to charge his account.
Suppose 2 methods occur at the same time, each method charges him for 10$, these steps occur concurrently in below orders:
Method 1 READ his balance and store in memory (100$)
Method 2 READ his balance and store in memory (100$)
... some business logics
Method 1 UPDATE his balance by subtracting variable in memory by 10 (100$ - 10$) and then save it
Method 2 UPDATE his balance by subtracting variable in memory by 10 (100$ - 10$) and then save it
This means he has been charged only 10$ instead of 20$.
I searched this situation a while and can not get it clear (sorry for my stupidity).
Really appreciate yours helps to enlighten my featherbrained. :)
You just discovered why financial transactions are complicated :-)
Have you ever wondered why it takes time for you to have an updated balance in your bank account? Or why you actually have two balances, instead of one?
That's because your account can actually go negative and (up to a certain point) that will be fine.
So in a real life scenario what happens is that you have a balance of 100$, you pay 10$ and until that transaction is processed and confirmed by the receiver, you still have your 100$. If you do 20 transactions of 10$ each, you'll be able to complete them because the system will most likely not be able to notice.
And honestly, it shouldn't. Think of credit cards, you might not have enough money now, but maybe you know you'll have enough when the credit is due.
So, the race condition you describe only works if you actually read the value and then update it.
There are a few approaches:
Read the current balance, and update the row using the old balance as a field in the where statement. This way if it updates no rows you know that you need to re-read and update.
Don't update the balance and only do it time-based, say once per hour. Yes, you might still have to do some checks, but the system will overall be more responsive.
Lock the database row as your first step. This would work but there's a chance that it will make the app slower.
Race condition you describe is low level design concern. With backend engine like Node that will handle the incomming request in first come first serve fashion you don't need to think about this case. Race condition you describe is not possible if you respect the order in which database update callbacks are fired. They are fired in the same order they have been issued in. So you should call next update only when the previous has finished. Promisses are great way to do this.
Using Event Sourcing, I have a domain in which aggregates should be updated from time to time. When I create an aggregate, I have an expiry time (this can be arbitrary) on it, and after that time I have to update some properties of the entity. (This can be forced using an UpdateCommand too.) I have few processes in mind:
After the aggregate creation, I store the aggregate ID and the expiry time in an RDBMS.
In a cron job I query the database for expired aggregates, and submit an UpdateCommand
Others include emitting UpdateCommands (or events?) from the read side.
Using a saga to coordinate updates, this is similar to the first. But either way, I have to store the expiry times.
So, I have to store the events and write into a database on the write side transactionally. However, I am not sure if creating a read-side for the write-side (?) is the correct solution in the DDD world, or is it applicable? What are the recommended solutions?
I also need to run some commands after some time expires.
For example, I need to emit a ContractExpiredEvent after 1 year (the ContractAggregate decides when but usually it is 1 year). The problem is that the Aggregate must be the one that decides when and what command to executes, so this is a Domain concern more than an Infrastructure one.
How I did that? I was inspired by Udi Dahan's video in which he introduce the term Timeout. Long story short, the Aggregate requests that a command should be send to itself after a period of time passes. It does that by yielding it from a command handler. The underlying CQRS framework gets that scheduled command and persists it in a special repository. Then, a cron job process all scheduled commands when their time comes.
There's well compatibility between ES and DDD.
However, I am not sure if creating a read-side for the write-side (?) is the correct solution in the DDD world, or is it applicable?
Yes, it's a part of domain aggregate in your case (if you talk about storing expiry times on write-side).
So, I have to store the events and write into a database on the write side transactionally.
I suggest you to use the saga for writing into a db.
John Carmack, 1998:
If you don't consider time an input value, think about it until you do -- it is an important concept
The pattern you should be looking for is that the real world (where time is) tells the aggregate the current time, and the aggregate decides whether or not to expire itself.
With that pattern in place, you can use any strategy you like for scheduling when the real world tells the aggregate what time it is.
You don't need immediately consistent scheduling in the aggregate, you just need some idempotent message handling and an "at least once" delivery process.
the aggregate has a method which can cause an update if it is necessary based on the current time, not blindly. At some time I have to fetch the right aggregate from the store, call that method and store the changes back (if any), or retry later, right?
Yes, that's the right idea.
Notice that if you call that method twice after the expiration time, the first call will load the history, append the expiration events, and store the updated history. The second call loads the history, can see that the aggregate is already expired, and retires without making any change to the history.
You can also use bi-temporal event sourcing. When events are stored, there are two dates:
the date when the event is added to the database (createdAt)
the date when the event has to be applied (validFrom)
The events are then applied in the order defined by validFrom property.
Using this, you can:
"fix the past" by adding a new event (createdAt = now and validFrom = now - x)
schedule events in the future by adding a new event (createdAt = now and validFrom = now + y)
I suggest to watch this great video of Thomas Pierrain at DDD Europe 2018: https://www.youtube.com/watch?v=xzekp1RuZbM
I've come up with a fancy issue of synchronization in node.js, which I've not able to find an elegant solution:
I setup a express/node.js web app for retrieving statistics data from a one row database table.
If the table is empty, populate it by a long calculation task
If the record in table is older than 15 minutes from now, update it by a long calculation task
Otherwise, respond with a web page showing the record in DB.
The problem is,
when multiple users issue requests simultaneously, in case the record is old, the long calculation task would be executed once per request, instead of just once.
Is there any elegant way that only one request triggers the calculation task, and all others wait for the updated DB record?
Yes, it is called locks.
Put an additional column in your table say lock which will be of timestamp type. Once a process starts working with that record put a now+timeout time into it (by the rule of thumb I choose timeout to be 2x the average time of processing). When the process stops processing update that column with NULL value.
At the begining of processing check that column. If the value > now condition is satisfied then return some status code to client (don't force client to wait, it's a bad user experience, he doesn't know what's going on unless processing time is really short) like 409 Conflict. Otherwise start processing (also ideally processing takes place in a separate thread/process so that user won't have to wait: respond with an appropriate status code like 202 Accepted).
This now+timeout value is needed in case your processing process crashes (so we avoid deadlocks). Also remember that you have to "check and set" this lock column in transaction because of race conditions (might be quite difficult if you are working with MongoDB-like databases).