We have a rare production issue on a Notes database. Our system assigns an ID to a request doc during submission (saving then submit, in real time). What happens is that sometimes 2 documents appear to have the same ID, which should not be the case.
The ID is in the format YYYY-MM-XXX, where YYYY is current year, MM is month in number, XXX is a number starting from 001 then beyond. The system, when assigning id, checks in a view where documents for the same months are there. If it does not see a document, it assigns 001 in the ID, else, it gets the latest document in the view, gets the number, then increment it by 1. The new number will be then assigned as ID.
How can I assure that during this process, I can assign a unique id based on the criteria above? The bug is so rare that we cannot simulate it again. Thank you for your help!
You could use the lock method of NotesDocument object to first lock the document before updating the number in it. In case another user comes in, the script would need to wait till the lock is released. Once the lock is released the next user can lock it down to update it.
Some code would be helpful to see where the problem lies, but it sounds like there is a race condition between two users submitting a document where they both look at the view at the same time.
There is a lot of time passing during the logic for looking at the view, getting the latest doc, incrementing it by one, storing that on the document, saving that document, and then updating the view so that the next time the view is checked the latest doc has a higher number. What you really need is something that wraps that all so it can only happen synchronously. That's easier said than done.
I'd suggest using the #UNIQUE formula, which is (I believe) guaranteed to be unique in place of the XXX part of the ID. If you were using the XXX part for sorting, perhaps you can still save that sequential number somewhere in the document and at worst case you'll have two documents with the same sorting key, which may be fine for your needs.
As per #Ken Pespisa's answer, #Unique is much more likely to give you a unique value, but it's not really guaranteed. If two users with similar names (i.e., same first initial, same first two letters of last name, and same last letter of last name) execute #Unique on two different client computers at exactly the same time (as per system clocks), then there can still be a collision. The probability is low, but not zero.
The definitive discussion of assigning unqiue sequential ids to Notes documents can be found in Andre Guirard's article here. In my opinion, the best technique is to defer assigning the unique sequential id, and allong an agent that runs on a server to create the id. That provides a true guarantee of uniqeuness (as long as you code it properly). The tradeoff is that the id is not immediately available.
Related
Our application creates/updates database entries based on an external service's webhooks. The webhook sends the external id of the object so that we can fetch more data for processing. The processing of a webhook with roundtrips to get more data is 400-1200ms.
Sometimes, multiple hooks for the same object ID are sent within microseconds of each other. Here are timestamps of the most recent occurrence:
2020-11-21 12:42:45.812317+00:00
2020-11-21 20:03:36.881120+00:00 <-
2020-11-21 20:03:36.881119+00:00 <-
There can also be other objects sent for processing around this time as well. The issue is that concurrent processing of the two hooks highlighted above will create two new database entries for the same single object.
Q: What would be the best way to prevent concurrent processing of the two highlighted entries?
What I've Tried:
Currently, at the start of an incoming hook, I create a database entry in a Changes table which stores the object ID. Right before processing, the Changes table is checked for entries that were created for this ID within the last 10 seconds; if one is found, it quits to let the other process do the work.
In the case above, there were two database entries created, and because they were SO close in time, they both hit the detection spot at the same time, found each other, and quit, resulting in nothing being done.
I've thought of adding some jitter'd timeout before the check (increases processing time), or locking the table (again, increases processing time), but it all feels like I'm fighting the wrong battle.
Any suggestions?
Our API is Django 3.1 with a Postgres db
Okay, this might not be a very satisfactory answer, but it sounds to me like the root of your problem isn't necessarily with your own app, but the webhooks service you are receiving from.
Due to inherent possibility for error in network communication, webhooks which guarantee delivery always use at-least-once semantics. A sender that encounters a failure that leaves receipt uncertain needs to try sending the webhook again, even if the webhook may have been received the first time, thus opening the possibility for a duplicate event.
By extension, all webhook sending services should offer some way of deduplicating an individual event. I help run our webhooks at Stripe, and if you're using those, every webhook sent will come with an event ID like evt_1CiPtv2eZvKYlo2CcUZsDcO6, which a receiver can use for deduplication.
So the right answer for your problem is to ask your sender for some kind of deduplication/idempotency key, because without one, their API is incomplete.
Once you have that, everything gets really easy: you'd create a unique index on that key in the database, and then use upsert to guarantee only a single entry. That would look something like:
CREATE UNIQUE INDEX index_my_table_idempotency_key ON my_table (idempotency_key);
INSERT INTO object_changes (idempotency_key, ...) VALUES ('received-key', ...)
ON CONFLICT (idempotency_key) DO NOTHING;
Second best
Absent an idempotency ID for deduping, all your solutions are going to be hacky, but you could still get something workable together. What you've already suggested of trying to round off the receipt time should mostly work, although it'll still have the possibility of losing two events that were different, but generated close together in time.
Alternatively, you could also try using the entire payload of a received webhook, or better yet, a hash of it, as an idempotency ID:
CREATE UNIQUE INDEX index_my_table_payload_hash ON my_table (payload_hash);
INSERT INTO object_changes (payload_hash, ...) VALUES ('<hash_of_webhook_payload>', ...)
ON CONFLICT (payload_hash) DO NOTHING;
This should keep the field relatively small in the database, while still maintaining accurate deduplication, even for unique events sent close together.
You could also do a combination of the two: a rounded timestamp plus a hashed payload, just in case you were to receive a webhook with an identical payload somewhere down the line. The only thing this wouldn't protect against is two different events sending identical payloads close together in time, which should be a very unlikely case.
If you look at the acquity webhook docs, they supply a field called action, which key to making your webhook idempotent. Here are the quotes I could salvage:
action either scheduled rescheduled canceled changed or order.completed depending on the action that initiated the webhook call
The different actions:
scheduled is called once when an appointment is initially booked
rescheduled is called when the appointment is rescheduled to a new time
canceled is called whenever an appointment is canceled
changed is called when the appointment is changed in any way. This includes when it is initially scheduled, rescheduled, or canceled, as well as when appointment details such as e-mail address or intake forms are updated.
order.completed is called when an order is completed
Based on the wording, I assume that scheduled, canceled, and order.completed are all unique per object_id, which means you can use a unique together constraint for those messages:
class AcquityAction(models.Model):
id = models.CharField(max_length=17, primary_key=True)
class AcquityTransaction(models.Model):
action = models.ForeignKey(AcquityAction, on_delete=models.PROTECT)
object_id = models.IntegerField()
class Meta:
unique_together = [['object_id', 'action_id']]
You can substitute the AcquityAction model for an Enumeration Field if you'd like, but I prefer having them in the DB.
I would ignore the change event entirely, since it appears to trigger on every event, according to their vague definition. For the rescheduled event, I would create a model that allows you to use a unique constraint on the new date, so something like this:
class Reschedule(models.Model):
schedule = models.ForeignKey(MyScheduleModel, on_delete=models.CASCADE)
schedule_date = models.DateTimeField()
class Meta:
unique_together = [['schedule', 'schedule_date']]
Alternatively, you could have a task specifically for updating your schedule model with a rescheduled date, that way it remains idempotent.
Now in your view, you will do something like this:
from django.db import IntegrityError
ACQUITY_ACTIONS = {'scheduled', 'canceled', 'order.completed'}
def webhook_view(request):
validate(request)
action = get_action(request)
if action in ACQUITY_ACTIONS:
try:
insert_transaction()
except IntegrityError:
return HttpResponse(200)
webhook_task.delay()
elif action == 'rescheduled':
other_webhook_task.delay()
...
I'm new to actor model and Orleans, so any suggestions on good practices to solve the following task is very appreciated:
we have [service1] that runs some logic and store some results in relational database (legacy thing). Now somewhere in the middle we want to call Orleans actor [Actor1] which holds a list of numbers, to get the next available number. The goal of the [Actor1] is to feed the numbers sequentially and consistently, so no skip over, no duplication is allowed, so it's sort of single-threaded stack. Single-threaded not only per process, but throughout the cluster of services, exactly what we need.
[service1] -> [Actor1]
Now the only problem I see here is that [service1] can fail with exception after it takes the next number, but before it stores results in database. Number is taken from the single-threaded stack, but it's lost as calling application did not manage to store results based on the fed number in database. In other words, I do not want the actor to feed next number, unless it ensures the last fed one is in good use, and only calling application knows if it is.
How would you suggest to handle these situations? Can I somehow keep Orleans actor's job open unless calling service (or another actor) commits it to database?
This is a Byzantine problem, so there is no easy solution: there will be "holes" in the number sequence or you will use the same number twice.
I would prefer to get holes and fill them with dummy data later if this is necessary (eg. if this is a billing system, end of month enter a cancelled empty bill for each bill number that is a "hole").
Even in SQL, an Insert and a Rollback will let the sequence incremented in an auto-increment primary-key ID column, so there can be holes after a failure.
I am trying to figure out how to handle consistency in the database.
In scenario:
User A has an accounting document in the database include a balance field representing the amount of his current money. (supposed initially he has 100$)
My system has many methods to charge his account.
Suppose 2 methods occur at the same time, each method charges him for 10$, these steps occur concurrently in below orders:
Method 1 READ his balance and store in memory (100$)
Method 2 READ his balance and store in memory (100$)
... some business logics
Method 1 UPDATE his balance by subtracting variable in memory by 10 (100$ - 10$) and then save it
Method 2 UPDATE his balance by subtracting variable in memory by 10 (100$ - 10$) and then save it
This means he has been charged only 10$ instead of 20$.
I searched this situation a while and can not get it clear (sorry for my stupidity).
Really appreciate yours helps to enlighten my featherbrained. :)
You just discovered why financial transactions are complicated :-)
Have you ever wondered why it takes time for you to have an updated balance in your bank account? Or why you actually have two balances, instead of one?
That's because your account can actually go negative and (up to a certain point) that will be fine.
So in a real life scenario what happens is that you have a balance of 100$, you pay 10$ and until that transaction is processed and confirmed by the receiver, you still have your 100$. If you do 20 transactions of 10$ each, you'll be able to complete them because the system will most likely not be able to notice.
And honestly, it shouldn't. Think of credit cards, you might not have enough money now, but maybe you know you'll have enough when the credit is due.
So, the race condition you describe only works if you actually read the value and then update it.
There are a few approaches:
Read the current balance, and update the row using the old balance as a field in the where statement. This way if it updates no rows you know that you need to re-read and update.
Don't update the balance and only do it time-based, say once per hour. Yes, you might still have to do some checks, but the system will overall be more responsive.
Lock the database row as your first step. This would work but there's a chance that it will make the app slower.
Race condition you describe is low level design concern. With backend engine like Node that will handle the incomming request in first come first serve fashion you don't need to think about this case. Race condition you describe is not possible if you respect the order in which database update callbacks are fired. They are fired in the same order they have been issued in. So you should call next update only when the previous has finished. Promisses are great way to do this.
I'm currently working on a app. This also includes some kind of group chats.
The users inside can make multiple votes, for example for kicking someone. The votes are all valid for 1 week. If all other users submit their opinion the vote gets deleted.So far so good.
I also want a logic, which deletes the vote automatically if it's expired.
So far I got the idea to store the expiration dates for the votes inside a database(MongoDB), sorted by their timestamp of expiration.
In NodeJs I'm always loading the vote with the smallest expiration date from the database.
Then I check how much time is left by subtracting the vote expiration date from the current Date
Date.now() - voteTmp;
Then I can set a timeout, which calls a function to delete the vote and automatically starts a new timeout for the next vote. Is it a problem to set a timeout with such a big number of seconds?
Do you have any better ideas?
Thank you:)
The node.js event loop is explained here:
when the event loop enters a given phase, it will perform any operations specific to that phase, then execute callbacks in that phase's queue until the queue has been exhausted or the maximum number of callbacks has executed.
On each iteration, the event loop checks for scheduled timers that satisfy the specified thresholds (delays) and executes their callbacks. Thus, the magnitude of delays for registered timers shouldn't matter.
However, in your scenario, there's a chance that you might accidentally register redundant or invalid timers (possibly after recovering from a crash). MongoDB supports (automatic) data expiration. You can instruct MongoDB to delete documents after a specified number of seconds has passed. That seems close enough to what you want to do.
I'm creating an application that will sort items in the DB in order to create a selective process.
This process, will consist of Users, and Registers in courses.
Each course, will have Users in it, and the SORT method to select them will vary depending on each course.
I'm implementing a way of 'simulating' the position of a user in a course, without matriculating it in the course, so that he can 'know' it's position prior to entering the selection process.
To do so, I imagined that I could use the same logic used after the user has already registered: Sort in the DB, return the list of ID's, and see what's the user index in that list.
However, I want just to simulate, without creating/updating anything. I cannot find a way to do that, without during the query, inserting a 'fake' document, but that cannot happen for reasons of security and integrity (inserting/removing items let the DB integrity broken during a short period of time, and can cause conflicts within logics of the application).
Doing the sorting on the DB, and re-doing it on the system is also not a good Idea as well, since there will be duplicated logic going on.
How can I 'fake' an document, without creating it during a query?