Below is the Question I was asked in an interview and I believe there are many solutions to this question but I want to know what can be the best solution (and stackoverflow is perfect for this :) ).
Q: We have a tree like structure and have three threads. Now we have to perform three operations: Insert, Delete and lookup. How will you design this?
My Approach: I will take mutex for insert and delete operation as I want only one thread to perform at a time insert or delete. While in case of lookup I will allow all the three threads to enter in the function but keep a count(counting semaphore) so that insert and delete operation can't be perform this time.
Similarly when insert or delete operation is going no thread is allowed to do lookup, same with the case for insert and delete.
Now he cross questioned me that as I am allowing only one thread at a time to insert so if two nodes on different leaf need to be inserted then still my approach will allow one at a time, this made be stuck.
Is my approach fine ?
What can be other approaches ?
How about like this? Similar to a traffic road block(broken paths).
Each node will have two flags say leftClear_f and rightClear_f indicating clear-path ahead
There will be only one MutEx for the tree
Lookup Operation:
If flags are set indicating path ahead is under modification, got to conditional_wait and wait for signal.
after getting signal check the flag and continue.
Insert Operation
follow the Lookup till you get to the location of insertion.
acquire MutEx and set relevant flag of parent_node and both child_nodes after checking their state.
Release the MutEx so that parallel Delete/Insert can happen on other valid unbroken paths
Acquire MutEx after insert operation and update the relevant flag in the parent_node and child_nodes.
Delete Operation
same as Insert operation except that it deletes nodes.
PS: You can also maintain the details of the nodes under Insert or Delete process someplace else. Other operation can jump the broken paths if necessary/needed! It sounds complicated yet doable.
Related
I would like to reserve a large amount of object ids prior to a complex COPY operation. I know setval and nextval have atomic guarantees, but do these guarantees hold in a multithreaded environment if I'm using them in a compound statement such as the following? I'm using postgresql 9.6.
SELECT setval('objects_id_seq', nextval('objects_id_seq') + 9999); -- returns the last reserved id
I know setval and nextval have atomic guarantees
Yes, but the guarantees may not be what you think they are. Remember that sequences are exempt from normal transaction boundaries. Your setval will take effect immediately on other concurrent transactions. It's atomic - it either happens in its entirety or not at all - but that doesn't mean it obeys all ACID properties.
Your query is definitely going to affect concurrent queries. Specifically, the value of the sequence will continue to increase between nextval and your subsequent setval, if there are concurrent nextval calls from other queries. So if your intent is "reserve 9999 IDs" it won't work, you might actually only reserve 9989 if 10 other sessions called 'nextval' in the small window between your nextval and setval calls.
You can't LOCK a sequence from SQL either, so you can't just take an EXCLUSIVE lock on it.
You can either:
Assign way more extra values than you need and hope you're fast enough;
Lock the table that uses the sequence in EXCLUSIVE mode and hope nobody concurrently calls nextval(...) on the sequence directly;
let COPY assign generated keys from a DEFAULT nextval(....) like usual;
Assign generated keys a different way, such as a counter table, where you can apply stronger locking.
What I think you really want in this case is a nextval variant that increments by your supplied value e.g. 9999, not by 1. PostgresSQL doesn't have such a function yet, but it'd be really handy to have one. Patches welcome!
You might be thinking "what if I ALTER SEQUENCE ... INCREMENT 9999 then call nextval then ALTER SEQUENCE ... INCREMENT 1. Yeah, don't do that. Unfortunately, ALTER SEQUENCE's effects will also become visible outside transaction boundaries due to the same properties that make nextval work. So concurrent queries calling nextval might get a 9999-row jump too. You might not care, but it's worth knowing about. I wouldn't recommend that you rely on this behaviour.
Here is the nice article which describes what is ES and how to deal with it.
Everything is fine there, but one image is bothering me. Here it is
I understand that in distributed event-based systems we are able to achieve eventual consistency only. Anyway ... How do we ensure that we don't book more seats than available? This is especially a problem if there are many concurrent requests.
It may happen that n aggregates are populated with the same amount of reserved seats, and all of these aggregate instances allow reservations.
I understand that in distributes event-based systems we are able to achieve eventual consistency only, anyway ... How to do not allow to book more seats than we have? Especially in terms of many concurrent requests?
All events are private to the command running them until the book of record acknowledges a successful write. So we don't share the events at all, and we don't report back to the caller, without knowing that our version of "what happened next" was accepted by the book of record.
The write of events is analogous to a compare-and-swap of the tail pointer in the aggregate history. If another command has changed the tail pointer while we were running, our swap fails, and we have to mitigate/retry/fail.
In practice, this is usually implemented by having the write command to the book of record include an expected position for the write. (Example: ES-ExpectedVersion in GES).
The book of record is expected to reject the write if the expected position is in the wrong place. Think of the position as a unique key in a table in a RDBMS, and you have the right idea.
This means, effectively, that the writes to the event stream are actually consistent -- the book of record only permits the write if the position you write to is correct, which means that the position hasn't changed since the copy of the history you loaded was written.
It's typical for commands to read event streams directly from the book of record, rather than the eventually consistent read models.
It may happen that n-AggregateRoots will be populated with the same amount of reserved seats, it means having validation in the reserve method won't help, though. Then n-AggregateRoots will emit the event of successful reservation.
Every bit of state needs to be supervised by a single aggregate root. You can have n different copies of that root running, all competing to write to the same history, but the compare and swap operation will only permit one winner, which ensures that "the" aggregate has a single internally consistent history.
There are going to be a couple of ways to deal with such a scenario.
First off, an event stream would have the current version as the version of the last event added. This means that when you would not, or should not, be able to persist the event stream if the event stream is not at the version when loaded. Since the very first write would cause the version of the event stream to be increased, the second write would not be permitted. Since events are not emitted, per se, but rather a result of the event sourcing we would not have the type of race condition in your example.
Well, if your commands are processed behind a queue any failures should be retried. Should it not be possible to process the request you would enter the normal "I'm sorry, Dave. I'm afraid I can't do that" scenario by letting the user know that they should try something else.
Another option is to start the processing by issuing an update against some table row to serialize any calls to the aggregate. Probably not the most elegant but it does cause a system-wide block on the processing.
I guess, to a large extent, one cannot really trust the read store when it comes to transactional processing.
Hope that helps :)
We are trying to create an algorithm/heuristic that will schedule a delivery at a certain time period, but there is definitely a race condition here, whereby two conflicting scheduled items could be written to the DB, because the write is not really atomic.
The only way to truly prevent race conditions is to create some atomic insert operation, TMK.
The server receives a request to schedule something for a certain time period, and the server has to check if that time period is still available before it writes the data to the DB. But in that time the server could get a similar request and end up writing conflicting data.
How to circumvent this? Is there some way to create some script in the DB itself that hooks into the write operation to make the whole thing atomic? By putting a locking mechanism on that script? What makes the whole thing non-atomic is the read and the wire time between the server and the DB.
Whenever I run into race condition I think of one immediate solution QUEUE.
Step 1) What you can do is that instead of adding data to a database directly you can add it to queue without checking anything.
Step 2) A separate reader will read from the queue check DB for any conflict and take necessary action.
This is one of the ways to solve this If you implement any better solution please do share it.
Hope that helps
Please explain why modifying many aggregates at the same time is a bad idea when doing CQRS, ES and DDD. Is there any situations where it still could be ok?
Take for example a command such as PurgeAllCompletedTodos. I want this command to lead to one event that update the state of each completed Todo-aggregate by setting IsActive to false.
Why is this not good?
One reason I could think of:
When updating the domain state it's probably good to limit the transaction to a well defined part of the entire state so that only this part need to be write locked during the update. Doing so would allow many writes on different aggregates in parallell which could boost performance in some extremely heavy scenarios.
The response of the question lie in the meaning of "aggregate".
As first thing I would say that you are not modifying 'n' aggregates, but you are modifying 'n' entities.
An aggregate contains more-than-one entity and it is just a transaction concept, the aggregate (pattern) is used when you need to modify the state of more than one entity in your application transactionally (all are modified or none).
Now, why you would modify more than one aggregate with one command?
If you feel this needs, before doing anything else check your aggregate boundaries to see if you can modify it to remove the needs to 1 command -> 'n' aggregate.
An aggregate can contains a lot of entities of the same type, so for your command PurgeAllCompletedTodos, you could also think about expand the transaction boundary from a single Todo to an aggregate UserTodosAggregate that contains all the user todos, and let it manage all the commands for the todos of a single user.
In this way you can modify all the todos of a user in a single transaction.
If this still doesn't solve your problem because, let's say that is needed to purge all completed todos of each user in the application, you will still need to send a command to 'n' aggregates, the aggregate boundary doesn't help, so we can think of having an AllApplicationTodosAggregate that manage the command.
Probably this isn't the best solution, because as you said it that command would block ALL the todos of the application, but, always check if it can be a good trade off (this part of the blocking is explained very well in both Blue Book and Red Book of DDD).
What if I need to modify some entities and can't have them in a single aggregate?
With the previous said, a command that modify more than one aggregate is bad because of transactions. What if you modify 3 aggregate, the first is good, and then the server is shut down?
In this case what you are doing is having a lot of single modification that needs to be managed to prevent inconsistency of the system.
It can be done using a process manager, whom responsabilities are modify all the aggregates sending them the right command and manage failures if they happen.
An aggregate still receive it's own command, but the process manager is in charge to send them in a way it knows (one at time, all in parallel, 5 per time, what-do-you-want)
So you can have a strategy to manage the failure between two transaction, and make decision like: "if something fail, roll back all the modification done untill now" (sending a rollback command to each aggregate), or "if an operation fail repeat it 3 times each 30 minutes and if doens't work then rollback", "if something fail create a notification for the system admin".
(sorry for the long post, at least hope it helps)
I have 2 threaded methods running in 2 separate places but sharing access at the same time to a list array object (lets call it PriceArray), the first thread Adds and Removes items from PriceArray when necessary (the content of the array gets updated from a third party data provider) and the average update rate is between 0.5 and 1 second.
The second thread only reads -for now- the content of the array every 3 seconds using a foreach loop (takes most items but not all of them).
To ensure avoiding the nasty Collection was modified; enumeration operation may not execute exception when the second thread loops through the array I have wrapped the add and remove operation in the first thread with lock(PriceArray) to ensure exclusive access and prevent that exception from occurring. The problem is I have noticed a performance issue when the second method tries to loop through the array items as most of the time the array is locked by the add/remove thread.
Having the scenario running this way, do you have any suggestions how to improve the performance using other thread-safety/exclusive access tactics in C# 4.0?
Thanks.
Yes, there are many alternatives.
The best/easiest would be to switch to using an appropriate collection in System.Collections.Concurrent. These are all thread-safe collections, and will allow you to use them without managing your own locks. They are typically either lock-free or use very fine grained locking, so will likely dramatically improve the performance impacts you're getting from the synchronization.
Another option would be to use ReaderWriterLockSlim to allow your readers to not block each other. Since a third party library is writing this array, this may be a more appropriate solution. It would allow you to completely block during writing, but the readers would not need to block each other during reads.
My suggestion is that ArrayList.Remove() takes most of the time, because in order to perform deletion it performs two costly things:
linear search: just takes elements one by one and compares with element being removed
when index of the element being removed is found - it shifts everything below it by one position to the left.
Thus every deletion takes time proportionally to count of elements currently in the collection.
So you should try to replace ArrayList with more appropriate structure for this task. I need more information about your case to suggest which one to choose.