I am working on a ledger module. In that process, I had to do these tasks in order
select balance from source (table1 row1)
select balance from destination (table1 row2)
modify balances with some logic
update balance for source (table1 row1)
update balance for destination (table1 row2)
commit the changes
insert the transactions into a transaction table.
In multithreaded environment, threads get balances before the previous thread updates and commits.
In Postgres locks are imposed for the row which is being accessed until the thread commits. I see an anamoly in this situation.
Tried using synchronized block for the whole pattern. Didn't help.
can't use SELECT FROM UPDATE because the logic for changing balances depends on other operation.
Here are some logs, before [1] thread goes to update section other threads collect the existing balances.
[DEBUG] [2016-12-15 10:49:53,893] [1] - Src Ledger_book_idLB001
[DEBUG] [2016-12-15 10:49:53,893] [1] - Src Balance2500.0
[DEBUG] [2016-12-15 10:49:53,897] [1] - Dest Ledger_book_idLB002
[DEBUG] [2016-12-15 10:49:53,897] [1] - Dest Balance0.0
[DEBUG] [2016-12-15 10:49:53,898] [15] - Src Ledger_book_idLB001
[DEBUG] [2016-12-15 10:49:53,898] [15] - Src Balance2500.0
[DEBUG] [2016-12-15 10:49:53,899] [16] - Src Ledger_book_idLB001
[DEBUG] [2016-12-15 10:49:53,899] [16] - Src Balance2500.0
[DEBUG] [2016-12-15 10:49:53,900] [16] - Dest Ledger_book_idLB002
[DEBUG] [2016-12-15 10:49:53,900] [15] - Dest Ledger_book_idLB002
Any help is appreciated :) Please comment if any specific doubt about the situation.
From the docs
When a locking clause appears in a sub-SELECT, the rows locked are those returned to the outer query by the sub-query. This might involve fewer rows than inspection of the sub-query alone would suggest, since conditions from the outer query might be used to optimize execution of the sub-query.
So you can do this...
You should also read the page on CTE's, especially DATA-MODIFYING
WITH s1 AS (
SELECT balance FROM source
FOR UPDATE
), s2 AS (
SELECT balance destination
FOR UPDATE
), u1 AS (
UPDATE source SET balance = ...
WHERE... -- potential join to S2 if you need destination
RETURNING -- whatever you need (if anything)
), u2 AS (
UPDATE source SET balance = ...
WHERE... -- potential join to S1 if you need source
RETURNING -- whatever you need (if anything)
)
INSERT INTO transactions (foo,date) VALUES (bar,now());
Related
I have a simple example of dish washers at a restaurant to illustrate the issue I am having.
Question
How can I ensure that the correct number of dish washers are seized & released when it's depended on the number of agents being used?
Problem
Using a function to assign the resources, the number of dish washers are not always correct due to different times in which sinks are used and not used.
Example
Main:
Generates dishes and randomly assigns them to one of three sinks in the exit block.
Sinks is a population of agents.
dish_washers is a ResourcePool with a capacity of 10.
Sink:
Dishes enter a queue and are entered one at a time using a hold block.
Once the dish is cleaned, the hold is unblocked to grab the next dish.
Details:
I have a shared ResourcePool of dish_washers at a restaurant.
There are 3 sinks at the restaurant.
Dishes are generated and randomly assigned to each sink.
If only 1 sink is being used, then two dish washers are needed.
However, if 2 or more sinks are being used then the number of dish washers becomes:
numberOfDishWashers = 2 + numberOfSinksInUse;
In order to change the numberOfDishWashers as more sinks are being used, I created a function that defines the numberOfDishWashers to be seized from the dish_washer ResourcePool.
int numberOfSinksUsed = 0;
int numberOfWorkersToSeize = 0;
int numberOfWorkersAlreadySeized = 0;
int numberOfWorkersToAssign = 0;
ResourcePool[][] dish_washers;
for(Sink curSink : main.sinks){
if(curSink.queue.size() > 0){
numberOfSinksUsed += 1;
}
}
numberOfWorkersAlreadySeized = main.dish_washers.busy();
numberOfWorkersToSeize = 2 + numberOfSinksUsed;
numberOfWorkersToAssign = numberOfWorkersToSeize - numberOfWorkersAlreadySeized;
dish_washers = new ResourcePool[1][numberOfWorkersToAssign];
for(int i = 0; i < numberOfWorkersToAssign; i++){
dish_washers[0][i] = main.dish_washers;
}
return dish_washers;
Error Description:
However, depending on which sink completes first & releases, the number of dish washer assigned will be incorrect. A traceln at the end of the sink process illustrates this where the numberOfDishWashers seized on the exit block doesn't match "2 + numberOfSinksInUse".
There is an instance where 3 sinks are in used but only 4 workers were seized.
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 2
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 3
Exit, Sink: C Workers Currently Seized: 5
Sinks in Use: 2
Exit, Sink: C Workers Currently Seized: 4
Sinks in Use: 2
Another way to look at the issue, is this Excel table outlining the current logic.
The number of busy workers doesn't match the number of busy workers there should be based on the number of active sinks.
Methods I have Tried
Custom function to release only the necessary workers to keep the correct total.
Generates an error because the resource gets assigned to the 'agent' or dish.
When the dish gets destroyed it has unreleased resources attached to it.
Passing the "sink" agent through an "enter", "seize", and "exit" block to assign the
resource to the agent "sink" instead of the dish that is generated.
Error regarding the "dish" agent being in the flowchart of the "sink" agent while the
"sink" agent is seizing the workers.
How can I ensure the correct number of dish washers are always grabbed?
So your fundamental problem here is that inside the sink you will seize a dishwasher, then the dish goes into the delay (With the number of dishwashers seized) and once out of the delay it will release what ever dishwashers it seized... But during the time it is in the delay the situation might have changed and you actually wanted to seize a different number of dishwashers for that specific sink...
Your options are to either
Remove dishes from the delay, release the correct amount of dishwashers, return back into the delay and delay for the remainder of the time...
Implement your own logic.
I would go for option 2 as option 1 means that you develop a workaround for the block created by AnyLogic and you will end up not using the blocks the way they were designed, this is unfortunately the issue with blockification
So I would have a collection inside of a sink that shows the number of dishwashers currently assigned to this sink. Then whenever a new dish enters a sink we recalculate the number of dishwashers to assign (perhaps at every sink? ) and then make the correct assignment.
Here is an example with some sample code - I did not test it but you will have something similar
We have two database tables : PointsMaster (PM) and PointsDetails (PD), the requirement here is to check if PM table has currentbalance of greater than "x" and if it is greater than "x" then update the PM table's currentbalance (i.e. reduce currentbalance by "x") and then also put a new record in the PD table about this "x" points.
If the currentbalance is lower than "x" then no updates to PM table and no insert to PD table.
If currentbalance is NULL then simply we need to return back null.
Right now writing these individual:- SELECT and then UPDATE (PM) and INSERT (PD) is causing lock issues and I am thinking of using "MERGE" (on PM Table) so I can do SELECT and UPDATE in one transaction and not cause any locks on PM table but my issue here is how do I check if CurrentBalance is Null or Lower than "X" and return that back so that processing can happen for those 2 conditions (i.e. of Null balance or when balance is less than "x").
MERGE can help me here with SELECT and if MATCHED UPDATE on PM.. but the INSERT on PD has to happen only if UPDATE on PM happened else not and also if Balance is lower than "x" or Null I have to return that back and not UPDATE PM or ADD record to PD.
Any help here? As the code we have is causing locks and waits and is not efficient with volume of data. PM and PD tables have required indexes in place.
All of the above is being done from C#.Net in a Azure hosted Function App.
Thinking of following SQL Code
Declare #rcount INT;
MERGE PointsMaster as tgt
Using PointsMaster as src
ON (tgt.ProfileID = src.ProfileID)
WHEN MATCHED AND tgt.ProfileID ='3589153' and tgt.CurrentBalance > 50000000
Then UPDATE SET tgt.CurrentBalance = tgt.CurrentBalance - 5;
Set #rcount= ##ROWCOUNT;
if (#rcount > 0)
Begin
INSERT INTO PointsDetails ( PointsHeaderID,Debit,Credit,ActionTypeCode,TransactionDate) Values (3587854,5,0,'AT2',Getdate())
End
if (#rcount = 0)
Begin
SELECT * from PointsMaster where ProfileID = '3589153'
End
USECASE
HazelcastJet version 0.6.1
Hazelcast version 3.10.2
Given this (simpified version) of a DAG
VERTICES
S1
Source that emits 5 items of type A (read from DB with partitioning)
Local parallelism = 1
S2
Source that emits 150K items of type B (Iterator that read from DB in batch of 100 with partitioning)
Local parallelism = 1
AD
Processor that adapts types A->A1 and B->B1 and emits one by one
FA
Processors.filterP that accepts only items of type A1 and emits one by one
FB
Processors.filterP that accepts only items of type B1 and emits one by one
CL
Processor that first accumulate all items of type A1, then when it receive an item of type B1, enriches it with some staff got from proper A1, and emit, one by one.
WR
Sink that writes B1
Local parallelism = 1
NOTE:
Just to give meaning to the filter processor: in the DAG there are other sources that flows into the same adapter AD and then goes to other paths using filter processors.
EDGES
S1 --> AD
S2 --> AD
AD --> FA (from ordinal 0)
AD --> FB (from ordinal 1)
FA --> CL (to ordinal 0 with priority 0 distributed and broadcast)
FB --> CL (to ordinal 1 with priority 1)
CL --> WR
PROBLEM
If source S2 have "few" items to load (i.e. 15K) the emitFromTraverser never returns false.
If source S2 have "many" items to load (i.e. 150K) the emitFromTraverser returns false after:
All A1 items have been processed by CL
About 30% of B1 items have already been transmitted to CL but no one have been processed by CL (DiagnosticProcessor log that element are sent to CL but not processed)
S2 code for reference:
protected void init(Context context) throws Exception {
super.init(context);
this.iterator = new BQueryIterator(querySupplier, batchSize);
this.traverser = Traversers.traverseIterator(this.iterator);
}
public boolean complete() {
boolean result = emitFromTraverser(this.traverser);
return result;
}
QUESTION
Is it correct that CL doesn't process items until source ends?
Is the usage of priority + distributed + broadcast correct on CL Vertex?
UPDATE
It seems that the completeEdge on CL edge 1 is never called.
Someone can tell me why?
Thanks!
You suffer from a deadlock caused by priority. Your DAG branches from AD and then rejoins in CL, but with a priority.
AD --+---- FA ----+-- CL
\ /
+-- FB --+
Setting a priority causes that no item from lower-priority edge is processed before all items from higher-priority edge are processed. AD will eventually get blocked by backpressure from the lower-priority path, which is not processed by CL. So AD is blocked because it can't emit to the lower priority edge and CL is blocked, because it it's still waiting for items from the higher priority edge, resulting in a deadlock.
In your case, you can resolve it by making 2 AD vertices, each processing items from one of the sources:
S1 --- AD1 ----+--- CL
/
S2 --- AD2 --+
After a while I've understood what's the problem...
CL processor cannot know when all the A1 items have been processed because all items they all come from the AD processor.
So it need to wait for all sources coming from AD before starting the processing of B1 items.
Not sure but probably after a lot of items B loaded, all Inboxes buffers in the DAG become full and can't accept any other B from S2, but at the same time cannot process B1 items to continue: that's the deadlock.
Maybe DAG would be able to detect this?
I don't know Jet so deeply but it would be nice to have that warning.
Maybe is there some logging to enable?
I hope someone can confirm my answer and suggest how to improve and detect these problems.
pt-checksum is getting to never ending loop in multi-source (channel based) replication :
A multi source replication in my environment
n1 -> n2 (created n1 as channel replication)
n2 -> n3
n1 -> n3 (n3 will replicate from both n1 and n2 channels)
Whenever without channel wise replication pt-checksum is working fine. (traditional replication/default replication works without any issues)
Once i enabled as channel wise(only n1, removed the n2 channel) replication (n1->n3) , the pt-checksum is not going through.
Replica n3-VirtualBox is stopped. Waiting.
Replica n3-VirtualBox is stopped. Waiting.
from general log n1:
Query SELECT 'pt-table-checksum keepalive'
Query SELECT 'pt-table-checksum keepalive'
I resolved the problem by following method:
On Master DB Server
(i) For safe side, I made copy of original /bin/pt-table-checksum
cp /bin/pt-table-checksum /bin/pt-table-checksumorg
(ii) Open the /bin/pt-table-checksum file
vi /bin/pt-table-checksum
(iii) Go to line number 8590
Press esc, type 8590 , press shift g
(iv) replace the line
my #lagged_slaves = map { {cxn=>$_, lag=>undef} } #$slaves;
by
my #lagged_slaves = ();
the program immediately works and returns the expected results.
I have same problem. I also get infinite loop of "Replica dbslave is stopped." However if I break the output by pressing "Ctl + c", I get the inconsistent database and table name. I verified it and found the inconsistency in slave. Then I used pt
Set rowcount 50000
declare #i int
select #i = 1
WHILE ( #i > 0 )
BEGIN
DELETE table1
FROM table1 (index index1)
WHERE
HIST_Timestamp < '2011/11/26'
select #i = ##rowcount
END
The query sometimes encounters a deadlock situation and terminates.. Not able to figure out what is going wrong .. Please help me!
A deadlock occurs when transaction A locks a record then has to wait for transaction B to unlock a record, while transaction B is waiting on a record already locked by transaction A.
If you really want to know why the deadlock is happening, you can do it with this command:
sp_configure "print deadlock information", 1
Creating a useful index for the query allows the delete statement to use page or row locks, improving concurrent access to the table. If creating an index for the delete transaction is not possible, you can perform the operation in a cursor, with frequent commit transaction statements to reduce the number of page locks.