I'm struggling with this issue since two days. We have a solution where multiple worker threads will try to select job requests from a single database/table, by setting a flag on the selected requests and thus effectively blocking the other workers to select the same requests.
I created a java test application to test my queries, but while in normal situations the test executes without issue, in high contention situation (ex. 1 table entry with 50 threads; no delays or processing) I still have threads which obtain the same request/entry, interestingly it happens when the test just starts. I cannot understand why. I've read all relevant Postgres locking and isolation related documentation... While is possible that the issue is with the test application itself, I suspect that I'm missing something about how the SELECT FOR UPDATE works in READ COMMITTED isolation context.
So the question would be can SELECT FOR UPDATE (with READ COMMITED isolation) guarantee that a general concurrency issue like I described can be safely solved?
Acquire query:
UPDATE mytable SET status = 'LOCK'
WHERE ctid IN (SELECT ctid FROM mytable
WHERE status = 'FREE'
ORDER BY id
LIMIT %d
FOR UPDATE)
RETURNING id, status;
Release query:
UPDATE mytable SET status = 'FREE'
WHERE id = %d AND status = 'LOCK'
RETURNING id, status;
So would you consider these two queries should be safe, or there is some weird case possible that would allow two threads to acquire the same row? I'd like to mention that I tried also SERIALIZABLE isolation and didn't helped.
It turns out (how could it be different?) that I made a mistake in my test. I didn't respected resource acquire/release order. The test was registering the release (decrementing a counter) after the Release Query, which lead another thread to Acquire the resource and register it in the meantime. An error from the category, which you know how to solve, but cannot see even if you look several times, because you wrote the code... Peer review helped in the end.
I suppose at this time I have a test to prove that:
The above two queries are safe
You don't need SERIALIZABLE isolation to solve problems with DB acquire/release as long as you use row locking as in SELECT... FOR UPDATE
You must ORDER (BY) results when using row locking (even if you use LIMIT 1!), otherwise you end up with deadlocks
Safe to acquire multiple resources with one query (LIMIT 2 and above)
Using ctid in the query is safe; it's actually a little faster, but this is insignificant in real world applications
I'm not sure if this will be helpful to others, but I was getting desperate. So all is good with Postgres 9.3 :)
Another aspect I'd like so share is regarding the speed of the Acquire query with LIMIT 2. See the test result:
Starting test...
DB setup done
All threads created & connections made
All threads started
Thread[36] 186/190/624=1000
Thread[19] 184/201/615=1000
Thread[12] 230/211/559=1000
Thread[46] 175/200/625=1000
Thread[ 9] 205/211/584=1000
...
Thread[ 4] 189/232/579=1000
Thread[ 3] 185/198/617=1000
Thread[49] 218/204/578=1000
Thread[ 1] 204/203/593=1000
...
Thread[37] 177/163/660=1000
Thread[31] 168/199/633=1000
Thread[18] 174/187/639=1000
Thread[42] 178/229/593=1000
Thread[29] 201/229/570=1000
...
Thread[10] 203/198/599=1000
Thread[25] 215/210/575=1000
Thread[27] 248/191/561=1000
...
Thread[17] 311/192/497=1000
Thread[ 8] 365/198/437=1000
Thread[15] 389/176/435=1000
All threads finished
Execution time: 31408
Test done; exiting
Compare the above with this query :
UPDATE mytable SET status = 'LOCK'
WHERE id IN (SELECT t1.id FROM (SELECT id FROM mytable
WHERE status = 'FREE' ORDER BY id LIMIT 2) AS t1
FOR UPDATE)
RETURNING id, status;
and the result:
Starting test...
DB setup done
All threads created & connections made
All threads started
Thread[29] 32/121/847=1000
Thread[22] 61/151/788=1000
Thread[46] 36/114/850=1000
Thread[41] 57/132/811=1000
Thread[24] 49/146/805=1000
Thread[13] 47/135/818=1000
...
Thread[20] 48/118/834=1000
Thread[47] 65/152/783=1000
Thread[18] 51/146/803=1000
Thread[ 8] 69/158/773=1000
Thread[14] 56/158/786=1000
Thread[ 0] 66/161/773=1000
Thread[38] 60/148/792=1000
Thread[27] 69/158/773=1000
...
Thread[45] 78/177/745=1000
Thread[30] 96/162/742=1000
...
Thread[32] 162/167/671=1000
Thread[17] 329/156/515=1000
Thread[33] 337/178/485=1000
Thread[37] 381/172/447=1000
All threads finished
Execution time: 15490
Test done; exiting
Conclusion
The test prints for each thread how many times the Acquire query returned 2, 1 or 0 resources totalling the number of test loops (1000).
From the above results we can conclude that we can speed up the query (halfing the time!) at the cost of increasing the thread contention. This means that we will receive more times 0 resources back from the Acquire query. Technically this is not a problem because we need to treat this situation anyway.
Of course situation changes if you add a wait time (sleeping) when no resources are returned, but choosing a correct value for the wait time depends on the application performance requirements...
Related
I am experimenting with KeyDB to see if and how much performance improvements can be gained, as there are definitely bottlenecks with Redis single-threaded query model. So I found KeyDB, and they say they use "real" multithreading to do parallel queries to the db, unlike Redis that only has IO multithreading and not the actual queries.
From the documentation link above:
Unlike Redis6 and Elasticache, KeyDB multithreads several aspects
including placing the event loop on multiple threads, with network IO, and query parsing done concurrently.
My simple test setup:
First, I install KeyDB on Ubuntu (WSL2) and get it running
I note that when starting KeyDb, 2 threads are active:
Thread 0 alive. Thread 1 alive.
I modify the keydb.conf to disable some saving/persisting, but most importantly, I change the server-threads option to 2: server-threads 2. note: I have also tried without the use of the config file and just add the cmd flag --server-threads 2 and also setting threads to 4, no difference.
Then I run a simple script:
Create 1M entries into a hash with some simple JSON objects
Create a simple console app that uses two threads; one thread starts doing very simple SETs (SET key1 1) or GETs (GET key1 1) in a loop, and another thread that does a "fetch all" from the hash (HGETALL testhash). The second thread waits 1 sec before it starts its "long query".
GitHub repo (using StackExchange.Redis lib) can be found here.
What I expect:
I expect that the simple quick SET/GETs takes approx the same time every time, without any delays or throttling due to a block in KeyDB while the long query is running.
What happens:
The simple quick SET/GETs are blocked/delayed for around 500-700 ms while the long query is running, indicating that only one thread is being used and thus blocking other operations. This is in line with how Redis works, and what I wanted to avoid with KeyDB.
Log:
The "Starting long query" is when we do the HGETALL and almost immediately after, the simple SET is throttled and takes over 500ms, when it should take 0-1 ms, as can be seen before and after.
Using ServiceStack Redis client:
10:50:55.336 GetValueFromHashAsync took 1
10:50:55.367 GetValueFromHashAsync took 1
10:50:55.397 GetValueFromHashAsync took 0
10:50:55.416 Starting long query
10:50:56.191 GetValueFromHashAsync took 766 <-- THROTTLED! Delayed with what I think is the actual query time, not the IO part, so at this point, the line fetching data has not completed yet
10:50:56.228 GetValueFromHashAsync took 0
10:50:56.261 GetValueFromHashAsync took 1
....
....
10:51:00.592 GetValueFromHashAsync took 1
10:51:00.620 GetValueFromHashAsync took 1
10:51:00.651 GetValueFromHashAsync took 1
10:51:00.663 Long query done in 5244 <-- The long query returns here, line is completed, total time was about 5 seconds, while the block was about 0.7 seconds
I have also tested to do a Get from hash instead of a SET, same thing.
Using StackExchange.Redis:
In the GitHub reproducable project, found here, I am instead using StackExchange.Redis instead of ServiceStack, and I get a different (worse!) behaviour:
11:27:12.084 HashGetAsync took 0
11:27:12.115 HashGetAsync took 0
11:27:12.146 HashGetAsync took 0
11:27:12.177 HashGetAsync took 1
11:27:12.183 Starting long query
11:27:14.877 Long query done in 2692
11:27:14.893 HashGetAsync took 2686 <-- THROTTLED! This time the other thread is delayed the entire time, query + IO.
11:27:14.929 HashGetAsync took 0
11:27:14.960 HashGetAsync took 0
11:27:14.992 HashGetAsync took 0
11:27:15.023 HashGetAsync took 0
11:27:15.053 HashGetAsync took 0
Conclusion
Regardless of what client library I use, KeyDB is throttling requests/queries while a "long query" is running, even though I have 2 threads. It does not matter if I start KeyDB with 4 threads, same behaviour.
I don't know why StackExchange behaves differently from ServiceStack, but that is not the main question right now.
KeyDB, in fact, only runs the IO operations and Redis protocol parsing operations in parallel. It processes the commands in serial, i.e. process commands one-by-one, and working threads are synced with a spin lock.
That's why those simple set/get commands are blocked by a slow command. So even with KeyDB, you should NOT run slow command either and, the multiple threading won't help.
UPDATE
KeyDB can have multiple threads listen on the same IP:port, so that it can accept multiple connections in parallel, i.e. SO_REUSEPORT. Also it reads (including parsing received data into commands with redis protocol, i.e. RESP) and writes socket in parallel.
While Redis only have a single thread, i.e. main thread, listen on the IP:port. By default, Redis reads and writes socket in a single thread. Since Redis 6.0, you can enable io-threads to make it write socket in parallel. Also, if you enable io-threads-do-reads, Redis will also reading and protocol parsing in parallel.
I am using python3.7 and sqlite3 module, with "auto commit" mode (i.e. creating the connection object by setting the isolation_level to None). I want to check if this is as efficient as possible? This is how I created by connection object.
conn = sqlite3.connect(DATABASE_NAME, check_same_thread=False, isolation_level=None)
The Python sqlite3 module has some odd behaviour with respect to transactions, and the documentation on the subject is rather confusing. Its reference to "autocommit mode" is especially confusing - I think there is a meaning for this term in Python which is different to the term used in the underlying SQLite library. I'll just avoid that term altogether in this answer.
If isolation_level is NOT None (indeed the default is "" which is not None) then transactions will sometimes be started automatically. It's up to you to commit them or roll them back, either by calling COMMIT or ROLLBACK explicitly or by using the connection as a context manager:
# Approach 1: transactions start automatically
conn = sqlite3.connect(path_to_db)
with conn:
values = [(row[0], row[1]) for row in conn.execute("SELECT a, b FROM Foo")]
# transaction starts here - in between the two statements!
conn.execute("UPDATE Foo SET a=? WHERE b=?", (values[0][0] + 1, values[0][1]))
# At this point, with conn: will automatically commit/rollback the transaction
(By the way this is a very poor way to increment a value. You don't need to read all rows, and you don't even need separate SELECT and UPDATE statements! But this is meant to illustrate how the transactions work, not how to write good SQL.)
In the above example, Python will NOT start a transaction when it gets to the SELECT statement, but WILL start a transaction when it gets to the INSERT statement. The with conn: context manager will ensure the transaction is either committed (if there's no exception) or rolled back (if there is an exception) at the end of the block.
I could say more about when and why transactions are started automatically, but my advice is to set isolation_level to None, which stops Python from ever starting transactions automatically. You can then begin them manually for yourself, as progmatico's answer said:
# Approach 2: you manually start transactions in your code
conn = sqlite3.connect(path_to_db, isolation_level=None)
with conn:
conn.execute("BEGIN") # Starts the transaction
values = [(row[0], row[1]) for row in conn.execute("SELECT a, b FROM Foo")]
conn.execute("UPDATE Foo SET a=? WHERE b=?", (values[0][0] + 1, values[0][1]))
# At this point, with conn: will automatically commit/rollback the transaction
Or don't use them at all:
# Approach 3: no explicit transactions at all
conn = sqlite3.connect(path_to_db, isolation_level=None)
values = [(row[0], row[1]) for row in conn.execute("SELECT a, b FROM Foo")]
conn.execute("UPDATE Foo SET a=? WHERE b=?", (values[0][0] + 1, values[0][1]))
In the last code snippet, we never started a transaction (and neither did Python!), so the underlying SQLite library automatically put each of the two statements in their own implicit transactions. (This is what the underlying SQLite library documentation calls "autocommit mode". Oops, I did mention that term again!)
So, hopefully you can see that it doesn't make sense to ask whether isolation_level=None makes your code more efficient. It lets you choose when to start and finish transactions, if at all, as in approach 2 vs approach 3. But it's still up to you when you do that, and that's what affects the efficiency of your program. If you're inserting a lot of records, it's more efficient to do them in one transaction, like approach 2. But you don't want to keep transactions open for too long because that could block other threads/processes trying to use that database. And there's typically not much speed benefit to executing SELECT statements in a transaction, so approach 3 might be the way to go (but a transaction might be necessary for correctness anyway, depending on the application).
The answer to your question is literally written here.
It basically says that you can stop the sqlite3 Python module from starting transactions implicitly on any data modification statement you send, allowing you the user, to assume the control of transactions ocurring in the underlying sqlite library with BEGIN, ROLLBACK, SAVEPOINT, and RELEASE statements in your code.
This is useful to group several statements that should be executed as part of a single transaction, instead of having every single modification statement executing in its own implicit transaction.
I am using PostgreSQL as a job queue. Following is my query to retrieve a job and update its state:
UPDATE requests AS re
SET
started_at = NOW(),
finished_at = NULL
FROM (
SELECT
_re.*
FROM requests AS _re
WHERE
_re.state = 'pending'
AND
_re.started_at IS NULL
LIMIT 1
FOR UPDATE SKIP LOCKED
) AS sub
WHERE re.id = sub.id
RETURNING
sub.*
Now, I have several machines, on each machine I have 1 process with several threads, and on each thread I have a worker. All workers in the same process shared a connection pool, typically having 10 - 20 connections.
The problem is, the above query will return some rows more than once!
I cannot find any reasons. Could anyone help?
To be more detailed, I am using Python3 and psycopg2.
Update:
I have tried #a_horse_with_no_name's answer, but seems not work.
I noticed that, one request is retrieved by two queries with the started_at updated to:
2016-04-21 14:23:06.970897+08
and
2016-04-21 14:23:06.831345+08
which are only differed by 0.14s.
I am wondering if at the time those two connections executes the inner SELECT subquery, both locks are not established yet?
Update:
To be more precise, I have 200 workers (i.e. 200 threads) in 1 process on 1 machine.
Please also note that it's essential that each thread has it's own connection if you do not want them to get in each others way.
If your application uses multiple threads of execution, they cannot
share a connection concurrently. You must either explicitly control
access to the connection (using mutexes) or use a connection for each
thread. If each thread uses its own connection, you will need to use
the AT clause to specify which connection the thread will use.
from: http://www.postgresql.org/docs/9.5/static/ecpg-connect.html
All kinds of wierd things happen if two threads share the same connection. I believe this is what is happening in your case. If you take a lock with one connection, all other threads that use the same connection will have access to the locked objects.
Permit me to suggest an alternative approach, that is really simple. The use of redis as a queue. You can either simply make use of redis-py and the lpush/rpop methods or use python-rq.
There is a chance a locking transaction is not yet issued at the time of the select, or the lock is lost by the time the results of the select are ready and the update statement begins. Have you tried explicitly beginning a transaction?
BEGIN;
WITH req AS (
SELECT id
FROM requests AS _re
WHERE _re.state = 'pending' AND _re.started_at IS NULL
LIMIT 1 FOR UPDATE SKIP LOCKED
)
UPDATE requests SET started_at = NOW(), finished_at = NULL
FROM req
WHERE requests.id = req.id;
COMMIT;
I've got some code that is trying to create 100 threaded http calls. It seems to be getting capped at about 40.
When I do threadJoin I'm only getting 38 - 40 sets of results from my http calls, despite the loop being from 1 - 100.
// thread http calls
pages = 100;
for (page="1";page <= pages; page++) {
thread name="req#page#" {
grabber.setURL('http://site.com/search.htm');
// request headers
grabber.addParam(type="url",name="page",value="#page#");
results = grabber.send().getPrefix();
arrayAppend(VARIABLES.arrResults,results.fileContent);
}
}
// rejoin threads
for (page="2";page <= pages; page++) {
threadJoin('req#page#',10000);
}
Is there a limit to the number of threads that CF can create? Is it to do with Java running in the background? Or can it not handle that many http requests?
Is there a just a much better way for me to do this than threaded HTTP calls?
The result you're seeing is likely because your variables aren't thread safe.
grabber.addParam(type="url",name="page",value="#page#");
That line is accessing Variables.Page which is shared by all of the spawned threads. Because threads start at different times, the value of page is often different from the value you think it is. This will lead to multiple threads having the same value for page.
Instead, if you pass page as an attribute to the thread, then each thread will have its own version of the variable, and you will end up with 100 unique values. (1-100).
Additionally you're writing to a shared variable as well.
arrayAppend(VARIABLES.arrResults,results.fileContent);
ArrayAppend is not thread safe and you will be overwriting versions of VARIABLES.arrResults with other versions of itself, instead of appending each bit.
You want to set the result to a thread variable, and then access that once the joins are complete.
thread name="req#page#" page=Variables.page {
grabber.setURL('http://site.com/search.htm');
// request headers
grabber.addParam(type="url",name="page",value="#Attributes.page#");
results = grabber.send().getPrefix();
thread.Result = results.fileContent;
}
And the join:
// rejoin threads
for (page="2";page <= pages; page++) {
threadJoin('req#page#',10000);
arrayAppend(VARIABLES.arrResults, CFThread['req#page#'].Result);
}
In the ColdFusion administrator, there's a setting for how many will run concurrently, mine's defaulted to 10. The rest apparently are queued. An Phantom42 mentions, you can up the number of running CF threads, however, with 100 or more threads, you may run into other problems.
On 32-bit processes, your whole process can only use 2gig of memory. Each thread uses up an amount of Stack memory, which isn't part of the heap. We've had problems with running out of memory with high numbers of threads as your Java Binary+Heap+Non-Heap(PermGen)+(threads*512k) can easily go over the 2-gig limit.
You'd also have to allow enough threads to handle your code above, as well as other requests coming into your app, which may bog down the app as a whole.
I would suggest changing your code to create N threads, each of which does more than 1 request. It's more work, but you break the N requests=N Threads problem. There's a couple of approaches you can take:
If you think that each request is going to take roughly the same time, then you can split up the work and give each thread a portion to work on before you start each one up.
Or each thread picks a URL off a list and processes it, you can then join to all N threads. You'd need to make sure you put locking around whatever counter you used to track progress though.
Check your Maximum number of running JRun threads setting in ColdFusion Administrator under the Request Tuning tab. The default is 50.
What is the difference between a dead lock and a race around condition in programming terms?
Think of a race condition using the traditional example. Say you and a friend have an ATM cards for the same bank account. Now suppose the account has $100 in it. Consider what happens when you attempt to withdraw $10 and your friend attempts to withdraw $50 at exactly the same time.
Think about what has to happen. The ATM machine must take your input, read what is currently in your account, and then modify the amount. Note, that in programming terms, an assignment statement is a multi-step process.
So, label both of your transactions T1 (you withdraw $10), and T2 (your friend withdraws $50). Now, the numbers below, to the left, represent time steps.
T1 T2
---------------- ------------------------
1. Read Acct ($100)
2. Read Acct ($100)
3. Write New Amt ($90)
4. Write New Amt ($50)
5. End
6. End
After both transactions complete, using this timeline, which is possible if you don't use any sort of locking mechanism, the account has $50 in it. This is $10 more than it should (your transaction is lost forever, but you still have the money).
This is a called race condition. What you want is for the transaction to be serializable, that is in no matter how you interleave the individual instruction executions, the end result will be the exact same as some serial schedule (meaning you run them one after the other with no interleaving) of the same transactions. The solution, again, is to introduce locking; however incorrect locking can lead to dead lock.
Deadlock occurs when there is a conflict of a shared resource. It's sort of like a Catch-22.
T1 T2
------- --------
1. Lock(x)
2. Lock(y)
3. Write x=1
4. Write y=19
5. Lock(y)
6. Write y=x+1
7. Lock(x)
8. Write x=y+2
9. Unlock(x)
10. Unlock(x)
11. Unlock(y)
12. Unlock(y)
You can see that a deadlock occurs at time 7 because T2 tries to acquire a lock on x but T1 already holds the lock on x but it is waiting on a lock for y, which T2 holds.
This bad. You can turn this diagram into a dependency graph and you will see that there is a cycle. The problem here is that x and y are resources that may be modified together.
One way to prevent this sort of deadlock problem with multiple lock objects (resources) is to introduce an ordering. You see, in the previous example, T1 locked x and then y but T2 locked y and then x. If both transactions adhered here to some ordering rule that says "x shall always be locked before y" then this problem will not occur. (You can change the previous example with this rule in mind and see no deadlock occurs).
These are trivial examples and really I've just used the examples you may have already seen if you have taken any kind of undergrad course on this. In reality, solving deadlock problems can be much harder than this because you tend to have more than a couple resources and a couple transactions interacting.
As always, use Wikipedia as a starting point for CS concepts:
http://en.wikipedia.org/wiki/Deadlock
http://en.wikipedia.org/wiki/Race_condition
A deadlock is when two (or more) threads are blocking each other. Usually this has something to do with threads trying to acquire shared resources. For example if threads T1 and T2 need to acquire both resources A and B in order to do their work. If T1 acquires resource A, then T2 acquires resource B, T1 could then be waiting for resource B while T2 was waiting for resource A. In this case, both threads will wait indefinitely for the resource held by the other thread. These threads are said to be deadlocked.
Race conditions occur when two threads interact in a negatve (buggy) way depending on the exact order that their different instructions are executed. If one thread sets a global variable, for example, then a second thread reads and modifies that global variable, and the first thread reads the variable, the first thread may experience a bug because the variable has changed unexpectedly.
Deadlock :
This happens when 2 or more threads are waiting on each other to release the resource for infinite amount of time.
In this the threads are in blocked state and not executing.
Race/Race Condition:
This happens when 2 or more threads run in parallel but end up giving a result which is wrong and not equivalent if all the operations are done in sequential order.
Here all the threads run and execute there operations.
In Coding we need to avoid both race and deadlock condition.
I assume you mean "race conditions" and not "race around conditions" (I've heard that term...)
Basically, a dead lock is a condition where thread A is waiting for resource X while holding a lock on resource Y, and thread B is waiting for resource Y while holding a lock on resource X. The threads block waiting for each other to release their locks.
The solution to this problem is (usually) to ensure that you take locks on all resources in the same order in all threads. For example, if you always lock resource X before resource Y then my example can never result in a deadlock.
A race condition is something where you're relying on a particular sequence of events happening in a certain order, but that can be messed up if another thread is running at the same time. For example, to insert a new node into a linked list, you need to modify the list head, usually something like so:
newNode->next = listHead;
listHead = newNode;
But if two threads do that at the same time, then you might have a situation where they run like so:
Thread A Thread B
newNode1->next = listHead
newNode2->next = listHead
listHead = newNode2
listHead = newNode1
If this were to happen, then Thread B's modification of the list will be lost because Thread A would have overwritten it. It can be even worse, depending on the exact situation, but that's the basics of it.
The solution to this problem is usually to ensure that you include the proper locking mechanisms (for example, take out a lock any time you want to modify the linked list so that only one thread is modifying it at a time).
Withe rest to Programming language if you are not locking shared resources and are accessed by multiple threads then its called as "Race condition", 2nd case if you locked the resources and sequences of access to shared resources are not defined properly then threads may go long waiting for the resources to use then its a case of "deadlock"