I have a lot of C++11 threads running which all need database access at some time. In main I do initalize the database connection and open the database. Qt documentation says that queries are not threadsafe so I use a global mutex until a QSqlQuery exists inside a thread.
This works but is that guaranteed to work or do I run into problems at some time?
A look at the Documentation tells us, that
A connection can only be used from within the thread that created it.
Moving connections between threads or creating queries from a
different thread is not supported.
So you do indeed need one connection per thread. I solved this by generating dynamic names based on the thread:
auto name = "my_db_" + QString::number((quint64)QThread::currentThread(), 16);
if(QSqlDatabase::contains(name))
return QSqlDatabase::database(name);
else {
auto db = QSqlDatabase::addDatabase( "QSQLITE", name);
// open the database, setup tables, etc.
return db;
}
In case you use threads not managed by Qt make use of QThreadStorage to generate names per thread:
// must be static, to be the same for all threads
static QThreadStorage<QString> storage;
QString name;
if(storage.hasLocalData())
name = storage.localData();
else {
//simple way to get a random name
name = "my_db_" + QUuid::createUuid().toString();
storage.setLocalData(name);
}
Important: Sqlite may or may not be able to handle multithreading. See https://sqlite.org/threadsafe.html. As far as I know, the sqlite embedded into Qt is threadsafe, as thats the default, and I could not find any flags that disable it in the sourcecode. But If you are using a different sqlite version, make shure it does actually support threads.
You can write class with SQL functions and use signals-slots to do the queries and get result from database.
It's thread-safe also no need to use mutex.
You choose not well approach. Should use shared QSqlDatabase object instead QSqlQuery. Please check next example of multithreading database access. If that will not clear for you please let me know. Will explain more.
Related
I've got a "database lock" issue despite the sqlite has been compiled with -DSQLITE_THREADSAFE=2 and each thread uses its own database connection like this simplified code:
thread *t = new thread(RandomTextThread);
t.join();
and
static void RandomTextThread(void * aArg)
{
RandomText *rt1;
rt1 = new RandomText();
rt1->genRandomText();
}
and
RandomText::RandomText() {
int rc;
//
// open the database
//
rc = sqlite3_open("mydata.db.sqlite", &this->db);
}
So within each thread constructed by the RandomTextThread statical function, a new RandomText object will be created which in turn creates a new sqlite connection stored within the ->db class variable.
Sometimes the code will be execute DML commands. And this works for one thread without an error.
But two or more threads created by the mechanism above fails until as soon the threads will be make concurrent DML.
The system is old, I could compile sqlite 3.27 but I can use only GCC 4.2 with the TinyThread++ solution which in turn uses POSIX pthread.h .
Any help appreciate.
SQLITE_BUSY is an expected error code when a database is both read and modified at the same time. Using Write-Ahead Logging, it is possibly to greatly reduce the number of occurrences if there is just one writer and many readers.
sqlite3_exec may keep a lock after completion if you have disabled auto-commit mode, for example using the BEGIN command. In this case, you must issue a COMMIT or ROLLBACK command to release the lock.
You can use sqlite3_busy_timeout to install a buys handler which performs waiting inside SQLite itself.
I'm dealing with a situation where multiple threads are accessing this method
using (var tx = StateManager.CreateTransaction())
{
var item = await reliableDictioanary.GetAsync(tx, key);
... // Do work on a copy of item
await reliableDictioanary.SetAsync(tx, key, item);
await tx.CommitAsync();
}
Single threading this works well, but when I try accessing the dictionary this way using multiple threads I encounter a System.TimeOutException.
The only way I've been able to get around it is to use LockMode.Update on the GetAsync(...) method. Has anyone here experienced something like this?
I'm wondering if there is a way to read with snapshot isolation, which would allow a read with no lock on it, as opposed to a read with a shared lock on the record.
I've tried doing this with both a shared transaction as shown above as well as individual transactions for the get and the set. Any help would be appreciated.
The default lock when reading, is a shared lock. (caused by GetAsync)
If you want to write, you need an exclusive lock. You can't get it if shared locks exist.
Getting the first lock as an update lock prevents this, like you noticed.
Snapshot isolation happens when enumerating records, which you're not doing with GetAsync.
More info here.
I've read a lot about thread safety when reading variable simultanously from multiple threads but I am still not sure whether my case is fine or not.
Consider that I have:
const
MySettings: TFormatSettings =
(
CurrencyFormat : 0;
NegCurrFormat : 0;
ThousandSeparator: ' ';
DecimalSeparator : '.';
CurrencyString : 'ยค';
ShortDateFormat : 'MM/dd/yyyy';
LongDateFormat : 'dddd, dd MMMM yyyy';
//All fields of record are initialized.
);
Can I use FormatDateTime('dd/mm/yyyy hh:nn:ss', MySettings, Now) in multiple threads without worries or should I spawn a separate copy of MySettings for each thread?
This scenario is threadsafe if and only if the format settings record is not modified during the simultaneous calls to formatting functions.
Indeed the old school formatting functions that used a shared global format settings record were threadsafe if and only if the shared object was not modified. This is the key point. Is the format settings object modified or not?
My take on all this is that you should avoid modifying format settings objects. Initialise them and then never modify them. That way you never have thread safety issues.
Yes this is perfectly safe.
As long as MySetting is not changed this is the way to use FormatDateTime and other similar procedures.
From documentation, System.SysUtils.TFormatSettings:
A variable of type TFormatSettings defines a thread-safe context that formatting functions can use in place of the default global context, which is not thread-safe.
N.B. You must provide this thread-safe context by programming. It is thread-safe only if you ensure that the parameter and its shares is not changed during execution.
Typically my serializing libraries are using a shared constant format setting variable, which provides a stable read/write interface in all locales.
I'm running MongoDB on Windows. I have 1 or more threads that drop and recreate a collection.
Using mongo.exe with the show collections() command, I'm seeing multiple collections with the same name (well over 1,000 collections with the same name!).
When I run validate:
> db.MY_COLLECTION.validate()
I get:
{ "errmsg" : "ns not found", "ok" : 0, "valid" : false }
The size() command returns 0, and find() returns nothing.
My question is: Is MongoDB thread safe? A follow on question would be something like 'Am I doing this correctly (dropping and recreating) or is there a better way to refresh the whole content of a collection?'
From mongodb documentation:
Thread safety
Only a few of the C# Driver classes are thread safe. Among them: MongoServer, MongoDatabase, MongoCollection and MongoGridFS. Common classes you will use a lot that are not thread safe include MongoCursor and all the classes from the BSON Library (except BsonSymbolTable which is thread safe). A class is not thread safe unless specifically documented as being thread safe.
All static properties and methods of all classes are thread safe.
You can search for the word Thread on this page:
http://mongodb.onconfluence.com/pages/viewpage.action?pageId=18907708&navigatingVersions=true#CSharpDriverTutorial-Threadsafety
Changed in version 2.2.
MongoDB allows multiple clients to read and write a single corpus of data using a locking system to ensure that all clients receive a consistent view of the data and to prevent multiple applications from modifying the exact same pieces of data at the same time. Locks help guarantee that all writes to a single document occur either in full or not at all.
http://docs.mongodb.org/manual/faq/concurrency/
What I need is a system I can define simple objects on (say, a "Server" than can have an "Operating System" and "Version" fields, alongside other metadata (IP, MAC address, etc)).
I'd like to be able to request objects from the system in a safe way, such that if I define a "Server", for example, can be used by 3 clients concurrently, then if 4 clients ask for a Server at the same time, one will have to wait until the server is freed.
Furthermore, I need to be able to perform requests in some sort of query-style, for example allocate(type=System, os='Linux', version=2.6).
Language doesn't matter too much, but Python is an advantage.
I've been googling for something like this for the past few days and came up with nothing, maybe there's a better name for this kind of system that I'm not aware of.
Any recommendations?
Thanks!
Resource limitation in concurrent applications - like your "up to 3 clients" example - is typically implemented by using semaphores (or more precisely, counting semaphores).
You usually initialize a semaphore with some "count" - that's the maximum number of concurrent accesses to that resource - and you decrement this counter every time a client starts using that resource and increment it when a client finishes using it. The implementation of semaphores guarantees the "increment" and "decrement" operations will be atomic.
You can read more about semaphores on Wikipedia. I'm not too familiar with Python but I think these two links can help:
Python Threading Library
Semaphore Objects in Python.
For Java there is a very good standard library that has this functionality:
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/package-summary.html
Just create a class with Semaphore field:
class Server {
private static final MAX_AVAILABLE = 100;
private final Semaphore available = new Semaphore(MAX_AVAILABLE, true);
// ... put all other fields (OS, version) here...
private Server () {}
// add a factory method
public static Server getServer() throws InterruptedException {
available.acquire();
//... do the rest here
}
}
Edit:
If you want things to be more "configurable" look into using AOP techniques i.e. create semaphore-based synchronization aspect.
Edit:
If you want completely standalone system, I guess you can try to use any modern DB (e.g. PostgreSQL) system that support row-level locking as semaphore. For example, create 3 rows for each representing a server and select them with locking if they are free (e.g. "select * from server where is_used = 'N' for update"), mark selected server as used, unmark it in the end, commit transaction.