Python os.remove() is buffered and executed at a later point - python-3.x

For a test, I want to delete an SQLite database file before each run, so that the process that I am testing creates a new one each time. However, it seems that os.remove() is somehow buffered and then executes after the database has already been recreated, deleting the (already partially filled) database in the middle of my test!
Is there any method to ensure that the file has been deleted (on disk)? I do not want to disable io buffering in general, as I want to test the performance of my process in a realistic environment.

Related

How to frequently flush logs into output.xml file in robot framework

While we run some robot script, the current logs of the script gets flushed into output.xml. But this happens only when execution of couple of keywords are complete.
Is there a way to frequently flush the current logs into the output.xml file? Maybe a timer that we can specify, and when that expires, the current logs should be flushed into output.xml, no matter if the code was in middle of running a keyword.
This would greatly help in identifying and debugging some issues quickly rather than waiting for the entire keyword or script to complete.
I am using Robot 3.1.2 (Python 3.5.0 on linux).
Is there a way to frequently flush the current logs into the output.xml file?
No, there is not.
However, you can create your own output file with whatever data you want in it by creating a listener. A listener has access to all of the data that appears in the output.xml file, except for statistics.

Why is sqlite db database locked when there's no other python process writing to it at the same time?

In here Python sqlite3 doc, I see that the default timeout is 5 seconds.
I do connect without specifying the timeout. python 3.7.
I have 2 python processes that write to the same DB. This file.db is in WAL mode. I see the 2 files -shm and -wal there.
I get this exception sqlite3.Error: 'database is locked'
At the time of this exception, the other process had finished its transaction (got out of the with context manager) and had gone to sleep with signal.pause
The file is on a locally mounted regular hard drive.
How can i investigate what's happening besides doing:
I can check fuser -v on myfile.db
I do assume that any other writing process only locks the database while it is executing a transaction. Once it's finished, it releases the lock.
Besides, in WAL mode, are things supposed to work differently?
From the docs:
The sqlite3 module opens transactions implicitly before a Data Modification Language (DML) statement (i.e. INSERT/UPDATE/DELETE/REPLACE), and commits transactions implicitly before a non-DML, non-query statement (i. e. anything other than SELECT or the aforementioned).
In my case, I only had a UPDATE, which means a BEGIN was inserted by sqlite3 before that, followed but the signal.pause(), which means I hadn't encountered a non DML stmt, and the transaction was still active right before sleeping
The other process tried to run a UPDATE while the 1st was still asleep ( and the transact active).
I've added a conn.commit() before signal.pause()

replace a process bin file when it is running

I have a server program(compile by g++) which is running. And I change some code and compile a new bin file. Without kill the running process, I mv the new created bin to overwrite the old one.
After a while, the server process crashed. Dose it relate to my replace action?
My server is an multi-thread high concurrent server. One crash is segfault, other one is deadlock.
I print all parameters in the core dump file and pass them exactly same to the function which was crashed. But it is OK.
And I carefully watch all thread info in the deadlock core dump, I can not find it is an possibility to cause deadlock.
So I doubt the replacement will cause strange things
According to this question, if swap action is happen, it indeed will generate strange things
For a simple standard program, even if it is currently opened by the running process, moving a new file will first unlink the original file which will remain untouched apart from that.
But for long running servers, many things can happen: some fork new processes and occasionally some can even exec a new fresh version. In that case, you could have different versions running side by side which could or not be supported depending on the change.
Said differently, without more info on what is the server program, how it is designed to run and what was the change, the only answer I can give is maybe.
If you can make sure that you remove ONLY the bin file, and the bin file isn't used by any other process (such as some daemon). Then it doesn't relate to your replace action.

Scheduled disk scan

I have a script that writes to a file and then dumps that file to a database. I need this task to run as frequently as possible, but never run more than instance at the same time (or it'd be writing redundant stuff to the same file).
How I am currently doing it is in the shell script I am checking to see if a file exists, and if it does, I exit the script. At the end of each script it deletes the file.
This works 95% of the time. However, if the server is restarted (which happens semi-frequently), the file that was being written to will remain, and every time the script is called after that it will exit because the file already exists.
What would be a good way around this problem?
You could check to see if any processes are using the file with 'fuser'. It will return the PID of any program using the file. If there are no PIDS, you are safe to wipe it and start again.

SQLite issue - DB is locked workaround

I have 2 processes that connect to the same DB.
The first one is used to read from the DB and the second is used to write to the DB.
The first process sends write procedures for executing to the second process via message-queue on linux.
Every SQL-statement is taken in the prepare, step, finalize routine; Where the prepare and step are made in loop of 10000 times till it succedd (did this to overcome DB locked issues).
To add a table i do the next procedure:
the first process sends request via msg-q to the second process to add a table and insert garbage in it's rows in a journal_mode=OFF mode.
then the first process checks for the existing table so it could continue in its algorithm. (It checks it in a loop with usleep command between iterations.)
The problem is that the second process is stuck in the step execute of 'PRAGMA journal_mode=OFF;' because it says the DB is locked (Here too, i use a loop of 10000 iterations with usleep to check 10000 time for the DB to be free, as i mentioned before).
When i add to the first process in the 'check for existing table' loop, the operation of closing the connection, the second process is ok. But now when i add tables and values sometime i get 'callback requested query abort' in the Step Statement.
Any help of whats happening here ?
Use WAL mode. It allows one writer and any number of readers without any problems. You don't need to check for the locked state and do retrys etc.
WAL limitation: The DB has to be on the local drive.
Performance: Large transactions (1000s of inserts or similar) are slower than classic rollback journal, but apart of that the speed is very similar, sometimes even better. Perceived performance (UI waiting for DB write to finish) improves dramatically.
WAL is a new technology, but already used in Firefox, Adroid/iOS phones etc. I did tests with 2 threads running at full speed - one writing and the other one reading - and did not encounter a single problem.
You may be able to simplify your app when adopting the WAL mode.

Resources