Is there a way (in Linux) of getting updates on the lockedness status of a file without polling?
I know that the status can be polled via a lockf(fd, F_TEST) or speculative LOCK_NB|LOCK_SH, but polling is bad(tm).
Of course, finding out when a file is NOT locked can be done with a simple lock attempt, but I want to sample the other edge too (use-case: a (large) program uses lockf to synchronize between instances - I can probably get it changed to flock, and I want to add a GUI that displays when the lock is acquirable, of course while not hogging the lock).
Note that inotify does not work in this case, at least on linux 3.9.1.
Related
I'm currently creating a Linux driver for block devices. This has been going on for some time, and I just recently changed the driver design from bio-mode to request-mode (I used to handle struct bio but now I'm operating on struct request) and it made the functionality simpler, but the change also presented some new issues of which I'm requesting advice. I'm also using a kthread for backgrond maintenance and periodic flushing of entries in my driver-device circular buffers.
One of those issues that I'm having difficulty handling or fully understanding is regarding the slowness and presence of sbin/blkid process after entering commands that alter partition or filesystem type. With my current design, whenever I run fdisk or mkfs or just any command that modify the partition/FS type, upon checking running processes via the ps -ef terminal command, the blkid process runs afterwards and will take around a minute or two before completing and updating drive info in Disk Utility (which the user sees). This is of course too long to wait, and I have to wait before entering another command that modify the partition/FS type, or else the drive info gets messed up, partition table is lost, etc.
I found out that when I came back to my previous bio-mode driver (which did not have kthread) that this blkid task also runs after the same commands as above, but finishes very quickly! I just became aware of blkid because of its slowness now, and not then because it easily completes. And so I decided to play with my driver code, removing stuffs to see which code segment causes slow blkid, and eventually found that the presence of kthread correlates to the slowness of blkid.
The body of my kthread BTW looks as follows:
while (kthread_should_stop()) {
set_current_state(TASK_INTERRUPTIBLE);
spin_lock_irq(&lock);
< driver code here... >
spin_unlock_irq(&lock);
schedule_timeout(msecs_to_jiffies(250));
}
Here's what I did to my kthread:
- slowly remove code until nothing is left between the spin_lock/unlock calls. sbin/blkid is still slow.
- remove kthread and replace it with timer that periodically expires, resets itself, and upon every expiry calls a function, to which I moved the contents of the kthread driver code. sbin/blkid is still slow.
- change the schedule_timeout time value from 250ms to 1s - blkid became even slower > 3mins. If I lessen it to 50ms, it's as slow as during 250ms timeout.
I'm really confused - how does my driver kthread and blkid interact in such a way it slows down blkid?
I'm using Open MPI 1.8 on Gentoo 3.13 to manage the data transfer from one program to another via a server/client concept. Both the server and the clients are launched via mpiexec as separate processes. After some days (this is quite a heavy computation...), I sometimes receive the error
mpiexec noticed that process rank 0 with PID 17213 on node XXX exited on signal 26 (Virtual timer expired).
Unfortunately, the error is not reproducible in a reliable way, i.e., the error does not appear always and not always at the same point in the program flow. I also experienced this error on other machines. I already tracked the issue down to the ITIMER_VIRTUAL which, upon expiration, delivers SIGVTALRM (see, e.g., http://man7.org/linux/man-pages/man2/setitimer.2.html). In the BUGS section of the man page, it says that
Under very heavy loading, an ITIMER_REAL timer may expire before the signal from a previous expiration has been delivered. The second signal in such an event will be lost.
I wonder if something similar might also hold for ITIMER_VIRTUAL? Did anyone experience similar problems and can confirm the error?
The only workaround I can think of is to invoke setitimer(...) and try to manipulate the timer myself. However, I hope there is another way since I can't always modify the clients' source code. Any suggestions?
Since this question has not been answered officially, I will do it on behalf of Hristo (#HristoIliev: I hope this is ok for you). As was pointed out in the first comment to my question, there is not a single hint in the Open MPI source code which can have caused the virtual timer expiration. Indeed, the timer problem was related to a third-party library which made the code crash after an unpredictable time (depending on the current loading of the machine).
The program I am developing uses threads to deal with long running processes. I want to be able to use Gauge Pulse to show the user that whilst a long running thread is in progress, something is actually taking place. Otherwise visually nothing will happen for quite some time when processing large files & the user might think that the program is doing nothing.
I have placed a guage within the status bar of the program. My problem is this. I am having problems when trying to call gauge pulse, no matter where I place the code it either runs to fast then halts, or runs at the correct speed for a few seconds then halts.
I've tried placing the one line of code below into the thread itself. I have also tried create another thread from within the long running process thread to call the code below. I still get the same sort of problems.
I do not think that I could use wx.CallAfter as this would defeat the point. Pulse needs to be called whilst process is running, not after the fact. Also tried usin time.sleep(2) which is also not good as it slows the process down, which is something I want to avoid. Even when using time.sleep(2) I still had the same problems.
Any help would be massively appreciated!
progress_bar.Pulse()
You will need to find someway to send update requests to the main GUI from your thread during the long running process. For example, if you were downloading a very large file using a thread, you would download it in chunks and after each chunk is complete, you would send an update to the GUI.
If you are running something that doesn't really allow chunks, such as creating a large PDF with fop, then I suppose you could use a wx.Timer() that just tells the gauge to pulse every so often. Then when the thread finishes, it would send a message to stop the timer object from updating the gauge.
The former is best for showing progress while the latter works if you just want to show the user that your app is doing something. See also
http://wiki.wxpython.org/LongRunningTasks
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
http://www.blog.pythonlibrary.org/2013/09/04/wxpython-how-to-update-a-progress-bar-from-a-thread/
I am new to VC++ and programming. I have a task in which I am supposed to design a file watcher in VC++.
The problem goes this way:
I have to monitor some log files continously; whenever a particular log file gets deleted(this deletion is done by some other program), I have to open a TextFile and write some data and the timestamp into it.
How do I go about it? Please help!!
First, you need to setup a system to monitor for file events from that folder.
To get started, take a look at FindFirstChangeNotification().
You'll basically get a waitable handle from that.
Then, were it me, I'd have a thread that waited on that event. Each time the event triggers, the thread resumes, queries for the change details (what file), then perform the needed actions, and resume sleeping on that handle again.
You'll need some additional semaphore or something to use to interrupt this worker-thread and wake it so that you can tell it to quit. Simple to do: have your thread's main loop do a WaitForMultipleObjects - the "wake up semaphore" and the FindFirstChangeNotification handle. When you wake up, check which even notified you, then either process the file change or quit.
MFC has a slightly different way of handling it (slightly) but to do this using the Win32 API what you'd typcially do is use the Directory Management Functions to set up a change notification handle for the directory the file goes in. Then you can wait on the handle and when something happens inside that directory your wait completes, and you can check to see if it was a change to the file that you care about.
Look at the docs for FindFirstChangeNotification and ReadDirectoryChangesW for more information.
Try the Windows Management Instrumentation (WMI) if you have enough privileges. AFAIK it is also the most efficient way to handle the filesystem events.
Handle or query the __InstanceDeletionEvent, __InstanceModificationEvent or __InstanceCreationEvent for the deletion, modification or creation events respectively and filter the files and target path that you want.
Take a look at the WMI Reference/C++ invocation.
For a full-scale example take a look at codeproject querying example.
I strongly recommmend you consider using the implementation here. This API is not 100% reliable, but this code does a good job of wrapping it. If your filesystem traffic is local and not too frequent, it should work well for you.
Can someone explain how persistent watch works?
Does it eat some resources on the PC if it is watching file(s) for changes?
Thanks ;)
fs.watchFile creates a StatWatcher which then performs a stat on the file that's being watched. How exactly that happens at the low level (besided doing the obvious stats call) is dependent on the event loop implementation with which node was compiled.
So yes, it takes up a bit of CPU, but at you can't do anything else besides polling here, that is, unless the underlying file system itself would issue the file change events.
See:
https://github.com/ry/node/blob/v0.3.2/lib/fs.js#L472
https://github.com/ry/node/blob/v0.3.2/src/node_stat_watcher.h#L39
https://github.com/ry/node/blob/v0.3.2/src/node_stat_watcher.cc#L78
Some more info on the parameters
Interval is relavent where inotify is not available - it determines
how long to poll for updates.
Persistent has to do with how the program should act when nothing but
watchFile is running. The default is to exit.
As far as I saw, it takes 3--5 seconds to notice the changes (with the
default settings), can i make it faster?
On linux it uses inotify - which is faster
how heavy is watching hundreds of files?
Heavy. It's not meant for that.
Source: Post on the Node.js Google Group by Ryan Dahl
In conclusion
If you're on linux, the interval option has no effect et all.
If you don't set persistent and there's nothing else in the event loop besides the file watcher, the program will exit.