I have to debug a program that writes a log file. It takes time for the actual log file to be generated because it takes a while to flush the contents to disk. On top of that, the log file is on a mounted Unix drive on my windows machine. I was wondering if there is a command to make the operating system flush the written contents to disk. Does it also take a while for the file to be updated on the mounted drive in windows?
PS. I actually; don't want to go in and edit the program.
Ted
Look at the following APIs:
http://www.cplusplus.com/reference/clibrary/cstdio/setvbuf/
fsync
Also see the ever-great eat my data: how everybody gets file IO wrong
Related
We are working on an iMX6Sx Freescale board, building the Linux kernel distro with Yocto.
I would like to know if there is a way to check if it is possible to check if file system operations (in particular, write) are really terminated, avoiding to close/kill a process while operations are still running.
To be more clear: we have to do some actions (copy of files, writes, ..) when our application has to switch-off and we have to know (since they are asynchronus I think) when they're are really completed.
Thanks in advance
Andrea
If you want to ensure all the writes are commited to storage and the filesystem is updated:
call fsync() on the file descriptor,
open the parent directory and call fsync() on that file descriptor
When both of these are done, the kernel has flushed everything from memory and ensured the filesystem is updated regarding the file you operate on.
Another approach is to call sync(), which ensures all kernel data are written to storage for all files and filesystem metadata.
Note:
if your application are working with FILE* instead of file descriptors, you need to first ensure written data are flushed from your application to the kernel, either by calling fflush() or fclose the FILE*
If you kill an application, any write operation it has performed will not be cancelled or interrupted, and you can make sure it's committed to storage by calling sync() or open the same file and call fsync() on it.
If you kill an application arbitrarily you can't expect everything to be consistent, perhaps the application was doing 2 important writes to a database, config file, etc. and you terminated it after the 1 write, the file might be damaged according to its format.
I have wrote a program and it will update a file periodically, sometimes I want to copy the file into another computer to check its content. If I copied the file when the program was not writing it, there is no problem. But, if I copied the file when the program was writing it, the copied file would be partial. So, I wonder that, if the Linux system exists the lock strategy to prevent the situation.
In fact, I copy the file in a bash script, so I want to check if the program is writing it in the bash script. If yes, the bash script will check its state after some seconds and then copy its completed version. So in bash script, how could we check the file was opened or modified by other programs?
You could check from your script whether the file is being written to, and abort/pause copy if it is...
fuser -v /path/to/your/file | awk 'BEGIN{FS=""}$38=="F"{num++}END{print num}'
If the output is smaller 1 you're good to copy :)
When your code writes into the file, it actually writes into an output buffer in memory. The buffer will be flushed out to disk when it becomes full. Thus, when you copy the file whose buffer has not been flushed out to disk, you will observe partial file.
You can modify the buffer size by using the call to setvbuf. If you set the buffer size to zero, it will get flushed out as it is written. Another thing you can do is to make a call to fflush() to flush the output to disk. Either of these two should update the file as it is written.
Say I want to synchronize data buffers of a file system to disk (in my case the one of an USB stick partition) on a linux box.
While searching for a function to do that I found the following
DESCRIPTION
sync() causes all buffered modifications to file metadata and
data to be written to the underlying file sys‐
tems.
syncfs(int fd) is like sync(), but synchronizes just the file system
containing file referred to by the open file
descriptor fd.
But what if the file system has no file on it that I can open and pass to syncfs? Can I "abuse" the dot file? Does it appear on all file systems?
Is there another function that does what I want? Perhaps by providing a device file with major / minor numbers or some such?
Yes I think you can do that. The root directory of your file system will have at least one inode for your root directory. You can use the .-file to do that. Play also around with ls -i to see the inode numbers.
Is there a possibility to avoid your problem by mounting your file system with sync? Does performance issues hamper? Did you have a look at remounting? This can sync your file system as well in particular cases.
I do not know what your application is, but I suffered problems with synchronization of files to a USB stick with the FAT32-file system. It resulted in weird read and write errors. I can not imagine any other valid reason why you should sync an empty file system.
From man 8 sync description:
"sync writes any data buffered in memory out to disk. This can include (but is not
limited to) modified superblocks, modified inodes, and delayed reads and writes. This
must be implemented by the kernel; The sync program does nothing but exercise the sync(2)
system call."
So, note that it's all about modification (modified inode, superblocks etc). If you don't have any modification, it don't have anything to sync up.
Assuming that i have Loadable-Kernel-Module inserted in linux-kernel and have hooked read, write, open and close functions. So now i can stop access to any file but i want to stop files from being copied outside the device like to a usb device, card, disk etc. The thing i want to know is that sitting in LKM and with function calls hooked how can i identify that a file is being written to external device?.
Also i want to know that which system calls are used during a copy operation ? I have idea that a program opens the file reads from it ( read system call) and then writes to second file( write system call) but i observed strange behavior when i was trying to stop write access to a file that a process which opens a file never calls write operation on that file for saving file (checked for pdf viewer).
If anybody have idea about this strange behavior or you have idea that how to stop writing to a file then please share it also.
They could mmap it to do read/write. Or they could read the entire original file into memory, close it, then open the destination.
Or they could encrypt the file, then write it out to a new file on the USB.
Or they could do minor edits to the contents, then save it out.
Or they could use gvfs to access the network/USB device.
Or the user could reboot and copy the file in a different OS.
All that really highlights is that the problem is really difficult - a determined user will always find a way to extract data from a system they have access to.
You're best bet is just to prevent accidental leakage - so scan files after close on the removable media, and check they don't have contents you don't want leaked. Overwrite and delete if they do.
Or else block the devices from being mounted in the first place, and disable gvfs as well.
As to why your hook isn't intercepting the write(), either:
Your hook isn't actually intercepting the operation.
The application isn't using write() to put the content in a file.
In Linux file system, there are files such as /dev/zero and /dev/random which are not real files on hard disk.
Is there any way that we can create a similar file and tell it to get ouput from executing a program?
For example, can I create file, say /tmp/tarfile, such that any program reading it actually gets the output from the execution of a different program (/usr/bin/tar ...)?
It is possible to create such a file/program, but it would require creation of a special filesystem in order to insert hooks into the VFS so that accesses can be detected and handled properly.