in Linux 2.6.27:
From "lsof" output I see a process holding open fd with a (deleted) file. The strange thing is that I can still see the file in the file system using "ls". Why is that?
thanks.
The file is not deleted as long as some process has the file open. When a file is closed, the kernel first checks the count of the number of process that have the file open. If this count has reached 0, the kernel then checks the link count; if it is 0, the file's contents are deleted.
To quote from man unlink:
If the name was the last link to a file but any processes still have
the file open the file will remain in existence until the last file
descriptor referring to it is closed.
When a file is deleted it would not been seen on the file system. However, it is quite possible another file with the same file name is created on the same location.
You can check the node number shown in lsof and ls -i to check whether they are really the same file.
Related
I'm considering making my application create a file in /tmp. All I'm doing is making a temporary file that I'll copy to a new location. On a good day this looks like:
Write the temp file
Move the temp file to a new location
However on my system (RHEL 7) there is a file at /usr/lib/tmpfiles.d/tmp.conf which indicates that /tmp gets cleaned up every 10 days. From what I can tell this is the default installation. So, what I'm concerned about is that I have the following situation:
Write the temp file
/tmp gets cleaned up
Move the temp file to a new location (explodes)
Is my concern founded? If so, how is this problem solved in sophisticated programs? If there are no sophisticated tricks, then it's a bit puzzling to me as I don't have a concrete picture of what the utility of /tmp is if it can be blown away completely at any moment.
this should not be a problem if you keep a file descriptor open during your operation. As long as a file descriptor is open, the FS keeps the file on disk but it just don't appear when using ls. So If you create another name for this file, it will "resurect" in some way. Keeping an open fd on a file that is deleted is a common way to create temporary files on linux
see the man 3 unlink:
The unlink() function shall remove a link to a file. [..] unlink() shall remove the link named by the pathname pointed to by
path and shall decrement the link count of the file
referenced by the link.
When the file's link count becomes 0 and no process has the file open, the space occupied by the file shall be freed and the file
shall no longer be accessible. If one or more
processes have the file open when the last link is removed, the link shall be removed before unlink() returns, but the removal of
the file contents shall be postponed until all
references to the file are closed.
I did some search but didn't find similar questions.
Here is my steps:
copy a file to /dev/shm (tmpfs)
mmap that file with lock
read data from that file
delete the file
After step 4, the previous loaded data is still available, why?
Thanks in advance
On Unix, a file's data is not deleted until every remaining process closes/munmaps it.
Since you've opened the file and have an active handle or mapping to it, you can continue reading and writing the file data for as long as you want. It will not be freed until after you close it.
I can redirect the output of a process to a file
./prog > a.txt
But if I delete a.txt and do not restart prog, then no more output will get into a.txt. The same is the case if I use the append-redirect >>.
Is there a way to make my redirection recreate the file when it is deleted during the runtime of prog?
Redirection is part of the OS I think and not of prog. So maybe there are some tools or settings.
Thanks!
At the OS level, a file is made up of many components:
the content, stored somewhere on the storage device;
an i-node that keeps all file information except the name;
the name, listed in a directory (also stored on the storage device);
when the file is open, each application that opens it handle memory buffers that keep some of the file content.
All these are linked and the OS keeps their booking.
If you delete the file while it is open by another application (the redirect operator > keeps it open until ./prog completes), only the name is removed from the directory. The other pieces of the puzzle are still there and they keep working until the last application that keeps the file open closes it. This is when the file content is discarded on the storage medium.
If you delete the file, while ./prog keeps running and producing output the file grows and uses space on the storage medium but it cannot be open again because there is no way to access it. Only the programs that have it already open when it was deleted can still access the file until they close it.
Even if you re-create the file, it is a different file that happens to have the same name as the deleted one. ./prog is not affected, its output goes to the old, deleted file.
When its output is redirected, apart from restarting ./prog, there is no way to persuade it to store its output in a different file when a.txt is deleted.
There are several ways to make this happen if ./prog writes itself into a.txt (they all require changing the code of ./prog).
You can use gdb to redirect the output of program to file when original file is deleted.
Refer to this post.
For later references, I give the only excerpt from the post:
Find the files that are opened by the process using /proc/<pid>/fd.
Attach the PID of program to gdb.
Close the file descriptor of the deleted file through gdb session.
Redirect the program output to another file using gdb calls.
Examples
Suppose that PID of program is 19080 and file descriptor of deleted file is 2.
gdb attach 19080
ls -l /proc/19080/fd
gdb> p close(2)
$1 = 0
gdb> p fopen("/tmp/file", "w")
$2 = 20746416
(gdb) p fileno($2)
$3 = 7
gdb> quit
N.B.: If data of the deleted file is required, recover the deleted text file before closing the file handle:
cp -pv /proc/19080/fd/2 recovered_file.txt
My process reads a files and deletes it. This activity happens more than 2000 times.
When I check the file in /proc/PID/fd, I see the file there and I see at the end of each line as (deleted). But I see 1024 records, with 1020 being the (deleted) entries. Later the new file operation from this PID fails.
To overcome this issue, kept process on debug and did
p close (id)
This (id) is taken from ll output on /proc/PID/fd.
Wanted to know the reason for the file not being deleted. fdclose is used first and then the file is deleted, even then file is shown with (deleted)
/proc/$PID/fd directory shows all the open files of the process named by their descriptors. Each file in /proc/$PID/fd represents an open file/socket/pipe etc., If the descriptor belongs to a disk file, then its symbolic link points to the absolute path of the file that is opened.
Here, (deleted) represents that the file that is opened by the process is deleted and no longer exist on disk. So, the issue in your case is that the file that is opened is not getting closed before unlink(delete). You need to close them before deleting it otherwise it leaks file descriptors.
If you are coding in C use fclose(C standard) or close(POSIX) appropriately to close the file before
I have a background process that is running for a long time and using a file to write the logs in it. It`s size has increased too large. I just deleted the file and created a new one with the same name and same permission and ownership but the new file does not get any entry.
Old file is marked as deleted and still being used by the process which can clearly be seen by lsof command.
Plz let me know, is there any way that I can recover that file and.
Your positive response will really be much helpful.
If the file is still open by some process, you can recover it using the /proc filesystem.
First, check the file descriptor number under which that file is opened in that process. If the file is opened in a process with PID X, use the lsof command as follows:
lsof -p X
This will show a list of files that are currently opened by X. The 4th column shows the file descriptors and the last column shows the name of the mount point and file system where the file lives (ignore the u, r and other flags after the file descriptor number, they just indicate whether the file is opened for reading, writing, etc.)
If the file descriptor number is Y, you can access its contents in /proc/X/fd/Y. So, something like this would recover it:
cp /proc/X/fd/Y /tmp/recovered_file