I have a background process that is running for a long time and using a file to write the logs in it. It`s size has increased too large. I just deleted the file and created a new one with the same name and same permission and ownership but the new file does not get any entry.
Old file is marked as deleted and still being used by the process which can clearly be seen by lsof command.
Plz let me know, is there any way that I can recover that file and.
Your positive response will really be much helpful.
If the file is still open by some process, you can recover it using the /proc filesystem.
First, check the file descriptor number under which that file is opened in that process. If the file is opened in a process with PID X, use the lsof command as follows:
lsof -p X
This will show a list of files that are currently opened by X. The 4th column shows the file descriptors and the last column shows the name of the mount point and file system where the file lives (ignore the u, r and other flags after the file descriptor number, they just indicate whether the file is opened for reading, writing, etc.)
If the file descriptor number is Y, you can access its contents in /proc/X/fd/Y. So, something like this would recover it:
cp /proc/X/fd/Y /tmp/recovered_file
Related
I'm considering making my application create a file in /tmp. All I'm doing is making a temporary file that I'll copy to a new location. On a good day this looks like:
Write the temp file
Move the temp file to a new location
However on my system (RHEL 7) there is a file at /usr/lib/tmpfiles.d/tmp.conf which indicates that /tmp gets cleaned up every 10 days. From what I can tell this is the default installation. So, what I'm concerned about is that I have the following situation:
Write the temp file
/tmp gets cleaned up
Move the temp file to a new location (explodes)
Is my concern founded? If so, how is this problem solved in sophisticated programs? If there are no sophisticated tricks, then it's a bit puzzling to me as I don't have a concrete picture of what the utility of /tmp is if it can be blown away completely at any moment.
this should not be a problem if you keep a file descriptor open during your operation. As long as a file descriptor is open, the FS keeps the file on disk but it just don't appear when using ls. So If you create another name for this file, it will "resurect" in some way. Keeping an open fd on a file that is deleted is a common way to create temporary files on linux
see the man 3 unlink:
The unlink() function shall remove a link to a file. [..] unlink() shall remove the link named by the pathname pointed to by
path and shall decrement the link count of the file
referenced by the link.
When the file's link count becomes 0 and no process has the file open, the space occupied by the file shall be freed and the file
shall no longer be accessible. If one or more
processes have the file open when the last link is removed, the link shall be removed before unlink() returns, but the removal of
the file contents shall be postponed until all
references to the file are closed.
I can redirect the output of a process to a file
./prog > a.txt
But if I delete a.txt and do not restart prog, then no more output will get into a.txt. The same is the case if I use the append-redirect >>.
Is there a way to make my redirection recreate the file when it is deleted during the runtime of prog?
Redirection is part of the OS I think and not of prog. So maybe there are some tools or settings.
Thanks!
At the OS level, a file is made up of many components:
the content, stored somewhere on the storage device;
an i-node that keeps all file information except the name;
the name, listed in a directory (also stored on the storage device);
when the file is open, each application that opens it handle memory buffers that keep some of the file content.
All these are linked and the OS keeps their booking.
If you delete the file while it is open by another application (the redirect operator > keeps it open until ./prog completes), only the name is removed from the directory. The other pieces of the puzzle are still there and they keep working until the last application that keeps the file open closes it. This is when the file content is discarded on the storage medium.
If you delete the file, while ./prog keeps running and producing output the file grows and uses space on the storage medium but it cannot be open again because there is no way to access it. Only the programs that have it already open when it was deleted can still access the file until they close it.
Even if you re-create the file, it is a different file that happens to have the same name as the deleted one. ./prog is not affected, its output goes to the old, deleted file.
When its output is redirected, apart from restarting ./prog, there is no way to persuade it to store its output in a different file when a.txt is deleted.
There are several ways to make this happen if ./prog writes itself into a.txt (they all require changing the code of ./prog).
You can use gdb to redirect the output of program to file when original file is deleted.
Refer to this post.
For later references, I give the only excerpt from the post:
Find the files that are opened by the process using /proc/<pid>/fd.
Attach the PID of program to gdb.
Close the file descriptor of the deleted file through gdb session.
Redirect the program output to another file using gdb calls.
Examples
Suppose that PID of program is 19080 and file descriptor of deleted file is 2.
gdb attach 19080
ls -l /proc/19080/fd
gdb> p close(2)
$1 = 0
gdb> p fopen("/tmp/file", "w")
$2 = 20746416
(gdb) p fileno($2)
$3 = 7
gdb> quit
N.B.: If data of the deleted file is required, recover the deleted text file before closing the file handle:
cp -pv /proc/19080/fd/2 recovered_file.txt
My process reads a files and deletes it. This activity happens more than 2000 times.
When I check the file in /proc/PID/fd, I see the file there and I see at the end of each line as (deleted). But I see 1024 records, with 1020 being the (deleted) entries. Later the new file operation from this PID fails.
To overcome this issue, kept process on debug and did
p close (id)
This (id) is taken from ll output on /proc/PID/fd.
Wanted to know the reason for the file not being deleted. fdclose is used first and then the file is deleted, even then file is shown with (deleted)
/proc/$PID/fd directory shows all the open files of the process named by their descriptors. Each file in /proc/$PID/fd represents an open file/socket/pipe etc., If the descriptor belongs to a disk file, then its symbolic link points to the absolute path of the file that is opened.
Here, (deleted) represents that the file that is opened by the process is deleted and no longer exist on disk. So, the issue in your case is that the file that is opened is not getting closed before unlink(delete). You need to close them before deleting it otherwise it leaks file descriptors.
If you are coding in C use fclose(C standard) or close(POSIX) appropriately to close the file before
in Linux 2.6.27:
From "lsof" output I see a process holding open fd with a (deleted) file. The strange thing is that I can still see the file in the file system using "ls". Why is that?
thanks.
The file is not deleted as long as some process has the file open. When a file is closed, the kernel first checks the count of the number of process that have the file open. If this count has reached 0, the kernel then checks the link count; if it is 0, the file's contents are deleted.
To quote from man unlink:
If the name was the last link to a file but any processes still have
the file open the file will remain in existence until the last file
descriptor referring to it is closed.
When a file is deleted it would not been seen on the file system. However, it is quite possible another file with the same file name is created on the same location.
You can check the node number shown in lsof and ls -i to check whether they are really the same file.
I using centos5.8
I wonder why syslog stop writing log after certain log edited.
for example, after cut the line of /var/log/messages, syslog doesn't write new log.
Just old logs are remained.
But If I delete the messages file and reboot the system, syslog works fine.
Is there any ways, syslogd write new logs continuosely after edit certain log??
It depends how exactly a file is edited.
Remember that syslogd keeps the file open for continuously writing. If your editor writes a new file with the old name after unlink()ing or rename()ing the old file, that old file remains in operation for syslogd. The new file, however, remains untouched.
It can be informed about the new file to be used with the HUP signal.
Well, that would depend on how syslogd works. I should mention that it's probably not a good idea to edit system log files, by the way :-)
You may have been caught out by one peculiarity of the way UNIX file systems can work.
If process A has a file open and is writing to it, process B can go and just delete the file (either with something like rm or, as seems likely here, in an edit session which deletes the old file and rewrites a new one).
Deletion of that original file does not destroy the data, it simply breaks the link between the file name (the directory entry) and that data.
A process that has the original file open can continue to write to it as long as it wants and, since there's no entry in the file system for it, you can't (easily) see it.
A new file with that file name may be bought into existence but process A will not be writing to it, it will be writing to the original. There will be no connection between the old and the new file.
You'll sometimes see this effect when you're running low on disk space and decide to delete a lot of files to clear it up somewhat. If processes have those files open, deleting them will not help you at all since the space is still being used (and quite possibly the usage is increasing).
You have to both delete the files and somehow convince those processes holding them open to relinquish them.
If you edited /var/log/syslog you need to restart the syslog service afterwards, because syslogd needs to open the file handle again.