What is the significance of 'flags' option of fs.createReadStream/createWritStream while ReadStream and WriteStream of fs module are self-explanatory whether they are readable or writable?
Flag option allows you to set different behaviour related to writing or overwriting files.
For example, when creating a WriteStream, if you pass the flag w it will overwrite the file if it exists (this is the default value), whereas if you use the flag r+, it will just modify the file if it exists as it actually open the files for reading and writing or will have an error if it doesnt exist.
Here is a lost of flags and there explanation:
'r' - Open file for reading. An exception occurs if the file does not exist.
'r+' - Open file for reading and writing. An exception occurs if the file does not exist.
'w' - Open file for writing. The file is created (if it does not exist) or truncated (if it exists).
'w+' - Open file for reading and writing. The file is created (if it does not exist) or truncated (if it exists).
'a' - Open file for appending. The file is created if it does not exist.
'a+' - Open file for reading and appending. The file is created if it does not exist.
Related
I'm considering making my application create a file in /tmp. All I'm doing is making a temporary file that I'll copy to a new location. On a good day this looks like:
Write the temp file
Move the temp file to a new location
However on my system (RHEL 7) there is a file at /usr/lib/tmpfiles.d/tmp.conf which indicates that /tmp gets cleaned up every 10 days. From what I can tell this is the default installation. So, what I'm concerned about is that I have the following situation:
Write the temp file
/tmp gets cleaned up
Move the temp file to a new location (explodes)
Is my concern founded? If so, how is this problem solved in sophisticated programs? If there are no sophisticated tricks, then it's a bit puzzling to me as I don't have a concrete picture of what the utility of /tmp is if it can be blown away completely at any moment.
this should not be a problem if you keep a file descriptor open during your operation. As long as a file descriptor is open, the FS keeps the file on disk but it just don't appear when using ls. So If you create another name for this file, it will "resurect" in some way. Keeping an open fd on a file that is deleted is a common way to create temporary files on linux
see the man 3 unlink:
The unlink() function shall remove a link to a file. [..] unlink() shall remove the link named by the pathname pointed to by
path and shall decrement the link count of the file
referenced by the link.
When the file's link count becomes 0 and no process has the file open, the space occupied by the file shall be freed and the file
shall no longer be accessible. If one or more
processes have the file open when the last link is removed, the link shall be removed before unlink() returns, but the removal of
the file contents shall be postponed until all
references to the file are closed.
External Updates that happened to that file content during the time between open() and first read() are not returned in the read() content.
How can I get the latest file content from the read()?
I've tried flush() and seek(0) but didn't help.
https://repl.it/repls/RealGreedyTransfer#main.py
import time
def myfoo(handle):
print("myfoo started", flush=True)
time.sleep(50)
# External updates that happen during that time don't show up in read()
# foo.flush()
# foo.seek(0)
# can't close and re-open file handle
print(handle.read()) # <-- Not reading updates done after file open
# Upstream code base passing a file handle under an exclusive fcntl.lockf() lock
handle = open('temp.txt', 'r+')
myfoo(handle)
The issue is with the way files are written. Many text editors don’t just write to the file, they use a different method: they write to a temporary file, and then rename it to the original filename. Since renames are atomic in POSIX, in the event of a system crash during saving, the old version of the file will be available, and the new version might or might not be available in the temporary file.
For most purposes, this works as desired. The only exception is in this case, where you’re holding onto a file handle. Renames/moves/deletions do not affect the file handles, they are still open with the file they were opened with, even if that file is no longer accessible from the filesystem. You can experiment with this by opening a file, then removing it with rm, and then reading from the file — it will still show you the file contents from before you deleted it. You can also access the file in Linux inside /proc/XX/fd.
Your file handle won’t see changes, unless they are actually written (and flushed) to the same file (without the rename dance). If you’re working with something that writes by renaming, you would need to reopen the file to see the new contents.
My process reads a files and deletes it. This activity happens more than 2000 times.
When I check the file in /proc/PID/fd, I see the file there and I see at the end of each line as (deleted). But I see 1024 records, with 1020 being the (deleted) entries. Later the new file operation from this PID fails.
To overcome this issue, kept process on debug and did
p close (id)
This (id) is taken from ll output on /proc/PID/fd.
Wanted to know the reason for the file not being deleted. fdclose is used first and then the file is deleted, even then file is shown with (deleted)
/proc/$PID/fd directory shows all the open files of the process named by their descriptors. Each file in /proc/$PID/fd represents an open file/socket/pipe etc., If the descriptor belongs to a disk file, then its symbolic link points to the absolute path of the file that is opened.
Here, (deleted) represents that the file that is opened by the process is deleted and no longer exist on disk. So, the issue in your case is that the file that is opened is not getting closed before unlink(delete). You need to close them before deleting it otherwise it leaks file descriptors.
If you are coding in C use fclose(C standard) or close(POSIX) appropriately to close the file before
I am looking at fs.open() here:
https://nodejs.org/api/fs.html#fs_fs_open_path_flags_mode_callback
I want to open a file for both reading and executing.
it gives these flags:
'r' - Open file for reading. An exception occurs if the file does not
exist.
'r+' - Open file for reading and writing. An exception occurs if the
file does not exist.
'rs+' - Open file for reading and writing in synchronous mode.
Instructs the operating system to bypass the local file system cache.
This is primarily useful for opening files on NFS mounts as it allows
skipping the potentially stale local cache. It has a very real impact
on I/O performance so using this flag is not recommended unless it is
needed.
Note that this doesn't turn fs.open() into a synchronous blocking
call. If synchronous operation is desired fs.openSync() should be
used.
'w' - Open file for writing. The file is created (if it does not
exist) or truncated (if it exists).
'wx' - Like 'w' but fails if path exists.
'w+' - Open file for reading and writing. The file is created (if it
does not exist) or truncated (if it exists).
'wx+' - Like 'w+' but fails if path exists.
'a' - Open file for appending. The file is created if it does not
exist.
'ax' - Like 'a' but fails if path exists.
'a+' - Open file for reading and appending. The file is created if it
does not exist.
'ax+' - Like 'a+' but fails if path exists.
I am guessing opening for reading and executing would be rx but that's just a guess, anybody know for sure? There must be a reason it's not listed?
You won't be able to execute from your file stream anyway, so there's no such thing as opening for reading and executing. Executing is independent from this stream.
To execute, take a look at child_process and its various methods.
I thought about a concurrency issue (in Solaris), what happen if while reading someone tries to delete the same file. I have a query regarding file existence in the Solaris/Linux. suppose I have a file test.txt, I have open it in vi editor, and then I have open a duplicate session and remove that file, but even after deleting that file I am able to read that file. so here are my questions:
Do I need to thinks about any locking mechanism while reading, so no one able to delete same file while reading.
What is the reason of showing different behavior from windows(like in windows if file is open in in some editor than we can not delete that file)
After removing that file, how I am still able to read that file, if I haven't closed file from vi editor.
I am asking files in general,but yes platform specific i.e. unix. what will happen if I am using a java program (buffer reader) for read file and file is deleted while reading, does buffer reader still able to read the file for next chunk or not?
You have basically 2 or 3 unrelated questions there. Text editors like to read the whole file into memory at the start of the editing session. Imagine every character you type being saved to disk immediately, with all characters after it in the file being rewritten one place further along to make room. That would be awful. Much better that the thing you're actually editing is a memory representation of the file (array of pointers to lines, probably with some metadata attached) which only gets converted back into a linear stream when you explicitly save.
Any relatively recent version of vim will notify you if the file you are editing is deleted from its original location with the message
E211: File "filename" no longer available
This warning is not just for unix. gvim on Windows will give it to you if you delete the file being edited. It serves as a reminder that you need to save the version you're working on before you exit, if you don't want the file to be gone.
(Note: the warning doesn't appear instantly - vim only checks for the original file's existence when you bring it back into the foreground after having switched away from it.)
So that's question 1, the behavior of text editors - there's no reason for them to keep the file open for the whole session because they aren't actually using it except at startup and during a save operation.
Question 2, why do some Windows editors keep the file open and locked - I don't know, Windows people are nuts.
Question 3, the one that's actually about unix, why do open files stay accessible after they're deleted - this is the most interesting one. The answer, guaranteed to shock you when presented directly:
There is no command, function, syscall, or any other method which actually requests deletion of a file.
Underlying rm and any other command that may appear to delete a file there is the system call unlink. And it's called unlink, not remove or deletefile or anything similar, because it doesn't remove a file. It removes a link (a.k.a. directory entry) which is an association between a file and a name in a directory. (Note: ANSI C added remove as a more generic function to appease non-unix people who had no intention of implementing unix filesystem semantics, but on unix, remove is just a rmdir if the target is a directory, and unlink for everything else.)
A file can have multiple links (see the ln command for how they are created), which means that the same file is known by multiple names. If you rm one of them, the others stick around and the file is not deleted. What happens when you remove the last link? Well, now you have a file with no name. But names are only one kind of reference to a file. There are at least 2 others: file descriptors and mmap regions. When the last reference to a file goes away, that's when the file is deleted.
Since references come in several forms, there are many kinds of events that can cause a file to be deleted. Here are some examples:
unlink (rm, etc.)
close file descriptor
dup2 (can implicitly closes a file descriptor before replacing it with a copy of a different file descriptor)
exec (can cause file descriptors to be closed via close-on-exec flag)
munmap (unmap memory region)
mmap (if you create a new memory map at an address that's already mapped, the old mapping is unmapped)
process death (which closes all file descriptors and unmaps all memory mappings of the process)
normal exit
fatal signal generated by the kernel (^C, segfault)
fatal signal sent from another process (kill)
I won't call that a complete list. And I don't encourage anyone to try to build a complete list. Just know that rm is "remove name", not "remove file", and files go away as soon as they're not in use.
If you want to destroy the contents of a file immediately, truncate it. All processes already using it will find that its size has suddenly become 0. (This is destruction as far as the normal file access methods are concerned. To destroy it more thoroughly so that even someone with raw disk access can't read what used to be there, you need to overwrite it. There's a tool called shred for that.)
I think your question has nothing to do with the difference between Windows/Linux. It's about how VI works.
when using VI to edit a file. VI will create a .swp file. And the .swp file is what you are actually editing. At the same time, if other users delete the original file will not effect your editing.
And when you type :w in VI, VI will use .swp file to overwrite the original file.