Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
My understanding is that 'files' are effectively just pointers to the memory location corresponding to the files content. If you 'rm' a file, you certainly must be deleting that pointer. If rm actually 'erases' the data, I would guess each bit is written over (set to 0 or something). But I know that there are special procedures/programs (i.e. srm) to make sure that data isn't 'recoverable' --- which suggests that nothing is actually being overwritten...
So, is deleting the pointer to a memory address the only thing rm does? Is the data still sitting there in a contiguous block like it was before?
My understanding is that 'files' are effectively just pointers to the memory location corresponding to the files content.
Be careful with your terminology. The files (and pointers) are on disk, not in memory (RAM).
If you 'rm' a file, you certainly must be deleting that pointer.
Yes. What happens is heavily file-system dependent. Some have a bitmap of which block are free/busy. So it would have to flip the bit for each block freed. Other filesystems use more sophisticated methods of tracking free space.
which suggests that nothing is actually being overwritten...
Correct. You can find various "undelete" utilities. But depending on the filesystem, it can get rather complex. But stuff you saved years ago could still be sitting around -- or it could be overwritten. It all depends on minute details. For example, see e2fsprogs.
So, is deleting the pointer to a memory address the only thing rm does?
Well, it also has to remove the "directory entry" that gives metadata about the file. (Sometimes it just wipes out the first byte of the filename).
Is the data still sitting there in a contiguous block like it was before?
Yes, the data is still there. But don't assume it is a contiguous block. Files can be freagmented all over the disk, with lots of pointers that tell it how to re-assemble. And if you are using RAID, things get real complex.
Yes. rm simply deletes the pointer. If you have multiple pointers to the file (hard links), then deleting one of those pointers with rm leaves the others completely untouched and the data still available.
Deleting all of those links still does not touch the data, however the OS is now free to reuse the blocks which previously were reserved for storing that data.
It's worth noting that any process which opens a file creates a file handle for it. This adds to the overall count of references to the file. If you delete all of the pointers from your filesystem, but the operating system still has a process running with an open file handle for your file, then the count of pointers will not be zero and the file will not really be deleted. Only when that final pointer is closed will the filesystem register the disk space as having been released, and only at that point will the OS be free to overwrite the blocks previously reserved for storing that data.
You may or may not be able to recover that data at any point in the future depending on whether any reuse of the blocks in question has occurred.
Incidentally, you have no guarantee that your data is sitting there in a contiguous block in the first place.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I bought a Highpoint a HBA card with 4 Samsung 960PRO in it.As the official site said this card can perform 7500MB/s in writing and 13000MB/s in reading.
When I test this card with fio in my ubuntu 16.04 system,I got a writing speed of about 7000MB/s,here is my test arguments:
sudo fio -filename=/home/xid/raid0_dir0/fio.test -direct=1 -rw=write -ioengine=sync -bs=2k -iodepth=32 -size=100G -numjobs=1 -runtime=100 -time_base=1 -group_reporting -name=test-seq-write
I have made a raid0 in the card and made a xfs filesystem.I want to know how to achieve disk writing speed as high as fio performed if I want to use functions such as "open(),read(),write()" or functions such as "fopen(),fread(),fwrite()" in my console applications.
I'll just note that the fio job you specified seems a little flawed:
-direct=1 -rw=write -ioengine=sync -bs=2k -iodepth=32
(for the sake of simplicity let's assume the dashes are actually double)
The above is asking trying to ask a synchronous ioengine to use an iodepth of greater than one. This usually doesn't make sense and the iodepth section of the fio documentation warns about this:
iodepth=int
Number of I/O units to keep in flight against the file. Note that
increasing iodepth beyond 1 will not affect synchronous ioengines
(except for small degrees when verify_async is in use). Even async
engines may impose OS restrictions causing the desired depth not to be
achieved. [emphasis added]
You didn't post the fio output so we can't tell if you ever achieved an iodepth of greater than one. 7.5GByte/s seems high for such a job and I can't help thinking your filesystem quietly went and did buffering behind your back but who knows? I can't say more because the output of you fio run is unavailable I'm afraid.
Also note the data fio was writing might not have been random enough to defeat compression thus helping to achieve an artificially high I/O speed...
Anyway to your main question:
how to achieve disk write speed as high as fio does?
Your example shows you are telling fio to use an ioengine that does regular write calls. With this in mind, theoretically you should be able to achieve a similar speed by:
Preallocating your file and only writing into the allocated portions of it (so you are not doing extending writes)
Fulfilling all the requirements of using O_DIRECT (there are strict memory alignment and size constraints that MUST be fulfilled)
Making sure your write operations are working on buffers writing in chunks of exactly 2048 bytes (or greater so long as they are powers of two)
Submitting your writes as soon as possible :-)
You may find not using O_DIRECT (and thus allowing buffered I/O to do coalescing) is better if for some reason you are unable to submit "large" well aligned buffers every time.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I have some large files and i am ok with them being read at disk I/O capacity. I wish to have file system cache free for other files.
Is it possible to turn of file system caching for specific files in linux ?
Your question hints that you might not be the author of the program you wish to control... If that's the case the answer is "not easily". If you are looking for something where you just mark (e.g. via extended attributes) a particular set of files "nocache" the answer is no. At best you are limited to having a LD_PRELOAD wrapper around the program and the wrapper would have to be written carefully to avoid impacting all files the program would try to open etc.
If you ARE the author of the program you should take a look at using fadvise (or the equivalent madvise if you're using mmap) because after you have finished reading the data you can hint to the kernel that it should discard the pieces it cached by using the FADV_DONTNEED parameter (why not use FADV_NOREUSE? Because with Linux kernels available at the time of writing it's a no-op).
Another technique if you're the author would be to open the file with the O_DIRECT flag set but I do not recommend this unless you really know what you're doing. O_DIRECT comes with a large set of usage constraints and conditions on its use (which people often don't notice or understand the impact of until it's too late):
You MUST do I/O in multiples of the disk's block size (no smaller than 512 bytes but not unusual for it to be 4Kbytes and it can be some other larger multiple) and you must only use offsets that are similarly well aligned.
The buffers of your program will have to conform to an alignment rule.
Filesystems can choose not to support O_DIRECT so your program has to handle that.
Filesystems may simply choose to put your I/O through the page cache anyway (O_DIRECT is a "best effort" hint).
NB: Not allowing caching to be used at all (i.e. not even on the initial read) may lead to the file being read in at speeds far below what the disk can achieve.
I think you can do this by the open system call with O_DIRECT for the file you don't want to cache file on the page cache of kernel.
The meaning of O_DIRECT flag from the open manual is the following:
O_DIRECT (Since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this file. In general this will degrade perfor‐
mance, but it is useful in special situations, such as when applications do their own caching. File
I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to
transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and neces‐
sary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to
O_DIRECT. See NOTES below for further discussion.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
i've been studying the linux operating system for a while now, i understand what file systems are but i'm curious as to how they're made. Is it possible for a programmer to create their own custom made file system in linux, is it possible to combine multiple file systems together and how much control do we have over a file system? Thanks.
Also does anyone know any online sources or books that talk about linux file systems
Certainly a programmer can create a file system, everyone can, you just have to use the command to do that. In addition to that a programmer theoretically can implement logic that creates what you probably refer to as "custom made filesystem", just as a programmer can change, remove or add anything he wants from any part of the system he uses. It is questionable though if many programmers actually are able to create a working and usable file system from scratch, since that is quite a complex thing to do.
Combining multiple filesystems is certainly possible, but maybe you should define in more detail what you actually ask by that. You certainly can use multiple filesystems inside a single system by simply mounting them. You can mount one filesystem inside another. You can even use a loopback device to store a whole filesystem inside a file contained in another filesystem. What you can not do is somehow take two separate file systems, hold a magic wand over them and declare them as one from now on. Well, actually you can do that but it won't work as expected ;-)
About the last question, how much control we have... well, difficult to answer without you giving any metric to do so... We certainly can configure a filesystem, we can use it and its content. We can even destroy or damage it, mount it, check it, examine it, monitor it, repair it, create it, ... I personally would indeed call that some amount of "control" we have over filesystems.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 9 years ago.
Improve this question
I'm writing a shell script in bash where I'm making some links between files, but I'm not sure what kind of link to use (physical or symbolic). Doing some research, I've noticed that it's more common to use symbolic links instead of physical ones. My question is why to use symbolic links when they require an indirection (it creates an extra I-node to safe the information about the real I-node of the file) instead of using hard links that point directly to the file?
In other words:
Why
ln -s ...
instead of
ln -P ...
The main reason for symlinks is that a 'soft' symlink can cross filesystem boundaries. The file representing the symlink will contain a string that is the actual path of the file being pointed at. As long as the end-user representations of that path remain the same, the symlink will work. If you move the file at the end of the symlink, the symlink will now be stale (aka "dangling"), because the resource it pointed at no longer exists.
A hard (aka physical) symlink works at the inode layer. Since inodes are only unique within a single file system, you cannot hardlink ACROSS file systems. You could quite easily run into a duplicate inode situation if this were allowed. The benefit is that no matter where you move the target of a hardlink, the links pointing at the resource will "follow", because they're pointing at the inode itself, and don't care what the actual path/resource name is.
Off the top of my head:
symbol links work across filesystems. If you don't want to keep track of what filesystem the source file and destination link are on, or if you sometimes move files across filesystems, it is less of a hassle to use symbolic links.
$#&#$& emacs backup files. When you edit, say file.txt and make a change to it in emacs, emacs renames the original file to file.txt~ and saves your changes under the original file name. If there was a hard link to file.txt, it is now linked to file.txt~, which is probably not what you want. A symbolic link to file.txt will still point to the updated inode.
A hardlink can only work on the same filesystem, renames the inode. A file can only be deleted when the last link to its inode is gone. Hardlinks usually are for files instead of directories.
A symlink is an actual file containing a path to another file. Symlinks can work across file systems as well. They can point to different file types as well. A symlink can also point to either files or directories.
Hard links don't make sense across filesystems, since they're basically pointers to inodes on the local FS. Symlinks don't care; since they're just names, they can live anywhere.
If you look at the directory listing and see a symlink, you know it's to a particular other location. Hard links, on the other hand, offer no such clue. You might not know you were playing around in some important file til you stat both its names.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have read the general answers to these related questions,
understanding-the-linux-kernel-source
How is the system call in Linux implemented?
How does a syscall actually happen on linux?
but still left with questions of my own. For example, on int 0x80 the kernel services the system call, but what does it mean to "service" a call? e.g. if a service call is made for getuid
#define __NR_getuid (__NR_SYSCALL_BASE+ 24)
then once int 0x80 occurs, the kernel services the call. So what exactly must the kernel do to implement getuid? Somewhere there must be some code which runs after int 0x80. Assuming having downloaded the Linux kernel source, where (e.g. what path) might you find the source code implementation for __NR_getuid?
The handler for getuid(2) is in kernel/timer.c, but you're going to find a single-line function there which won't enlighten you at all.
I found that file by saying make tags at the top level of the kernel source directory, then saying vi -t sys_getuid. sys_*() is how the syscall entry points are named in the kernel. Having done that, you can see why the find command given by 0xDen should work, and in fact does here on my system. But, using tags is faster and easier.
This is why I keep recommending that those who want to understand how the kernel works read a book on it. These books don't literally tell you how each and every syscall is implemented, since that would require that it basically walk you through all the code, line-by-line. There are millions of SLOC in the kernel. If you printed it out, it would literally require a bookcase to hold it all, even if you printed it in small text, double-sided.
Even if you ignore all the non-core parts, such as drivers, weird filesystems, less popular CPU types, etc., so that you were able to cut this down to 1% its total size, you'd still be left with a hundred thousand SLOC to plow through. That fills a big book all by itself, without leaving room for much commentary.
So, what a book on the kernel is going to do is give you the overall understanding of what goes on in there so that you will be able to find out where things live and how they are activated yourself. You will learn about the data structures that tie it all together, so you can follow the call chains, etc.
find -name "*.c"| xargs grep -n -E "SYSCALL_DEFINE" | grep getuid