Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I have some large files and i am ok with them being read at disk I/O capacity. I wish to have file system cache free for other files.
Is it possible to turn of file system caching for specific files in linux ?
Your question hints that you might not be the author of the program you wish to control... If that's the case the answer is "not easily". If you are looking for something where you just mark (e.g. via extended attributes) a particular set of files "nocache" the answer is no. At best you are limited to having a LD_PRELOAD wrapper around the program and the wrapper would have to be written carefully to avoid impacting all files the program would try to open etc.
If you ARE the author of the program you should take a look at using fadvise (or the equivalent madvise if you're using mmap) because after you have finished reading the data you can hint to the kernel that it should discard the pieces it cached by using the FADV_DONTNEED parameter (why not use FADV_NOREUSE? Because with Linux kernels available at the time of writing it's a no-op).
Another technique if you're the author would be to open the file with the O_DIRECT flag set but I do not recommend this unless you really know what you're doing. O_DIRECT comes with a large set of usage constraints and conditions on its use (which people often don't notice or understand the impact of until it's too late):
You MUST do I/O in multiples of the disk's block size (no smaller than 512 bytes but not unusual for it to be 4Kbytes and it can be some other larger multiple) and you must only use offsets that are similarly well aligned.
The buffers of your program will have to conform to an alignment rule.
Filesystems can choose not to support O_DIRECT so your program has to handle that.
Filesystems may simply choose to put your I/O through the page cache anyway (O_DIRECT is a "best effort" hint).
NB: Not allowing caching to be used at all (i.e. not even on the initial read) may lead to the file being read in at speeds far below what the disk can achieve.
I think you can do this by the open system call with O_DIRECT for the file you don't want to cache file on the page cache of kernel.
The meaning of O_DIRECT flag from the open manual is the following:
O_DIRECT (Since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this file. In general this will degrade perfor‐
mance, but it is useful in special situations, such as when applications do their own caching. File
I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to
transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and neces‐
sary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to
O_DIRECT. See NOTES below for further discussion.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 months ago.
The community reviewed whether to reopen this question 4 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I have a large directory of gzip-compressed files.
I want to use a tool to index these files. The tool works by walking over a folder and reading all text files. Unfortunately, the tool doesn't support reading gzip-compressed text files, and it's not practical for me to temporarily decompress all the files so the tool can access them. It would consume a massive amount of disk space, and even though disk space is cheap, it's still impractical in my use-case.
I also don't have access to the tool to modify it to add support for gzip-compression.
So I was thinking of a way to insert a middle-man, between the tool and my files, that would transparently perform the decompression on the fly.
To that end, is there any way for me, under Linux, to create a sort of symbolic filesystem that mirrors my folder contents, and create a "fake" file for each original file so that, when read, it silently calls a script that accesses the original file, pipes it through gunzip, and returns the output? The effect would be, from the tool's perspective, it's reading un-compressed files without me having to decompress them all at once.
Are there any other solutions that I'm overlooking?
There are a few approaches that occur to me, each with varying amounts of difficulty. The options are ordered by how easy they would be IMHO.
Option 1 -- A compressed-at-rest filesystem
Several modern file systems support compression at rest -- i.e., the data is stored compressed, and decompressed for you on demand. You could set up a partition of your disk with one of these filesystems (I would recommend zfs), and then copy all of your data into the partition.
Once you've done that, you'd have the disk usage of compressed data, but would be able to interact with the filesystem as if it were uncompressed.
Option 2 -- FUSE Wrapper
If you're willing to do some coding for this, using FUSE would be an attractive option. FUSE is a library that effectively lets you describe a file system, and implement reading/writing as just callbacks to user code.
If you weren't worried about performance, it would be relatively straightforward to write some python script that mirrors a directory tree and wraps all read calls with gunzip.
Between option 1 and 2, I would lean towards 1. It will be more performant than any script you could hack out yourself, and would give you added convenience being able to use the data directly.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I bought a Highpoint a HBA card with 4 Samsung 960PRO in it.As the official site said this card can perform 7500MB/s in writing and 13000MB/s in reading.
When I test this card with fio in my ubuntu 16.04 system,I got a writing speed of about 7000MB/s,here is my test arguments:
sudo fio -filename=/home/xid/raid0_dir0/fio.test -direct=1 -rw=write -ioengine=sync -bs=2k -iodepth=32 -size=100G -numjobs=1 -runtime=100 -time_base=1 -group_reporting -name=test-seq-write
I have made a raid0 in the card and made a xfs filesystem.I want to know how to achieve disk writing speed as high as fio performed if I want to use functions such as "open(),read(),write()" or functions such as "fopen(),fread(),fwrite()" in my console applications.
I'll just note that the fio job you specified seems a little flawed:
-direct=1 -rw=write -ioengine=sync -bs=2k -iodepth=32
(for the sake of simplicity let's assume the dashes are actually double)
The above is asking trying to ask a synchronous ioengine to use an iodepth of greater than one. This usually doesn't make sense and the iodepth section of the fio documentation warns about this:
iodepth=int
Number of I/O units to keep in flight against the file. Note that
increasing iodepth beyond 1 will not affect synchronous ioengines
(except for small degrees when verify_async is in use). Even async
engines may impose OS restrictions causing the desired depth not to be
achieved. [emphasis added]
You didn't post the fio output so we can't tell if you ever achieved an iodepth of greater than one. 7.5GByte/s seems high for such a job and I can't help thinking your filesystem quietly went and did buffering behind your back but who knows? I can't say more because the output of you fio run is unavailable I'm afraid.
Also note the data fio was writing might not have been random enough to defeat compression thus helping to achieve an artificially high I/O speed...
Anyway to your main question:
how to achieve disk write speed as high as fio does?
Your example shows you are telling fio to use an ioengine that does regular write calls. With this in mind, theoretically you should be able to achieve a similar speed by:
Preallocating your file and only writing into the allocated portions of it (so you are not doing extending writes)
Fulfilling all the requirements of using O_DIRECT (there are strict memory alignment and size constraints that MUST be fulfilled)
Making sure your write operations are working on buffers writing in chunks of exactly 2048 bytes (or greater so long as they are powers of two)
Submitting your writes as soon as possible :-)
You may find not using O_DIRECT (and thus allowing buffered I/O to do coalescing) is better if for some reason you are unable to submit "large" well aligned buffers every time.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 4 years ago.
Improve this question
PREFACE
I tried to put as much effort and work into this question as I reasonably could, so if you could at least read it through, I would highly appreciate it; I, also, have tried researching this question, but I never seemed to find anything useful, in terms of anything that directly answered my question; I do not know if this is right place for this question, even though it is related to programming, it is more related to operating system development and the Linux kernel, and if there is a better place for this question that I am unaware of, please move it there; feel free to do whatever, edit the question if need be, I do not care, I just need an answer to this question, because this is stressing me out.
The following is some background on why I am asking this question; if you are uninterested, and if you just want to see what I am asking, then skip to the 'MY QUESTION' label; I thought that I would put this is here, so that anyone who is reading this question would know why I am asking this question.
BACKGROUND
I have recently begun setting up an operating system development project; and after I get some things ready, it will be only me working on it, as of right now, and I plan to write the whole thing (yes, I know it will take a whole lot of work, but I can try, right? :p), including the bootstrapping, the CLI, and most of what is necessary to have to either my own kernel or Linux kernel function; GUI and much more; granted, eventually I may end up having a team, but that is for the future.
MY QUESTION
My question, which is actually consists of three parts, and I narrowed them down to specifically those thee things, which are the following:
(1) If I were to build everything else, and use the Linux kernel as-is, and if I were to not tie the other parts of the system into the kernel, but use the kernel for I/O and system calls, would I be violating the GPL in any way, and would I think need to open source the rest of my code?
(2) If I were to only use the kernel for I/O and for system calls, but not have the code that I wrote actually interface with any kernel functions, would that still be considered linking?
(3) If I were to do the above, would that be considered a derived work, when I wrote everything else, but used Linux as the system's kernel?
All these legal issues are making my head spin and extremely confusing to me.
No
No
No
The linux kernel considers the system calls a boundary, and code that communicates with the kernel via system calls is not covered by the licensing of the kernel. So, the user space code you write is not a derivative work of the kernel.
There's also a set of header files provided by the kernel, collectively named the UAPI headers which you can use without having your code become a derivative work
This is covered at https://www.kernel.org/doc/html/v4.17/process/license-rules.html and https://github.com/torvalds/linux/blob/master/LICENSES/exceptions/Linux-syscall-note
If you need legal advice though, contact a lawyer.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
i've been studying the linux operating system for a while now, i understand what file systems are but i'm curious as to how they're made. Is it possible for a programmer to create their own custom made file system in linux, is it possible to combine multiple file systems together and how much control do we have over a file system? Thanks.
Also does anyone know any online sources or books that talk about linux file systems
Certainly a programmer can create a file system, everyone can, you just have to use the command to do that. In addition to that a programmer theoretically can implement logic that creates what you probably refer to as "custom made filesystem", just as a programmer can change, remove or add anything he wants from any part of the system he uses. It is questionable though if many programmers actually are able to create a working and usable file system from scratch, since that is quite a complex thing to do.
Combining multiple filesystems is certainly possible, but maybe you should define in more detail what you actually ask by that. You certainly can use multiple filesystems inside a single system by simply mounting them. You can mount one filesystem inside another. You can even use a loopback device to store a whole filesystem inside a file contained in another filesystem. What you can not do is somehow take two separate file systems, hold a magic wand over them and declare them as one from now on. Well, actually you can do that but it won't work as expected ;-)
About the last question, how much control we have... well, difficult to answer without you giving any metric to do so... We certainly can configure a filesystem, we can use it and its content. We can even destroy or damage it, mount it, check it, examine it, monitor it, repair it, create it, ... I personally would indeed call that some amount of "control" we have over filesystems.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
My understanding is that 'files' are effectively just pointers to the memory location corresponding to the files content. If you 'rm' a file, you certainly must be deleting that pointer. If rm actually 'erases' the data, I would guess each bit is written over (set to 0 or something). But I know that there are special procedures/programs (i.e. srm) to make sure that data isn't 'recoverable' --- which suggests that nothing is actually being overwritten...
So, is deleting the pointer to a memory address the only thing rm does? Is the data still sitting there in a contiguous block like it was before?
My understanding is that 'files' are effectively just pointers to the memory location corresponding to the files content.
Be careful with your terminology. The files (and pointers) are on disk, not in memory (RAM).
If you 'rm' a file, you certainly must be deleting that pointer.
Yes. What happens is heavily file-system dependent. Some have a bitmap of which block are free/busy. So it would have to flip the bit for each block freed. Other filesystems use more sophisticated methods of tracking free space.
which suggests that nothing is actually being overwritten...
Correct. You can find various "undelete" utilities. But depending on the filesystem, it can get rather complex. But stuff you saved years ago could still be sitting around -- or it could be overwritten. It all depends on minute details. For example, see e2fsprogs.
So, is deleting the pointer to a memory address the only thing rm does?
Well, it also has to remove the "directory entry" that gives metadata about the file. (Sometimes it just wipes out the first byte of the filename).
Is the data still sitting there in a contiguous block like it was before?
Yes, the data is still there. But don't assume it is a contiguous block. Files can be freagmented all over the disk, with lots of pointers that tell it how to re-assemble. And if you are using RAID, things get real complex.
Yes. rm simply deletes the pointer. If you have multiple pointers to the file (hard links), then deleting one of those pointers with rm leaves the others completely untouched and the data still available.
Deleting all of those links still does not touch the data, however the OS is now free to reuse the blocks which previously were reserved for storing that data.
It's worth noting that any process which opens a file creates a file handle for it. This adds to the overall count of references to the file. If you delete all of the pointers from your filesystem, but the operating system still has a process running with an open file handle for your file, then the count of pointers will not be zero and the file will not really be deleted. Only when that final pointer is closed will the filesystem register the disk space as having been released, and only at that point will the OS be free to overwrite the blocks previously reserved for storing that data.
You may or may not be able to recover that data at any point in the future depending on whether any reuse of the blocks in question has occurred.
Incidentally, you have no guarantee that your data is sitting there in a contiguous block in the first place.