I wanted to check the read/write performance of my disk. I am executing the below command to write into a file
time dd if=/dev/zero of=/home/test.txt bs=2k count=32k;
which gives about 400MB/s
For checking the read performance i have executed below commands.with and without 'of' parameter. There is a huge difference between those results
time dd if=/home/test.txt of=/dev/zero bs=2k (gives about 2.8GB/s)
time dd if=/home/test.txt bs=2k (9MB/s)
I read that "of=/dev/zero" is used to read data from some temp file while creating the file.
But why is it required while checking for read performance and why there is a huge difference in speed with and without "of=/dev/zero"
/dev/zero is a special file. It's contents stem from a device driver. All write operations on /dev/zero are guaranteed to succeed. A bit more about that here and here
Without specifying of dd prints to stdout. Thus the data which the terminal receives has to be formatted and printed. The terminal you're using is very likely to bottleneck the performance of your drive.
Also if likely stands for input file, likewise of means output file.
Edit:
Writing to /dev/zero can have unexpected results. I wouldn't say this is an accurate way of measuring read performance.
You use HDD's and filesystem's caches on read operations. Try oflag=direct flag.
Related
I'm trying a more efficient way of reading file contents in Linux without using the "cat" command, especially for larger file contents, as in such cases cat just shoots up the memory and CPU on the server.
One thing that comes to my mind is using a grep -v "character-set-which-is-unlikely-in-the-file" filename
But using different character sets every time and hoping it would not appear in the file, wouldn't be efficient.
Any other thoughts ?
If you just want to read through the file, so it gets cached, the simplest way is perhaps this:
cat filename > /dev/null
Note that you don't need to show the data on screen to read it from disk. That command reads the file, and ignores the content by dumping it in /dev/null, but it still reads all the data.
If the CPU load goes up, that is probably a good thing, meaning that the computer is working hard, and will be finished sooner rather than later. If it crashes, though, there is something else wrong.
If you have some specific reason not to use the "cat" command, you can try "dd" instead, but it is more complicated to write and will not be faster:
dd if=filename of=/dev/null bs=1M
Addendum:
This inspired me to run some tests. On my particular computer both "cat" and "dd" took 24.27-24.31 seconds to read a large file on a mechanical disk when it wasn't already cached, and 0.39-0.40 seconds when it was cached. (Three tests of each case, with very little variability.)
Both these programs contain code to write the data, even if it is dumped to /dev/null, so one could expect that a program specifically written to just read would be slightly faster, but I got the same times when I tried that.
I want to know under what circumstances a direct I/O transfer will fail?
I have following three sub-queries for that. As per "Understanding Linux kernel" book..
Linux offers a simple way to bypass the page cache: direct I/O transfers. In each I/O direct transfer, the kernel programs the disk controller to transfer the data directly from/to pages belonging to the User Mode address space of a self-caching application.
-- So to explain failure one needs to check whether application has self caching feature or not? Not sure how that can be done.
2.Furthermore the book says "When a self-caching application wishes to directly access a file, it opens the file specifying the O_DIRECT flag . While servicing the open( ) system call, the dentry_open( ) function checks whether the direct_IO method is implemented for the address_space object of the file being opened, and returns an error code in the opposite case".
-- Apart from this any other reason that can explain direct I/O failure ?
3.Will this command "dd if=/dev/zero of=myfile bs=1M count=1 oflag=direct" ever fail in linux (assuming ample disk space available) ?
The underlying filesystem and block device must support O_DIRECT flag. This command will fail because tmpfs doesn't support O_DIRECT.
dd if=/dev/zero of=/dev/shm/test bs=1M count=1 oflag=direct
The write size must be the multiply of the block size of underlying driver. This command will fail because 123 is not multiply of 512:
dd if=/dev/zero of=myfile bs=123 count=1 oflag=direct
There are many reasons why direct I/O can go on to fail.
So to explain failure one needs to check whether application has self caching feature or not?
You can't do this externally - you have the either deduce this from the source code or watching how the program used resources as it ran (binary disassembly I guess). This is more a property of how the program does its work rather than a "turn this feature on in a call". It would be a dangerous assumption to think all programs that use O_DIRECT have self caching (probabilistically I'd say it's more likely but you don't know for sure).
There are strict requirements for using O_DIRECT and they are mentioned in the man page of open (see the O_DIRECT section of NOTES).
With modern kernels the area being operated on must be aligned to, and its size must be a multiple of, the disk's block size. Failure to do this correctly may even result in silent fallback to buffered I/O.
Yes, for example trying to use it on a filesystem (such as tmpfs) doesn't support O_DIRECT. I suppose it could also fail if the path down to the disk returns failures for some reason (e.g. disk is dying and is returning the error much sooner in comparison to what happens with writeback).
I need to be able to create a large number of files, randomly-sized 1k-64k averaging 32k, very quickly. One method I've found uses the unix command dd to copy bytes from /dev/urandom into a new file. However, I'm trying to save the time by just creating the file link to unallocated space, without writing random bytes to disk.
Is this possible in a Unix-like environment and, if so, how would I go about it?
If not, could this be done with a small C program?
From the commandline, mkfile or dd or any number of other (not-necessarily POSIX-standard) utilities. I would note that copying from /dev/zero instead of /dev/urandom can be significantly quicker, especially on older systems with slower CPUs.
If you want to do something in C or C++ code, there's truncate() and ftruncate(). Probably most Perl/Python/whatever bindings expose the system/libc call in some way as well.
So you want to mkfile {size}[b|k|m|g] filename in cwd?
Suppose I have a console program that outputs trace debug lines on stdout, that I want to run on a server.
And then I do:
./serverapp > someoutputfile
If I need to see how the program's doing, I would just log into the server and do:
tail -f someoutputfile
However, understandably over time, someoutputfile would become pretty big.
Is there a way to make it so that someoutputfile is limited to a certain size, and only the most recent parts of it?
I mean, the hard way would be to make a custom script/program that cycles the output between different files, but that seems like overkill.
You can truncate the log file. One way to do this is to type:
>someoutputfile
at the shell command-line. It's a redirect with no output and it will erase all the contents of the file.
The tricky bit here is that any program writing to that file will continue to write into the file at its last output position. So the file will immediately gain a "hole" from 0 to X bytes, where X is the output position.
In most Linux file systems these holes result in sparse files, which don't actually use the space in the hole. So the file may contain many gigabytes of 0's at the beginning but only use 500 KB on disk.
Another way to do fast logging is to memory map a file on disk of fixed size: 16 MB for example. Then the logging writes into a memory pointer which wraps around when it reaches the size limit. It then continues to write at the front of the file. It's a good idea to have some kind of write position marker. I use <====>, for example. I find this method to be ridiculously fast and great for debug logging.
I haven't used it, but it gets good reviews here on SO, try logrotate
A more general discussion of managing output files may show you that a custom script/solution is not out of the question ;-) : Problem with Bash output redirection
I hope this helps.
I have a program that reads and writes very large text files. However, because of the format of these files (they are ASCII representations of what should have been binary data), these files are actually very easily compressed. For example, some of these files are over 10GB in size, but gzip achieves 95% compression.
I can't modify the program but disk space is precious, so I need to set up a way that it can read and write these files while they're being transparently compressed and decompressed.
The program can only read and write files, so as far as I understand, I need to set up a named pipe for both input and output. Some people are suggesting a compressed filesystem instead, which seems like it would work, too. How do I make either work?
Technical information: I'm on a modern Linux. The program reads a separate input and output file. It reads through the input file in order, though twice. It writes the output file in order.
Check out zlibc: http://zlibc.linux.lu/.
Also, if FUSE is an option (i.e. the kernel is not too old), consider: compFUSEd http://www.biggerbytes.be/
named pipes won't give you full duplex operations, so it will be a little bit more complicated if you need to provide just one filename.
Do you know if your applications needs to seek through the file ?
Does your application work with stdin, stdout ?
Maybe a solution is to create a mini compressed file system that contains only a directory with your files
Since you have separate input and output file you can do the following :
mkfifo readfifo
mkfifo writefifo
zcat your inputfile > readfifo &
gzip writefifo > youroutputfile &
launch your program !
Now, you probably will get in trouble with the read twice in order of the input, because as soon as zcat is finished reading the input file, yout program will get a SIGPIPE signal
The proper solution is probably to use a compressed file system like CompFUSE, because then you don't have to worry about unsupported operations like seek.
btrfs:
https://btrfs.wiki.kernel.org/index.php/Main_Page
provides support for pretty fast "automatic transparent compression/decompression" these days, and is present (though marked experimental) in newer kernels.
FUSE options:
http://apps.sourceforge.net/mediawiki/fuse/index.php?title=CompressedFileSystems
Which language are you using?
If you are using Java, take a look at GZipInputStream and GZipOutputStream classes in the API doc.
If you are using C/C++, zlibc is probably the best way to go about it.