Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
If there isn't, how feasible would it be to write one? A filesystem which for each directory keeps the size of its contents recursively and which is kept updated not by re-calculating the size on each change on the filesystem, but for example update the dir size when a file is removed or grows.
I am not aware of such a file system. From filesystem's point of view a directory is a file.
You can use:
du -s -h <dir>
to display the total size of all the files in the directory.
From the filesystem point of view, size of directory is size of information about its existence, which needs to be saved on the medium physically. Note, that "size" of directory containing files which have 10GB in total, will be actually the same as "size" of empty directory, because information needed to mark its existence will take same storage space. That's why size of files ( sockets, links and other stuff inside ), isn't actually the same as "directory size". Subdirectories can be mounted from various locations, including remote, and recursively mounted. Somewhat directory size is just a human vision, for real files are not "inside" directories physically - a directory is just a mark of container, exactly the same way as special file ( e.g. device file ) is marked a special file. Recounting and updating total directory size depends more on NUMBER of items in it, than sum of their sizes, and modern filesystem can keep hundreds of thousands of files ( if not more ) "in" one directory, even without subdirs, so counting their sizes could be quite heavy task, in comparison with possible profit from having this information. In short, when you execute e.g. "du" ( disk usage ) command, or when you count directory size in windows, actually doing it someway by the kernel with filesystem driver won't be faster - counting is counting.
There are quota systems, which keep and update information about total size of files owned by particular user or groups, they're, however, limited to monitor partitions separately, as for particular partition quota may be enabled or not. Moreover, quota usage gets updated, as you said, when file grows or is removed, and that's why information may be inaccurate - for this reason quota is rebuild from time to time, e.g. with cron job, by scanning all files in all directories "from the scratch", on the partition on which it is enabled.
Also note, that bottleneck of IO operations speed ( including reading information about the files ) is usually speed of the medium itself, then communication bus, and then CPU, while you're considering every filesystem to be fast as RAM FS. RAM FS is probably most trivial files system, virtually kept in RAM, which makes IO operations go very fast. You can build it at module and try to add functionality you've described, you will learn many interesting things :)
FUSE stands for "file system in user space", FS implemented with fuse are usually quite slow. They make sense when functionality in particular case is more important than speed, e.g. you can create a pseudo-filesystem basing on temperature reading from your newly bought e-thermometer you connected to your computer via USB, however they're not speed daemons, you know :)
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I have two same files in different directories. When i looked for the inode number, it was same for all the files in both directories. Does those files in different directory consume diskspaces individually ?
They are not symlinks. they are hardlinks.
I had a similar query. The Answer is long but hopefully, it will help you:
2 files can have the same inode, but only if they are part of different partitions.
Inodes are only unique on a partition level, not on the whole system.
On each partition, there is a superblock. This superblock tells the system which inodes are used, which are free, etc (I'll spare you the technical details).
Each item on the disk -so files, but also directories, Fifo pipes and special device files- each have their own inode. All the inodes are stored on the disk, right beside the superblock (normally).
For instance in the case of regular files, inodes simply contain some informations like the last access/modification times, the size of the files, the file permissions, the disk blocks it occupies, etc.
For directories, inodes tell the system where the blocks that contain the contents of the directory are stored, as well as the last access/modified dates and permission on the directory.
You can see this if you look at the size of a directory via "ls -ld dir". It is usually an multiple of the size of a disk block (512kB or 1MB usually). The contents of directory blocks are nothing more than a list of name-inode pairs (ie a filename + it's inode number). When you do an "ls", those contents get printed, without the need for the system to actually locate all files in the directory. If you access a file or subdirectory, the system simply looks up the inode number from the directory contents and then retrieves the inode in question, so that you can have fast access to the file/subdirectory in question.
So, you can immediately see that the inode information describes directly what's on a partition to the system. It is the core of the filesystem on your partition.
Unless you are using special software like LVM to make the system believe that multiple physical disks or multiple partitions act as one partition, each partition needs to know it's own contents. Otherwise, you would never be able to share disks for instance via NFS mounting (each computer that shares the disk must know it's contents).
Continuing this line of thought, it is logical that inodes are unique on the (logical) partition level.
To answer your question on hard links. You first need to know what the difference is between a hard and a soft link (or symbolic link). Let's say you want to link A to B, regardless of what A and B really are (files, directories, device files, etc). Thus creating two ways of accessing the same item on your filesystem.
Using a soft link is easy. A and B both have their own inodes and both are part of different directories.
However, A actually contains the full path and name of B. When your system tries to access A, it will see the reference to B (via the full path), locate B by following the path and then access it. Since the full filesystem path is used, soft links work across different partitions. If A is indeed a soft link to B on a different partition and B's partition is unmounted, then the link will continue to exist but will simply point to something unreachable. So you can't access A either. Same goes when B gets deleted. If A (the soft link) would be deleted, then B is still there, unaltered.
Hard links are a different story. As I explained, the contents of a directory are nothing more than pairs of inode numbers and names. The inodes are used to access the actual items in the directory. The names are just there for the ease-of-use, more or less. A hard link is nothing more than copying the inode number from one entry in some directory's contents into another entry. This second entry can be in a different directory's contents or even in the same directory (under a different name).
Since both directory entries have the same inode number, they point to the same item on the disk (ie the same physical file). Of course, inode numbers, as explained above, are partition-specific. So, duplicating an inode number on a different partition would not work as expected. That's why hard links cannot work across partitions.
Internally, the inodes of items such as files and directories also contain a link counter. This counter holds the number of (hard) links to the item. When you delete the item (using "rm" for instance), you internally "unlink" it (hence the term "unlink" instead of "delete" or "remove" that you see in some shells, like Perl). Unlinking simply decreases the link counter in the inode and deletes the entry in the directory's contents list. If the link counter drops to 0 (the last link is deleted), then the disk blocks occupied by the item get freed. When new items are created on the disk later on, they may use the freed blocks and overwrite them. In other words, when the last link is deleted, the item becomes unreachable (as if it was deleted from the disk). So, as long as the last link isn't deleted the contents of the item (file/directory) are still accessible and usable via the remaining hard links.
Symbolic links are created by "ln -s", hard links via simple "ln". See ln's man pages for details.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I recently accidentally called "rm -rf *" on a directory an deleted some files that I needed. However, I was able to recover most of them using photorec. Apparently, "deleting" a file just removes references to said file and is not truly deleted until it is overwritten by something else.
So if I wanted to remove the file completely, couldn't I just execute
mv myfile.txt /temp/myfile.txt
(or move to external storage)
You should consider using the Linux command shred, which overwrites the target file multiple times before deleting it completely, which makes it 'impossible' to recover the file.
You can read a bit about the shred command here.
Just moving the file does not cover you for good, if you moved it to external storage, the local version of the file is deleted just as it is with the rm command.
No. that won't help either.
A move when going between file systems is really still just a "copy + rm" internally. The original storage location of the file on the "source" media is still there, just marked as available. A moving WITHIN a file system doesn't touch the file bytes at all, it just updates the bookkeeping to say "file X is now in location Y".
To truly wipe a file, you must overwriteall of its bytes. And yet again, technology gets in the way of that - if you're using a solid state storage medium, there is a VERY high chance that writing 'garbage' to the file won't touch the actual transistors the file's stored in, but actually get written somewhere completely different.
For magnetic media, repeated overwriting with alternating 0x00, 0xFF, and random bytes will eventually totally nuke the file. For SSD/flash systems, it either has to offer a "secure erase" option, or you have to smash the chips into dust. For optical media, it's even more complicated. -r media cannot be erased, only destroyed. for -rw, I don't know how many repeated-write cycles are required to truly erase the bits.
No (and not just because moving it somewhere else on your computer is not removing it from the computer). The way to completely remove a file is to completely overwrite the space on the disk where it resided. The linux command shred will accomplish this.
Basically, no, in most file systems you can't guarantee that a file is overwritten without going very low level. Removing a file and/or moving it will only change the pointer to the file, not the files existence in the file system in any way. Even the linux command shred won't guarantee a file's removal in many file systems since it assumes files are overwritten in place.
On SSDs, it's even more likely that your data stays there for a long time, since even if the file system would attempt to overwrite blocks, the SSD will remap to write to a new block (erasing takes a lot of time, if it wrote in place things would be very slow)
In the end, with modern file systems and disks, the best chance you have to have files stored securely is to keep them encrypted to begin with. If they're stored anywhere in clear text, they can be very hard to remove, and recovering an encrypted file from disk (or a backup for that matter) won't be much use to anyone without the encryption key.
I'm developing a LAMP online store, which will allow admins to upload multiple images for each item.
My concern is - right off the bat there will be 20000 items meaning roughly 60000 images.
Questions:
What is the maximum number of files and/or directories on Linux?
What is the usual way of handling this situation (best practice)?
My idea was to make a directory for each item, based on its unique ID, but then I'd still have 20000 directories in a main uploads directory, and it will grow indefinitely as old items won't be removed.
Thanks for any help.
ext[234] filesystems have a fixed maximum number of inodes; every file or directory requires one inode. You can see the current count and limits with df -i. For example, on a 15GB ext3 filesystem, created with the default settings:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda 1933312 134815 1798497 7% /
There's no limit on directories in particular beyond this; keep in mind that every file or directory requires at least one filesystem block (typically 4KB), though, even if it's a directory with only a single item in it.
As you can see, though, 80,000 inodes is unlikely to be a problem. And with the dir_index option (enablable with tune2fs), lookups in large directories aren't too much of a big deal. However, note that many administrative tools (such as ls or rm) can have a hard time dealing with directories with too many files in them. As such, it's recommended to split your files up so that you don't have more than a few hundred to a thousand items in any given directory. An easy way to do this is to hash whatever ID you're using, and use the first few hex digits as intermediate directories.
For example, say you have item ID 12345, and it hashes to 'DEADBEEF02842.......'. You might store your files under /storage/root/d/e/12345. You've now cut the number of files in each directory by 1/256th.
If your server's filesystem has the dir_index feature turned on (see tune2fs(8) for details on checking and turning on the feature) then you can reasonably store upwards of 100,000 files in a directory before the performance degrades. (dir_index has been the default for new filesystems for most of the distributions for several years now, so it would only be an old filesystem that doesn't have the feature on by default.)
That said, adding another directory level to reduce the number of files in a directory by a factor of 16 or 256 would drastically improve the chances of things like ls * working without over-running the kernel's maximum argv size.
Typically, this is done by something like:
/a/a1111
/a/a1112
...
/b/b1111
...
/c/c6565
...
i.e., prepending a letter or digit to the path, based on some feature you can compute off the name. (The first two characters of md5sum or sha1sum of the file name is one common approach, but if you have unique object ids, then 'a'+ id % 16 is easy enough mechanism to determine which directory to use.)
60000 is nothing, 20000 as well. But you should put group these 20000 by any means in order to speed up access to them. Maybe in groups of 100 or 1000, by taking the number of the directory and dividing it by 100, 500, 1000, whatever.
E.g., I have a project where the files have numbers. I group them in 1000s, so I have
id/1/1332
id/3/3256
id/12/12334
id/350/350934
You actually might have a hard limit - some systems have 32 bit inodes, so you are limited to a number of 2^32 per file system.
In addition of the general answers (basically "don't bother that much", and "tune your filesystem", and "organize your directory with subdirectories containing a few thousand files each"):
If the individual images are small (e.g. less than a few kilobytes), instead of putting them in a folder, you could also put them in a database (e.g. with MySQL as a BLOB) or perhaps inside a GDBM indexed file. Then each small item won't consume an inode (on many filesystems, each inode wants at least some kilobytes). You could also do that for some threshold (e.g. put images bigger than 4kbytes in individual files, and smaller ones in a data base or GDBM file). Of course, don't forget to backup your data (and define a backup stategy).
The year is 2014. I come back in time to add this answer.
Lots of big/small files? You can use Amazon S3 and other alternatives based on Ceph like DreamObjects, where there are no directory limits to worry about.
I hope this helps someone decide from all the alternatives.
md5($id) ==> 0123456789ABCDEF
$file_path = items/012/345/678/9AB/CDE/F.jpg
1 node = 4096 subnodes (fast)
I'm writing an app that needs to store lots of files up to approx 10 million.
They are presently named with a UUID and are going to be around 4MB each but always the same size. Reading and writing from/to these files will always be sequential.
2 main questions I am seeking answers for:
1) Which filesystem would be best for this. XFS or ext4?
2) Would it be necessary to store the files beneath subdirectories in order to reduce the numbers of files within a single directory?
For question 2, I note that people have attempted to discover the XFS limit for number of files you can store in a single directory and haven't found the limit which exceeds millions. They noted no performance problems. What about under ext4?
Googling around with people doing similar things, some people suggested storing the inode number as a link to the file instead of the filename for performance (this is in a database index. which I'm also using). However, I don't see a usable API for opening the file by inode number. That seemed to be more of a suggestion for improving performance under ext3 which I am not intending to use by the way.
What are the ext4 and XFS limits? What performance benefits are there from one over the other and could you see a reason to use ext4 over XFS in my case?
You should definitely store the files in subdirectories.
EXT4 and XFS both use efficient lookup methods for file names, but if you ever need to run tools over the directories such as ls or find you will be very glad to have the files in manageable chunks of 1,000 - 10,000 files.
The inode number thing is to improve the sequential access performance of the EXT filesystems. The metadata is stored in inodes and if you access these inodes out of order then the metadata accesses are randomized. By reading your files in inode order you make the metadata access sequential too.
Modern filesystems will let you store 10 million files all in the same directory if you like. But tools (ls and its friends) will not work well.
I'd recommend putting a single level of directories, a fixed number, perhaps 1,000 directories, and putting the files in there (10,000 files is tolerable to the shell, and "ls").
I've seen systems which create many levels of directories, this is truly unnecessary and increases inode consumption and makes traversal slower.
10M files should not really be a problem either, unless you need to do bulk operations on them.
I expect you will need to prune old files, but something like "tmpwatch" will probably work just fine with 10M files.
I'd like to know how many I/O operations (iops) does it take to create an empty file. I am interested in linux and GFS file system, however other file systems information is also very welcome.
Suggestions how to accurately measure this would be also very welcome.
Real scenario (requested by answers):
Linux
GFS file system (if you can estimate for another - pls do)
create a new file in existing directory, the file does not exist,
using the following code
Assume directory is in cache and directory depth is D
Code:
int fd = open("/my_dir/1/2/3/new_file", O_CREAT | S_IRWXU);
// assuming fd is valid
fsync(fd);
For an artifical measurement:
Create a blank filesystem on its own block device (e.g. vmware scsi etc)
Mount it, call sync(), then record the number of IOPS present on that block dev.
Run your test program against the filesystem, and do no further operations (not even "ls").
Wait until all unflushed blocks have been flushed - say about 1 minute or so
Snapshot the iops count again
Of course this is highly unrealistic, because if you created two files rather than one, you'd probably find that there were less than twice as many.
Also creating empty or blank files is unrealistic - as they don't do anything useful.
Directory structures (how deep the directories are, how many entries) might contribute, but also how fragmented it is and other arbitrary factors.
The nature of the answer to this question is; best case, normal case, worst case. There is no single answer, because the number of IOPS required will vary according to the current state of the file system. (A pristine file system is highly unrealistic scenario).
Taking FAT32 as an example, best case is 1. Normal case depends on the degree of file system fragmentation and the directory depth of the pathname for the new file. Worse case is unbounded (except by the size of the file system, which imposes a limit on the maximum possible number of IOPs to create a file).
Really, the question is not answerable, unless you define a particular file system scenario.
I did the following measurement, we wrote an application that creates N files as described in the question.
We ran this application on a disk which was devoted to this application only, and measured IOps amount using iostat -x 1
The result, on GFS and linux kernel 2.6.18 is 2 IOps per file creation.
This answer is based on MarkR answer.