Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I am not looking for a tool. I am looking to script out a nice clean breakdown, so I can send it to my VPS users that don't know how to do this.
I was looking for a way to collect a lot of data through a shell script about disk usage. In my job, a LOT of people call in asking "why is my disk full"
I run du -h --max-depth=1 | sort -rn but that is a little clumsy and I have to keep digging around further....
I am trying to breakdown disk usage via a shell script like this:
**home/ is consuming 400GB of disk space**</br>
home/user1 120GB
home/user2 200GB
**var/ is using 100GB of disk space**
So far I came up with the following:
#!/bin/bash
for i in $(ls -d */ | grep -v proc);
do
printf "**** $i has the following breakdown ********\n"
du -h --max-depth=1 $i
done
What would be a way to give a cleaner, easier to understand breakdown of disk usage for my users? Just want to be able to wrap a usage in a pretty pink bow and say "good luck"
Give a try to ncdu (NCurses Disk Usage), for example:
$ ncdu -q -x
The options:
-q Quiet mode. While scanning or importing the directory, ncdu will
update the screen 10 times a second by default, this will be
decreased to once every 2 seconds in quiet mode. Use this
feature to save bandwidth over remote connections.
And
-x Do not cross filesystem boundaries, i.e. only count files and
directories on the same filesystem as the directory being
scanned.
More examples here: https://dev.yorhel.nl/ncdu/man#EXAMPLES
Here's an alternative solution. You can use tree to list the contents of the directory in a 'tree-like' format. You can then combine it with a disk usage flag to see the nested directories and their size.
tree -L 2 --du -h
The '-L' flag defines the depth or level of your query.
This will give you output like this:
You can even create an xml file you can open in the browser like so:
tree -L 2 --du -h -H s > disk_usage.html
You can then open the file in a browser and the output will look like this:
Hope this helps!
you snippet is a good start, you can't make this thing in 1 line.
some folders, you need a depth > 1 (home) and some other a depth = 1 (var)
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
I am trying to perform an image on a hard disk which is failing.
The issue I am encountering causes the program to fail as the disk will routinely drop during the image process and when it is re-recognised by the system it is under a different address (/dev/sdb is now /dev/sde).
I have tried imaging each partition independently but on a 500GB disk I am strugging to get past 100GB a session before the disk will drop (i think the head is going as it clicks).
My question is, if using dd is there a way to image the disk, breaking it down into say 50GB parts so that I can get the whole disk over a number of images and then consolodate.
Or better still, is there a way to force the disk to re-identify on the previous location?
I have found little information on this topic so any insight would be useful.
Thanks.
When the device is lost, your stream will be lost, too. You cannot recover it even if it gets the same device name assigned. However you might want to employ udev rules to get the same name back just for your convenience.
In dd, you can use four useful parameters:
bs=BYTES the size of a "block"
skip=N number of blocks to skip in input
seek=N number of blocks to skip in output
count=N number of blocks to be copied (we don't need it here)
Also, dd has the, albeit a bit hidden, feature of providing progress reports. You can either use "status=progress" or send a signal to the process. The latter is more complicated but it allows you to define the frequency of progress reports. For example, you can do this in another terminal:
for ((;;)); do sleep 1; kill -USR1 `pidof -s dd`; done
Putting all of this together, you can use bs=4M as a reasonable blocksize. Then you can run aforementioned command in a secondary terminal, then start dd, initially with
dd bs=4M seek=0 skip=0 if=/dev/… of=…
After it fails the first time, you use the last block number that was successfully copied by dd as parameters to seek and skip. You can be a bit conservative here (decrease the number a bit) to ensure you don't get any "holes" in your output.
Repeat until the whole disk is done. Good luck!
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
I hope this is the right place to ask this question.
While trying to move a big directory "mydirname" (abt900GB), in a remote linux server, from /abc/source to /xyz/target ; I used the following command in sourcedirectory,
mv mydirname /xyz/target/ &
However, after a while the process got interrupted and gave an error,
mv: cannot stat `mydirname/GS9/set04/trans/run.3/acc': Stale file handle
mv: cannot stat `mydirname/GS9/set04/trans/run.4/amc': Stale file handle
.
.
.
and many more such messages mentioning different subdirectories locations.
The problem is that, the process has moved about 300gb of data. However, there are many directories which are not fully moved. Similar, problem occurred with another transfer (about 500 GB) that was running at the same machine.
Also, I am no longer in the same working session. I have disconnected and reconnected to the remote server.
It would be great if you help with following queries.
Is it possible that some of the file are not fully-transferred (i have seen such cases in 'cp' command where if a process interrupts, it results in lesser size file at the destination.
How can I resume the process so that I do not loose any data. Will 'mv' command be enough? or is there any special command that can work in background.
Else, is there a command to undo the process and restore the 'mydirname' to original location 'source'.
Use "rsync" to complete a job like this:
rsync -av --delete mydirname/ /xyz/target
It will verify that all files are moved, of the proper length, correct timestamps and will delete any leftover garbage.
You can test first with a "dry run" to see what the damages are:
rsync -avn --delete mydirname/ /xyz/target
This goes through the whole rsync process but doesn't actually do anything. It's usually a good idea to run this test to check your command syntax and see if it's going to do what you think it should do.
The "rsync" command is actually more like a copy "cp" than a move "mv". It will leave the source files in place and you can delete them later when you are satisfied that everthing has transferred correctly.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Hi guys I need help steps to free my swap memory on CENTOS 5.9 during production. Also let me know can i add more swap in my existing swap memory during production time.
please let me the commands step by step.
Thanks
First, query the currently used swap partitions or files with
cat /proc/swaps
(and query the current state of the memory with the free command)
to remove a swap area on partition /dev/sdc1, use
swapoff /dev/sdc1
to add a swap area on partition /dev/sdc2 use
swapon /dev/sdc2
you need to be root to run these; be very careful.
You could, e.g. if you need more swap than usual, use (temporarily) a file for swap (but using partitions is faster). For that, create a big file /var/tmp/bigswap, e.g of 8192 Mbytes using
dd if=/dev/zero of=/var/tmp/bigswap bs=1M count=8192
then, make it a swap area with
mkswap /var/tmp/bigswap
at last, add it as swap using
swapon /var/tmp/bigswap
when you've done, remove it as swap using
swapoff /var/tmp/bigswap
and remove the file with rm
In general, permanent swap areas are defined in /etc/fstab (and activated using swapon -a).
See swapon(8), mkswap(8), fstab(5)
Don't forget to read http://linuxatemyram.com/
PS. It is rarely useful to explicitly free the swap memory, since the kernel is managing it quite well.
you can use swapon and swaoff utility:
sudo swapoff -a # here -a option will disable all swap partitions
sudo swapon -a # here -a option will enable all swap partitions
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I'm using Ubuntu 14.04 and I made an empty directory on /tmp with the mkdir command:
cd /tmp
mkdir foo
and then I checked it's size using ls:
ls -ldh foo
and the result shows that the size of the directory is 4KB, although it has nothing inside!
then I created an empty file with touch:
touch empty
and then I checked its size:
ls -l empty
the result shows that the empty file is of 0B, which differs from the empty directory.
I've read about some Q&A's saying that the 4KB is the metadata of the directory. But if it is the metadata, what kind of information is stored inside and why it is so huge, and why an empty file don't have such kind of metadata? If it is not the metadata, what does the 4KB mean?
I'm going to break this question down into 3 parts, 2 of which I can answer...
Part 1: why isn't an empty directory size 0?
Because it contains . and .. so it's not really empty.
Part 2: Why is 4K the minimum?
Because that's the filesystem's block size. You can set it smaller when you create the filesystem, but there is overhead. The filesystem must remember a free-or-in-use flag for every block, so smaller blocks = more blocks = more overhead. (In the early days of ext2, the default block size was 1K. Disks were small enough that the space saved by not allocating a multiple of 4K for every file was more important than the space used for the free block map.)
Block sizes over 4K aren't possible because 4K is the page size (the smallest unit of virtual memory) on most processors, and Linux wasn't designed to deal with filesystem blocks bigger than memory pages.
Part 3: When you ls -l a regular file, you get the actual number of bytes used but when you ls -ld a directory, you get the number of bytes allocated. Why?
This part I don't know. For regular files, there is an allocation size you can view with ls -s, and the two sizes actually tell you different things. But on directories, the -l size is like a redundant copy of the -s size. Presumably the kernel could report a size that indicates how much of the 4K block is actually used, but it doesn't. I don't know why.
The metadata a directory contains is a series of directory entries. It's not empty upon creation because two dirents are immediately created: one for that directory, called ".", and one for its parent directory, called "..".
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 5 years ago.
Improve this question
Is there a maximum number of inodes in a single directory?
I have a directory of over 2 million files and can't get the ls command to work against that directory. So now I'm wondering if I've exceeded a limit on inodes in Linux. Is there a limit before a 2^64 numerical limit?
df -i should tell you the number of inodes used and free on the file system.
Try ls -U or ls -f.
ls, by default, sorts the files alphabetically. If you have 2 million files, that sort can take a long time. If ls -U (or perhaps ls -f), then the file names will be printed immediately.
No. Inode limits are per-filesystem, and decided at filesystem creation time. You could be hitting another limit, or maybe 'ls' just doesn't perform that well.
Try this:
tune2fs -l /dev/DEVICE | grep -i inode
It should tell you all sorts of inode related info.
What you hit is an internal limit of ls. Here is an article which explains it quite well:
http://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/
Maximum directory size is filesystem-dependent, and thus the exact limit varies. However, having very large directories is a bad practice.
You should consider making your directories smaller by sorting files into subdirectories. One common scheme is to use the first two characters for a first-level subdirectory, as follows:
${topdir}/aa/aardvark
${topdir}/ai/airplane
This works particularly well if using UUID, GUIDs or content hash values for naming.
As noted by Rob Adams, ls is sorting the files before displaying them. Note that if you are using NFS, the NFS server will be sorting the directory before sending it, and 2 million entries may well take longer than the NFS timeout. That makes the directory unlistable via NFS, even with the -f flag.
This may be true for other network file systems as well.
While there's no enforced limit to the number of entries in a directory, it's good practice to have some limit to the entries you anticipate.
Can you get a real count of the number of files? Does it fall very near a 2^n boundry? Could you simply be running out of RAM to hold all the file names?
I know that in windows at least file system performance would drop dramatically as the number of files in the folder went up, but I thought that linux didn't suffer from this issue, at least if you were using a command prompt. God help you if you try to get something like nautilus to open a folder with that many files.
I'm also wondering where these files come from. Are you able to calculate file names programmatically? If that's the case, you might be able to write a small program to sort them into a number of sub-folders. Often listing the name of a specific file will grant you access where trying to look up the name will fail. For example, I have a folder in windows with about 85,000 files where this works.
If this technique is successful, you might try finding a way to make this sort permanent, even if it's just running this small program as a cron job. It'll work especially well if you can sort the files by date somewhere.
Unless you are getting an error message, ls is working but very slowly. You can try looking at just the first ten files like this:
ls -f | head -10
If you're going to need to look at the file details for a while, you can put them in a file first. You probably want to send the output to a different directory than the one you are listing at the moment!
ls > ~/lots-of-files.txt
If you want to do something to the files, you can use xargs. If you decide to write a script of some kind to do the work, make sure that your script will process the list of files as a stream rather than all at once. Here's an example of moving all the files.
ls | xargs -I thefilename mv thefilename ~/some/other/directory
You could combine that with head to move a smaller number of the files.
ls | head -10000 | xargs -I x mv x /first/ten/thousand/files/go/here
You can probably combine ls | head into a shell script to that will split up the files into a bunch of directories with a manageable number of files in each.
For NetBackup, the binaries that analyze the directories in clients perform some type of listing that timeouts by the enormous quantity of files in every folder (about one million per folder, SAP work directory).
My solution was (as Charles Duffy write in this thread), reorganize the folders in subfolders with less archives.
Another option is find:
find . -name * -exec somcommands {} \;
{} is the absolute filepath.
The advantage/disadvantage is that the files are processed one after each other.
find . -name * > ls.txt
would print all filenames in ls.txt
find . -name * -exec ls -l {} \; > ls.txt
would print all information form ls for each file in ls.txt