For example, the dir "/app/source"
There is an 100GB filesystem mount on "/"
So when I use "df /app/source" I can get the capacity is 100GB.
Then there is a dir "/app/source/video" and an 100GB filesystem mount on it.
Is there any easy way to get the real capacity (200GB) of "/app/source" ?
/app/source don't have a capacity of 200G, thus you cannot expect to see it. It's "real" capacity is 100G as the underlying disk capacity is 100G. If you think it has a capacity of 200G, then you expect to be able to store 200G of data in /app/source but you cannot ! You can store 100G in /app/source and 100G in /app/source/video
Maybe you would like to really merge the capacity of both partitions, for this you could use LVM.
Trying to merge only the reported numbers, which you could do with a simple script or alias (see below), would give you bad information and then you may try to add files on a full partition.
If at the end you still need the added total, maybe something like this can help:
# df -h --total /app/source /app/source/video | grep total | awk -F" " '{print $2}'
Related
Sorted output of free space for each partition.
In other words, a table with at least two columns: partition name and free or unsupported storage space sorted according to free or unused storage space.
Motivation: I want to back up files and use a hard drive that still has as much space as possible.
what i tried:
df -h | sort -h -r
if found that's sound great
diskutil info disk1s4 | awk '/Free Space:.* GB/ {print $3,$4}'
but not working at my Manjaro-Linux. Command not found
First of all, I guess you meant "use partition" instead of hard drive.
df -l --output=source,avail|sed '/\/dev\//!d'|sort -nr -k2
Notes:
-l only local devices, that is, network shares are not listed
the sed part removes the title and filesystems not under /dev/ adjust this if you want something else.
sort does sort.
the mountpoints are not listed, if they are required, add target in the --output list.
I deploy one seaweedfs master and one volume server
/usr/bin/weed master -ip=10.110.200.149 -port=9333 -mdir=/weed/mdir
/usr/bin/weed volume -ip=10.110.200.149 -dir=/weed/vdir -port=8080 -mserver=10.110.200.149:9333 -max=7
When running for several weeks, it show error:
curl -X POST http://10.110.200.149:9333/dir/assign
{"error":"No free volumes left!"}
I change volumes from 7 to 50(parameter max), it solved. But I check the disk size of seaweedfs usage
[root#node149 vdir]# ls
1.dat 1.idx 2.dat 2.idx 3.dat 3.idx 4.dat 4.idx 5.dat 5.idx 6.dat 6.idx 7.dat 7.idx
[root#node149 vdir]# du . -hs
14M .
[root#node149 vdir]#
It show only 14M disk space usage, so what's the really meaning of number of volumes?
If you have uploaded any files into the FS. Then the total space used by the files will be what you are seeing as 14M.
If you were expecting to see 50 volumes use 50 times the space allocated for each volume then you are wrong because this space will be used when fully filled with files.
Before you store any files in your weed-fs, the space used by all volumes should be a low number and it increases as you add more files.
If i know that partition is for example /dev/sda1 how can i get disk name (/dev/sda in this case) that contain the partition?
The output should be only path to disk. (like '/dev/sda')
EDIT: It shouldn't be string manipulation
You can use the shell's built-in string chopping:
$ d=/dev/sda1
$ echo ${d%%[0-9]*}
/dev/sda
$ d=/dev/sda11212
$ echo ${d%%[0-9]*}
/dev/sda
This works for some of the disk names only. If there can be several digits in the name, it will chop everything after the first.
What is the exact specification to separate a disk name from a partition name?
You can use sed to get the disk. Because partitions are just increments of disk names, it's easy to perform:
echo "/dev/sda1" | sed 's/[0-9]*//g'
which produces the output /dev/sda
Another command you can use to obtain disk information is lsblk. Just typing it without args prints out all info pertaining to your disks and partitions.
Problem Statement:-
I am getting this below exception-
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException:
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace
quota of /tmp is exceeded: quota=659706976665600 diskspace consumed=614400.1g
So I just wanted to know how much is the size of /tmp directory currently and because of that I am getting this exception. How can I see the free space in /tmp?
Update:-
bash-3.00$ df -h /tmp
Filesystem size used avail capacity Mounted on
rpool/tmp 10G 2.2G 7.8G 22% /tmp
I am puzzled right now why I am getting that exception then as it clearly states above that I have space available.
You can do (For SunOS)
# du -sh /tmp
To see how much it uses now, but that you already saw.
To see how much total, free and used space is on the partition where /tmp resides you can use:
# df -h /tmp
Note that filling up space is not the only thing that can prevent writing to filesystem.
Running out of inodes is another popular reason.
You can check that with
# df -i /tmp
for a in ls; do du -ch ${a} | grep total ; echo ${a}; done
try this but it takes time if the size of dir is in GBs
Managing HDFS Space Quotas
It’s important to understand that in HDFS, there must be enough quota space to accommodate an entire block. If the user has, let’s say, 200MB free in their allocated quota, they can’t create a new file, regardless of the file size, if the HDFS block size happens to be 256MB. You can set the HDFS space quota for a user by executing the setSpace-Quota command. Here’s the syntax:
$ hdfs dfsadmin –setSpaceQuota <N> <dirname>...<dirname>
The space quota you set acts as the ceiling on the total size of all files in a directory. You can set the space quota in bytes (b), megabytes (m), gigabytes (g), terabytes (t) and even petabytes (by specifying p—yes, this is big data!). And here’s an example that shows how to set a user’s space quota to 60GB:
$ hdfs dfsadmin -setSpaceQuota 60G /user/alapati
You can set quotas on multiple directories at a time, as shown here:
$ hdfs dfsadmin -setSpaceQuota 10g /user/alapati /test/alapati
Assuming that there are only primary partitions on a disk, what is the best way to find the current number of partitions?
Is there any better way than:
fdisk -l > temp
#Following returns first column of the last line of temp e.g. /dev/sda4
lastPart=$(tail -n 1 temp | awk '{print $1}')
totalPartitions=$(echo ${lastPart:8})
$totalPartitions variable sometimes returns NULL. That's why, I was wondering if there is a more reliable way to find the current number of partitions.
What about:
totalPartitions=$(grep -c 'sda[0-9]' /proc/partitions)
?
(Where sda is the name of the disk you're interested in, replacing it as appropriate)
I found this question while I was writing a script to safely wipe test and re-provision storage, which is sometimes a memory card, so mmcblk0p1 is often the format of its partitions.
Here's my answer:
diskICareAbout="sda"
totalPartitions="$( ls /sys/block/${diskICareAbout}/*/partition | wc -l )"
/proc/partitions is archaic and flat. The sys filesystem can comunicate the heirarchal nature of partitions well enough that grep is not needed.
You can use partx for this.
partx -g /dev/<disk> | wc -l
will return the total number of partitions (-g omits the header line). To get the last partition on a disk, use
partx -rgo NR -n -1:-1 /dev/<disk>
which may be useful if there are gaps in the partition numbers. -r omits aligning spaces, and -o specifies the comma-separated columns to include. -n specifies a range of partitions start:end, where -1 is the last partition.