Join of output from DF and LSBLK Linux commands via bash - linux

I need to merge two outputs in Linux.
This:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | grep -v "fd0" | grep -v "loop" | grep -v "sr0" | grep -v "hdc" | grep -v "cdrom"
In a result I have:
sda sda 53687091200
sda1 └─sda1 53684994048
dm-3 └─dockerVG-rootLV 53682896896 /
sdb sdb 2147483648000
sdb1 └─sdb1 2147482599424
dm-1 ├─hddVG-dockerLV 536866717696 /var/lib/docker
dm-2 └─hddVG-hddLV 1610612736000 /dockerhdd
sdc sdc 536870912000
sdc1 └─sdc1 536869863424
dm-0 └─ssdVG-ssdLV 536866717696 /dockerssd
And this:
df --exclude={tmpfs,devtmpfs,squashfs,overlay} | sed -e /^Filesystem/d | awk '{print $6 " " $1 " " $3 " " $4 " " $5}'
In a result I have:
/ /dev/mapper/dockerVG-rootLV 8110496 40591632 17%
/dockerssd /dev/mapper/ssdVG-ssdLV 214133656 274642488 44%
/dockerhdd /dev/mapper/hddVG-hddLV 83278236 1385191240 6%
/var/lib/docker /dev/mapper/hddVG-dockerLV 76046204 412729940 16%
So, I want to Join via these points /, /var/lib/docker, /dockerhdd, /dockerssd.
Important! I want to check this in another place, where we will have another mount points. Also I have to save structure of first output without sorting.
In a result I have to receive something like this:
sda sda 53687091200
sda1 └─sda1 53684994048
dm-3 └─dockerVG-rootLV 53682896896 / /dev/mapper/dockerVG-rootLV 8110496 40591632 17%
sdb sdb 2147483648000
sdb1 └─sdb1 2147482599424
dm-1 ├─hddVG-dockerLV 536866717696 /var/lib/docker /dev/mapper/hddVG-dockerLV 76046204 412729940 16%
dm-2 └─hddVG-hddLV 1610612736000 /dockerhdd /dev/mapper/hddVG-hddLV 83278236 1385191240 6%
sdc sdc 536870912000
sdc1 └─sdc1 536869863424
dm-0 └─ssdVG-ssdLV 536866717696 /dockerssd /dev/mapper/ssdVG-ssdLV 214133656 274642488 44%
Of course better to have one-liner, but if it is not possible, we can send output to separate files and join them. Could You please help me in this ?

Using awk:
awk '!/^\/&^fd0&^loop&^sr0&^hdc&^cdrom/ { print $0" "arr[$4] } /^Filesystem/ { mrk=1;next } mrk==1 && /^\// { arr[$1]=$0 }' <<< $(df --exclude={tmpfs,devtmpfs,squashfs,overlay};lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT)
Redirect the two commands back into awk, stripping out any grep and sed processing. We process the df command first and where we find a line beginning with "Filesystem" we set a marker (mrk) to 1 and move to the next line. We then create an array (arr) indexed with the mountpoint and containing the line returned from the df command. We move onto the lsblk command and search for the lines starting with the KNAMEs required. We print the line from the lsblk command and append the value in the arr array indexed by the mount point ($4)

Related

Add the UUID from blkid into the /etc/fstab

Need to add the UUID of the disk into the /etc/fstab file.
Input
cat /blkid | awk '{print $2}' | <TODO:>
UUID=e3vm2eea-9oe6-4k01-420f-554fd5frc0
UUID=e4vm2eea-9oe6-4j01-420f-143fx5fkc0
UUID=e5vm2eea-9oe6-4i01-420f-154fd5lhc0
Expected Output :
<file system> <mount point> <type default value> <options default value> <dump default value> <pass default value>
UUID=e3vm2eea-9oe6-4k01-420f-554fd5frc0 /part/1 ext4 acl,rw,noatime 0 2
UUID=e4vm2eea-9oe6-4j01-420f-143fx5fkc0 /part/2 ext4 acl,rw,noatime 0 2
UUID=e5vm2eea-9oe6-4i01-420f-154fd5lhc0 /part/3 ext4 acl,rw,noatime 0 2
Along with UUID need to add the mount partitions, type, option, dump, pass and Mount Partitions is dynamic (1,2,3) All should expect in shell command.
like this ?
# blkid | awk '{print $2" /part/"NR" ext4 acl,rw,noatime 0 2"}'

Create udev rules file from two input file

I am looking for a solution to create Oracle ASM udev rules file for linux. I have two input file. file1 has info of ASM disk requirement and file2 has disk information.
For example, line 2 of file1 is showing DATA12 need 3 disk(DATA12_01,DATA12_02,DATA12_03) of each 128G. file2 has all disk info with size. From these two input file I need to create output file shown bellow.
cat file1
Count - size - name
3 - 128 GB DATA12
1 - 128 GB TEMP02
2 - 4 GB ARCH03
2 - 1 GB ARCH04
1 - 3 GB ORAC01
cat file2
UUID Size
360060e80166ef70000016ef700006700 128.00 GiB
360060e80166ef70000016ef700006701 128.00 GiB
360060e80166ef70000016ef700006702 128.00 GiB
360060e80166ef70000016ef700006703 128.00 GiB
360060e80166ef70000016ef700006730 4.00 GiB
360060e80166ef70000016ef700006731 4.00 GiB
360060e80166ef70000016ef700006733 1.00 GiB
360060e80166ef70000016ef700006734 1.00 GiB
360060e80166ef70000016ef700006735 3.00 GiB
Output File
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006700", SYMLINK+="udevlinks/DATA12_01"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006701", SYMLINK+="udevlinks/DATA12_02"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006702", SYMLINK+="udevlinks/DATA12_03"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006703", SYMLINK+="udevlinks/TEMP02_01"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006730", SYMLINK+="udevlinks/ARCH03_01"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006731", SYMLINK+="udevlinks/ARCH03_02"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006733", SYMLINK+="udevlinks/ARCH04_01"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006734", SYMLINK+="udevlinks/ARCH04_02"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006735", SYMLINK+="udevlinks/ORAC01_01"
Here is one in AWK:
$ cat > test.awk
BEGIN {FS="([.]| +)"} # field separator do deal with "." in file2 128.00
FNR==1 {next} # skip header
NR==FNR { # read available disks to pool from file1
for(i=1; i<=$1; i++)
a[$5"_"0i]=$3 # name and set the disks into pool
next}
{
for(i in a) { # look for right sized disk
if(a[i]==$2) { # when found, print...
printf "%s%s%s%s%s", "ACTION==\"add|change\", ENV{DM_NAME}==\"",$1,"\",\"SYMLINK+=\"udevlinks/",i,"\"\n"
delete a[i] # ... and remove from pool
break
}
} # if no device was found:
old=len; len=length(a); if(old==len) {print "No device found for ",$0}
}
$ awk -f test.awk file1 file2
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006700","SYMLINK+="udevlinks/DATA12_01"
ACTION=="add|change", ENV{DM_NAME}=="360060e80166ef70000016ef700006701","SYMLINK+="udevlinks/DATA12_02"
...
No device found for THIS_IS_AN_EXAMPLE_OF_MISSING_DISK 666.00 GiB
Due to disk search using for(i in a) no order in which disks are read from the pool is guaranteed.

Perl - get free disk space usage on linux

I'm wondering how I can get the value of the second row, 4th column from df ("/"). Here's the output from df:
Filesystem Size Used Avail Use% Mounted on
rootfs 208G 120G 78G 61% /
fakefs 208G 120G 78G 61% /root
fakefs 1.8T 1.3T 552G 70% /home4/user
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/bin
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/etc
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/php
fakefs 208G 120G 78G 61% /var/lib
fakefs 208G 120G 78G 61% /var/lib/mysql
fakefs 208G 120G 78G 61% /var/log
fakefs 208G 120G 78G 61% /var/spool
fakefs 208G 120G 78G 61% /var/run
fakefs 4.0G 361M 3.7G 9% /var/tmp
fakefs 208G 120G 78G 61% /var/cache/man
I'm trying to get the available free space (78GB) using perl which I'm fairly new to. I'm able to get the value using the following linux command but I've heard it's not necessary to use awk in perl at all because perl can do what awk can natively.
df -h | tail -n +2 | sed -n '2p' | awk '{ print $4 }'
I'm stumped. I tried using the Filesys::df module but when I'd print out the available usage percent, it'd give me a different value than what running df from command line does. Help is appreciated.
A little more succinctly:
df -h | perl -wlane 'print $F[3] if $. == 2;'
-w enable warnings
-l add newline to output(and chomps newline from input line)
-a splits the fields on whitespace into the #F array, which you access using the syntax $F[n] (first column is at index position 0)
-n puts the code inside the following loop:
LINE:
while (<>) {
... # code goes here
}
# <> reads lines from STDIN if no filenames are given on the command line
-e execute the string
$. current line number in the file (For the first line, $. is 1)
If you wish to do this all in perl, then:
df -h | perl -e 'while (<stdin>) { if ($. == 2) { #x = split; print $x[3] }}'
This uses perl alone to read the output of df -h and, for the second record ($. == 2) splits the record into fields, based on whitespace, and outputs field 3 (counting from 0).
This seems to work ok too:
df -h | awk 'NR==2 {print $4}'
Get the second line and pint fourth field.

Linux differences between consecutive lines

I need to loop trough n lines of a file and for any i between 1 and n - 1 to get the difference line(n - 1) - line(n).
And here is the source file:
root#syncro:/var/www# cat cron.log | grep "/dev/vda"
/dev/vda 20418M 14799M 4595M 77% /
/dev/vda 20418M 14822M 4572M 77% /
/dev/vda 20418M 14846M 4548M 77% /
/dev/vda 20418M 14867M 4527M 77% /
/dev/vda 20418M 14888M 4506M 77% /
/dev/vda 20418M 14910M 4484M 77% /
/dev/vda 20418M 14935M 4459M 78% /
/dev/vda 20418M 14953M 4441M 78% /
/dev/vda 20418M 14974M 4420M 78% /
/dev/vda 20418M 15017M 4377M 78% /
/dev/vda 20418M 15038M 4356M 78% /
root#syncro:/var/www# cat cron.log | grep "/dev/vda" | cut -b 36-42 | tr -d " M"
4595
4572
4548
4527
4506
4484
4459
4441
4420
4377
4356
those /dev/vda... lines are logged hourly with df -BM in cron.log file and the difference between lines will reveal the hourly disk consumption.
So, the expected output will be:
23 (4595 - 4572)
24 (4572 - 4548)
...
43 (4420 - 4377)
21 (4377 - 4356)
I don't need the text between ( and ), I put it here for explanation only.
I'm not sure if I got you correctly, but the following awk script should work:
awk '{if(NR>1){print _n-$4};_n=$4}' your.file
Output:
23
24
21
21
22
25
18
21
43
21
You don't need the other programs in the pipe. Just:
awk '/\/dev\/vda/ {if(c++>0){print _n-$4};_n=$4}' src/checkout-plugin/a.txt
will be enough. The regex on start of the awk scripts tells awk to apply the following block only to lines which match the pattern. A side effect is that NR can't be used anymore to detect the "second line" in which the calculation starts. I introduced a custome counter c for that purpose.
Also note that awk will remove the M on it's own, because the column has been used in a numeric calculation.

How to find user memory usage in linux

How i can see memory usage by user in linux centos 6
For example:
USER USAGE
root 40370
admin 247372
user2 30570
user3 967373
This one-liner worked for me on at least four different Linux systems with different distros and versions. It also worked on FreeBSD 10.
ps hax -o rss,user | awk '{a[$2]+=$1;}END{for(i in a)print i" "int(a[i]/1024+0.5);}' | sort -rnk2
About the implementation, there are no shell loop constructs here; this uses an associative array in awk to do the grouping & summation.
Here's sample output from one of my servers that is running a decent sized MySQL, Tomcat, and Apache. Figures are in MB.
mysql 1566
joshua 1186
tomcat 353
root 28
wwwrun 12
vbox 1
messagebus 1
avahi 1
statd 0
nagios 0
Caveat: like most similar solutions, this is only considering the resident set (RSS), so it doesn't count any shared memory segments.
EDIT: A more human-readable version.
echo "USER RSS PROCS" ; echo "-------------------- -------- -----" ; ps hax -o rss,user | awk '{rss[$2]+=$1;procs[$2]+=1;}END{for(user in rss) printf "%-20s %8.0f %5.0f\n", user, rss[user]/1024, procs[user];}' | sort -rnk2
And the output:
USER RSS PROCS
-------------------- -------- -----
mysql 1521 1
joshua 1120 28
tomcat 379 1
root 19 107
wwwrun 10 10
vbox 1 3
statd 1 1
nagios 1 1
messagebus 1 1
avahi 1 1
Per-user memory usage in percent using standard tools:
for _user in $(ps haux | awk '{print $1}' | sort -u)
do
ps haux | awk -v user=${_user} '$1 ~ user { sum += $4} END { print user, sum; }'
done
or for more precision:
TOTAL=$(free | awk '/Mem:/ { print $2 }')
for _user in $(ps haux | awk '{print $1}' | sort -u)
do
ps hux -U ${_user} | awk -v user=${_user} -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done
The first version just sums up the memory percentage for each process as reported by ps. The second version sums up the memory in bytes instead and calculates the total percentage afterwards, thus leading to a higher precision.
If your system supports, try to install and use smem:
smem -u
User Count Swap USS PSS RSS
gdm 1 0 308 323 820
nobody 1 0 912 932 2240
root 76 0 969016 1010829 1347768
or
smem -u -t -k
User Count Swap USS PSS RSS
gdm 1 0 308.0K 323.0K 820.0K
nobody 1 0 892.0K 912.0K 2.2M
root 76 0 937.6M 978.5M 1.3G
ameskaas 46 0 1.2G 1.2G 1.5G
124 0 2.1G 2.2G 2.8G
In Ubuntu, smem can be installed by typing
sudo apt install smem
This will return the total ram usage by users in GBs, reverse sorted
sudo ps --no-headers -eo user,rss | awk '{arr[$1]+=$2}; END {for (i in arr) {print i,arr[i]/1024/1024}}' | sort -nk2 -r
You can use the following Python script to find per-user memory usage using only sys and os module.
import sys
import os
# Get list of all users present in the system
allUsers = os.popen('cut -d: -f1 /etc/passwd').read().split('\n')[:-1]
for users in allUsers:
# Check if the home directory exists for the user
if os.path.exists('/home/' + str(users)):
# Print the current usage of the user
print(os.system('du -sh /home/' + str(users)))

Resources