I need to loop trough n lines of a file and for any i between 1 and n - 1 to get the difference line(n - 1) - line(n).
And here is the source file:
root#syncro:/var/www# cat cron.log | grep "/dev/vda"
/dev/vda 20418M 14799M 4595M 77% /
/dev/vda 20418M 14822M 4572M 77% /
/dev/vda 20418M 14846M 4548M 77% /
/dev/vda 20418M 14867M 4527M 77% /
/dev/vda 20418M 14888M 4506M 77% /
/dev/vda 20418M 14910M 4484M 77% /
/dev/vda 20418M 14935M 4459M 78% /
/dev/vda 20418M 14953M 4441M 78% /
/dev/vda 20418M 14974M 4420M 78% /
/dev/vda 20418M 15017M 4377M 78% /
/dev/vda 20418M 15038M 4356M 78% /
root#syncro:/var/www# cat cron.log | grep "/dev/vda" | cut -b 36-42 | tr -d " M"
4595
4572
4548
4527
4506
4484
4459
4441
4420
4377
4356
those /dev/vda... lines are logged hourly with df -BM in cron.log file and the difference between lines will reveal the hourly disk consumption.
So, the expected output will be:
23 (4595 - 4572)
24 (4572 - 4548)
...
43 (4420 - 4377)
21 (4377 - 4356)
I don't need the text between ( and ), I put it here for explanation only.
I'm not sure if I got you correctly, but the following awk script should work:
awk '{if(NR>1){print _n-$4};_n=$4}' your.file
Output:
23
24
21
21
22
25
18
21
43
21
You don't need the other programs in the pipe. Just:
awk '/\/dev\/vda/ {if(c++>0){print _n-$4};_n=$4}' src/checkout-plugin/a.txt
will be enough. The regex on start of the awk scripts tells awk to apply the following block only to lines which match the pattern. A side effect is that NR can't be used anymore to detect the "second line" in which the calculation starts. I introduced a custome counter c for that purpose.
Also note that awk will remove the M on it's own, because the column has been used in a numeric calculation.
Related
I need to merge two outputs in Linux.
This:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | grep -v "fd0" | grep -v "loop" | grep -v "sr0" | grep -v "hdc" | grep -v "cdrom"
In a result I have:
sda sda 53687091200
sda1 └─sda1 53684994048
dm-3 └─dockerVG-rootLV 53682896896 /
sdb sdb 2147483648000
sdb1 └─sdb1 2147482599424
dm-1 ├─hddVG-dockerLV 536866717696 /var/lib/docker
dm-2 └─hddVG-hddLV 1610612736000 /dockerhdd
sdc sdc 536870912000
sdc1 └─sdc1 536869863424
dm-0 └─ssdVG-ssdLV 536866717696 /dockerssd
And this:
df --exclude={tmpfs,devtmpfs,squashfs,overlay} | sed -e /^Filesystem/d | awk '{print $6 " " $1 " " $3 " " $4 " " $5}'
In a result I have:
/ /dev/mapper/dockerVG-rootLV 8110496 40591632 17%
/dockerssd /dev/mapper/ssdVG-ssdLV 214133656 274642488 44%
/dockerhdd /dev/mapper/hddVG-hddLV 83278236 1385191240 6%
/var/lib/docker /dev/mapper/hddVG-dockerLV 76046204 412729940 16%
So, I want to Join via these points /, /var/lib/docker, /dockerhdd, /dockerssd.
Important! I want to check this in another place, where we will have another mount points. Also I have to save structure of first output without sorting.
In a result I have to receive something like this:
sda sda 53687091200
sda1 └─sda1 53684994048
dm-3 └─dockerVG-rootLV 53682896896 / /dev/mapper/dockerVG-rootLV 8110496 40591632 17%
sdb sdb 2147483648000
sdb1 └─sdb1 2147482599424
dm-1 ├─hddVG-dockerLV 536866717696 /var/lib/docker /dev/mapper/hddVG-dockerLV 76046204 412729940 16%
dm-2 └─hddVG-hddLV 1610612736000 /dockerhdd /dev/mapper/hddVG-hddLV 83278236 1385191240 6%
sdc sdc 536870912000
sdc1 └─sdc1 536869863424
dm-0 └─ssdVG-ssdLV 536866717696 /dockerssd /dev/mapper/ssdVG-ssdLV 214133656 274642488 44%
Of course better to have one-liner, but if it is not possible, we can send output to separate files and join them. Could You please help me in this ?
Using awk:
awk '!/^\/&^fd0&^loop&^sr0&^hdc&^cdrom/ { print $0" "arr[$4] } /^Filesystem/ { mrk=1;next } mrk==1 && /^\// { arr[$1]=$0 }' <<< $(df --exclude={tmpfs,devtmpfs,squashfs,overlay};lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT)
Redirect the two commands back into awk, stripping out any grep and sed processing. We process the df command first and where we find a line beginning with "Filesystem" we set a marker (mrk) to 1 and move to the next line. We then create an array (arr) indexed with the mountpoint and containing the line returned from the df command. We move onto the lsblk command and search for the lines starting with the KNAMEs required. We print the line from the lsblk command and append the value in the arr array indexed by the mount point ($4)
I'm wondering how I can get the value of the second row, 4th column from df ("/"). Here's the output from df:
Filesystem Size Used Avail Use% Mounted on
rootfs 208G 120G 78G 61% /
fakefs 208G 120G 78G 61% /root
fakefs 1.8T 1.3T 552G 70% /home4/user
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/bin
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/etc
fakefs 4.0G 1.3G 2.8G 31% /ramdisk/php
fakefs 208G 120G 78G 61% /var/lib
fakefs 208G 120G 78G 61% /var/lib/mysql
fakefs 208G 120G 78G 61% /var/log
fakefs 208G 120G 78G 61% /var/spool
fakefs 208G 120G 78G 61% /var/run
fakefs 4.0G 361M 3.7G 9% /var/tmp
fakefs 208G 120G 78G 61% /var/cache/man
I'm trying to get the available free space (78GB) using perl which I'm fairly new to. I'm able to get the value using the following linux command but I've heard it's not necessary to use awk in perl at all because perl can do what awk can natively.
df -h | tail -n +2 | sed -n '2p' | awk '{ print $4 }'
I'm stumped. I tried using the Filesys::df module but when I'd print out the available usage percent, it'd give me a different value than what running df from command line does. Help is appreciated.
A little more succinctly:
df -h | perl -wlane 'print $F[3] if $. == 2;'
-w enable warnings
-l add newline to output(and chomps newline from input line)
-a splits the fields on whitespace into the #F array, which you access using the syntax $F[n] (first column is at index position 0)
-n puts the code inside the following loop:
LINE:
while (<>) {
... # code goes here
}
# <> reads lines from STDIN if no filenames are given on the command line
-e execute the string
$. current line number in the file (For the first line, $. is 1)
If you wish to do this all in perl, then:
df -h | perl -e 'while (<stdin>) { if ($. == 2) { #x = split; print $x[3] }}'
This uses perl alone to read the output of df -h and, for the second record ($. == 2) splits the record into fields, based on whitespace, and outputs field 3 (counting from 0).
This seems to work ok too:
df -h | awk 'NR==2 {print $4}'
Get the second line and pint fourth field.
I'm writing script is ksh. Need to find all users who has over N process and echo them in shell.
N reads from ksh.
I know what I should use ps -elf but how parse it, find users with >N process and create array with them. Little troubles with array in ksh. Please help. Maybe simple solutions can help me instead of array creating.
s162103#helios:/home/s162103$ ps -elf
0 S s153308 4804 1 0 40 20 ? 17666 ? 11:03:08 ? 0:00 /usr/lib/gnome-settings-daemon --oa
0 S root 6546 1327 0 40 20 ? 3584 ? 11:14:06 ? 0:00 /usr/dt/bin/dtlogin -daemon -udpPor
0 S webservd 15646 485 0 40 20 ? 2823 ? п╪п╟я─я ? 0:23 /opt/csw/sbin/nginx
0 S s153246 6746 6741 0 40 20 ? 18103 ? 11:14:21 ? 0:00 iiim-panel --disable-crash-dialog
0 S s153246 23512 1 0 40 20 ? 17903 ? 09:34:08 ? 0:00 /usr/bin/metacity --sm-client-id=de
0 S root 933 861 0 40 20 ? 5234 ? 10:26:59 ? 0:00 dtgreet -display :14
...
when i type
ps -elf | awk '{a[$3]++;}END{for(i in a)if (a[i]>N)print i, a[i];}' N=1
s162103#helios:/home/s162103$ ps -elf | awk '{a[$3]++;}END{for(i in a)if (a[i]>N)print i, a[i];}' N=1
root 118
/usr/sadm/lib/smc/bin/smcboot 3
/usr/lib/autofs/automountd 2
/opt/SUNWut/lib/utsessiond 2
nasty 31
dima 22
/opt/oracle/product/Oracle_WT1/ohs/ 7
/usr/lib/ssh/sshd 5
/usr/bin/bash 11
that is not user /usr/sadm/lib/smc/bin/smcboot
there is last field in ps -elf ,not user
Something like this(assuming 3rd field of your ps command gives the user id):
ps -elf |
awk '{a[$3]++;}
END {
for(i in a)
if (a[i]>N)
print i, a[i];
}' N=3
The minimal ps command you want to use here is ps -eo user=. This will just print the username for each process and nothing more. The rest can be done with awk:
ps -eo user= |
awk -v max=3 '{ n[$1]++ }
END {
for (user in n)
if (n[user]>max)
print n[user], user
}'
I recommend to put the count in the first column for readability.
read number
ps -elfo user= | sort | uniq -c | while read count user
do
if (( $count > $number ))
then
echo $user
fi
done
That is best solution and it works!
I'm using Bash on Debian GNU/Linux 6.0. Is it possible to get the file creation date/time? Not the modification date/time.
ls -lh a.txt and stat -c %y a.txt both only give the modification time.
Unfortunately your quest won't be possible in general, as there are only 3 distinct time values stored for each of your files as defined by the POSIX standard (see Base Definitions section 4.8 File Times Update)
Each file has three distinct associated timestamps: the time of last
data access, the time of last data modification, and the time the file
status last changed. These values are returned in the file
characteristics structure struct stat, as described in <sys/stat.h>.
EDIT: As mentioned in the comments below, depending on the filesystem used metadata may contain file creation date. Note however storage of information like that is non standard. Depending on it may lead to portability problems moving to another filesystem, in case the one actually used somehow stores it anyways.
ls -i file #output is for me 68551981
debugfs -R 'stat <68551981>' /dev/sda3 # /dev/sda3 is the disk on which the file exists
#results - crtime value
[root#loft9156 ~]# debugfs -R 'stat <68551981>' /dev/sda3
debugfs 1.41.12 (17-May-2010)
Inode: 68551981 Type: regular Mode: 0644 Flags: 0x80000
Generation: 769802755 Version: 0x00000000:00000001
User: 0 Group: 0 Size: 38973440
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 76128
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x526931d7:1697cce0 -- Thu Oct 24 16:42:31 2013
atime: 0x52691f4d:7694eda4 -- Thu Oct 24 15:23:25 2013
mtime: 0x526931d7:1697cce0 -- Thu Oct 24 16:42:31 2013
**crtime: 0x52691f4d:7694eda4 -- Thu Oct 24 15:23:25 2013**
Size of extra inode fields: 28
EXTENTS:
(0-511): 352633728-352634239, (512-1023): 352634368-352634879, (1024-2047): 288392192-288393215, (2048-4095): 355803136-355805183, (4096-6143): 357941248-357943295, (6144
-9514): 357961728-357965098
mikyra's answer is good. The fact just like what he said.
[jason#rh5 test]$ stat test.txt
File: `test.txt'
Size: 0 Blocks: 8 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 588720 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 500/ jason) Gid: ( 500/ jason)
Access: 2013-03-14 01:58:12.000000000 -0700
Modify: 2013-03-14 01:58:12.000000000 -0700
Change: 2013-03-14 01:58:12.000000000 -0700
if you want to verify, which file was created first, you can structure your file name by appending system date when you create a series of files.
Note that if you've got your filesystem mounted with noatime for performance reasons, then the atime will likely show the creation time. Given that noatime results in a massive performance boost (by removing a disk write for every time a file is read), it may be a sensible configuration option that also gives you the results you want.
Creation date/time is normally not stored. So no, you can't.
You can find creation time - aka birth time - using stat and also match using find.
We have these files showing last modified time:
$ ls -l --time-style=long-iso | sort -k6
total 692
-rwxrwx---+ 1 XXXX XXXX 249159 2013-05-31 14:47 Getting Started.pdf
-rwxrwx---+ 1 XXXX XXXX 275799 2013-12-30 21:12 TheScienceofGettingRich.pdf
-rwxrwx---+ 1 XXXX XXXX 25600 2015-05-07 18:52 Thumbs.db
-rwxrwx---+ 1 XXXX XXXX 148051 2015-05-07 18:55 AsAManThinketh.pdf
To find files created within a certain time frame using find as below.
Clearly, the filesystem knows about the birth time of a file:
$ find -newerBt '2014-06-13' ! -newerBt '2014-06-13 12:16:10' -ls
20547673299906851 148 -rwxrwx--- 1 XXXX XXXX 148051 May 7 18:55 ./AsAManThinketh.pdf
1407374883582246 244 -rwxrwx--- 1 XXXX XXXX 249159 May 31 2013 ./Getting\ Started.pdf
We can confirm this using stat:
$ stat -c "%w %n" * | sort
2014-06-13 12:16:03.873778400 +0100 AsAManThinketh.pdf
2014-06-13 12:16:04.006872500 +0100 Getting Started.pdf
2014-06-13 12:16:29.607075500 +0100 TheScienceofGettingRich.pdf
2015-05-07 18:32:26.938446200 +0100 Thumbs.db
stat man pages explains %w:
%w time of file birth, human-readable; - if unknown
ls -i menus.xml
94490 menus.xml
Here the number 94490 represents inode
Then do a:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-root 4.0G 3.4G 408M 90% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 124M 27M 92M 23% /boot
/dev/mapper/vg-var 7.9G 1.1G 6.5G 15% /var
To find the mounting point of the root "/" filesystem, because the file menus.xml is on '/' that is '/dev/mapper/vg-root'
debugfs -R 'stat <94490>' /dev/mapper/vg-root
The output may be like the one below:
debugfs -R 'stat <94490>' /dev/mapper/vg-root
debugfs 1.41.12 (17-May-2010)
Inode: 94490 Type: regular Mode: 0644 Flags: 0x0
Generation: 2826123170 Version: 0x00000000
User: 0 Group: 0 Size: 4441
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 16
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
atime: 0x5266e47b -- Wed Oct 23 09:47:55 2013
mtime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
Size of extra inode fields: 4
Extended attributes stored in inode body:
selinux = "unconfined_u:object_r:usr_t:s0\000" (31)
BLOCKS:
(0-1):375818-375819
TOTAL: 2
Where you can see the creation time:
ctime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
stat -c %w a.txt
%w returns the file creation(birth) date if it is available, which is rare.
Here's the link
As #mikyra explained, creation date time is not stored anywhere.
All the methods above are nice, but if you want to quickly get only last modify date, you can type:
ls -lit /path
with -t option you list all file in /path odered by last modify date.
If you really want to achieve that you can use a file watcher like inotifywait.
You watch a directory and you save information about file creations in separate file outside that directory.
while true; do
change=$(inotifywait -e close_write,moved_to,create .)
change=${change#./ * }
if [ "$change" = ".*" ]; then ./scriptToStoreInfoAboutFile; fi
done
As no creation time is stored, you can build your own system based on inotify.
Cited from https://unix.stackexchange.com/questions/50177/birth-is-empty-on-ext4/131347#131347 , the following shellscript would work to get creation time:
get_crtime() {
for target in "${#}"; do
inode=$(stat -c %i "${target}")
fs=$(df "${target}" | tail -1 | awk '{print $1}')
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null | grep -oP 'crtime.*--\s*\K.*')
printf "%s\t%s\n" "${target}" "${crtime}"
done
}
even better:
lsct ()
{
debugfs -R 'stat <'`ls -i "$1" | (read a b;echo -n $a)`'>' `df "$1" | (read a; read a b; echo "$a")` 2> /dev/null | grep --color=auto crtime | ( read a b c d;
echo $d )
}
lsct /etc
Wed Jul 20 19:25:48 2016
Another trick to add to your arsenal is the following:
$ grep -r "Copyright" /<path-to-source-files>/src
Generally speaking, if one changes a file they should claim credit in the “Copyright”. Examine the results for dates, file names, contributors and contact email.
example grep result:
/<path>/src/someobject.h: * Copyright 2007-2012 <creator's name> <creator's email>(at)<some URL>>
How i can see memory usage by user in linux centos 6
For example:
USER USAGE
root 40370
admin 247372
user2 30570
user3 967373
This one-liner worked for me on at least four different Linux systems with different distros and versions. It also worked on FreeBSD 10.
ps hax -o rss,user | awk '{a[$2]+=$1;}END{for(i in a)print i" "int(a[i]/1024+0.5);}' | sort -rnk2
About the implementation, there are no shell loop constructs here; this uses an associative array in awk to do the grouping & summation.
Here's sample output from one of my servers that is running a decent sized MySQL, Tomcat, and Apache. Figures are in MB.
mysql 1566
joshua 1186
tomcat 353
root 28
wwwrun 12
vbox 1
messagebus 1
avahi 1
statd 0
nagios 0
Caveat: like most similar solutions, this is only considering the resident set (RSS), so it doesn't count any shared memory segments.
EDIT: A more human-readable version.
echo "USER RSS PROCS" ; echo "-------------------- -------- -----" ; ps hax -o rss,user | awk '{rss[$2]+=$1;procs[$2]+=1;}END{for(user in rss) printf "%-20s %8.0f %5.0f\n", user, rss[user]/1024, procs[user];}' | sort -rnk2
And the output:
USER RSS PROCS
-------------------- -------- -----
mysql 1521 1
joshua 1120 28
tomcat 379 1
root 19 107
wwwrun 10 10
vbox 1 3
statd 1 1
nagios 1 1
messagebus 1 1
avahi 1 1
Per-user memory usage in percent using standard tools:
for _user in $(ps haux | awk '{print $1}' | sort -u)
do
ps haux | awk -v user=${_user} '$1 ~ user { sum += $4} END { print user, sum; }'
done
or for more precision:
TOTAL=$(free | awk '/Mem:/ { print $2 }')
for _user in $(ps haux | awk '{print $1}' | sort -u)
do
ps hux -U ${_user} | awk -v user=${_user} -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done
The first version just sums up the memory percentage for each process as reported by ps. The second version sums up the memory in bytes instead and calculates the total percentage afterwards, thus leading to a higher precision.
If your system supports, try to install and use smem:
smem -u
User Count Swap USS PSS RSS
gdm 1 0 308 323 820
nobody 1 0 912 932 2240
root 76 0 969016 1010829 1347768
or
smem -u -t -k
User Count Swap USS PSS RSS
gdm 1 0 308.0K 323.0K 820.0K
nobody 1 0 892.0K 912.0K 2.2M
root 76 0 937.6M 978.5M 1.3G
ameskaas 46 0 1.2G 1.2G 1.5G
124 0 2.1G 2.2G 2.8G
In Ubuntu, smem can be installed by typing
sudo apt install smem
This will return the total ram usage by users in GBs, reverse sorted
sudo ps --no-headers -eo user,rss | awk '{arr[$1]+=$2}; END {for (i in arr) {print i,arr[i]/1024/1024}}' | sort -nk2 -r
You can use the following Python script to find per-user memory usage using only sys and os module.
import sys
import os
# Get list of all users present in the system
allUsers = os.popen('cut -d: -f1 /etc/passwd').read().split('\n')[:-1]
for users in allUsers:
# Check if the home directory exists for the user
if os.path.exists('/home/' + str(users)):
# Print the current usage of the user
print(os.system('du -sh /home/' + str(users)))