Related
I am attempting to plot a multicolum file using gnuplot script.
I am doing it like
plot "100.dat" u ($1-CONS):($2*$3) w l lt 4 ,
"200.dat" u ($1-CONS):($2*$3) w l lt 2 ,
"300.dat" u ($1-CONS):($2*$3) w l lt 1
where CONS is my variable defined at the top of file.
My set xrange is [-0.2:0.2] while data in the scale is beyond this scale.
What I want to capture is (in loop form for multiple files):
maximum value of above three plots in negative and positive both sides and corresponding value of column 1 in my xrange for both the maximum.
in a shell script I can do it easily but I am facing problem in defining in my gnuscript
my shell script is below
for i in 100.0000 200.0000 200.0000
do
grep $i data.dat > $i.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.2 && $1 >= 0.0) { print }}' > $i.p2.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.p2.dat | awk '{print $2}' > $i.p2Max.dat ; PMAX=$(cat $i.p2Max.dat) ; grep "$PMAX" $i.p2.dat | tail -n 1 >> MAX.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.0 && $1 >= -0.2) { print }}' > $i.mi.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.mi.dat | awk '{print $2}' > $i.mi_Max.dat ; N_MAX=$(cat $i.mi_Max.dat) ; grep "$N_MAX" $i.mi.dat | tail -n 1 >> MAX.dat
done
I am looking for a simple script that can be used in the gnuplot script in loop form so that if I have multiple data file and I need to grep the maximum of a colum two (on both the sides of the zero) then it store the maximum value of column two wrt corresponding value of column 1 separately for negative and positive scale.
I would love to see if this can be done using a loop so that I do not need to write all the lines repetitively.
Your description is a bit confusing to me. My understanding is the following: loop through several files and extract the maxima in the xranges [-0.2:0] and [0:0.2],
respectively.
Test data:
100.dat
-0.17 0.447 0.287
-0.13 0.353 0.936
-0.09 0.476 0.309
-0.05 0.504 0.220
-0.01 0.340 0.564
0.03 0.096 0.947
0.07 0.564 0.885
0.11 0.312 0.957
0.15 0.058 0.347
0.19 0.016 0.923
0.23 0.835 0.461
200.dat
-0.17 0.608 0.875
-0.13 0.266 0.805
-0.09 0.948 0.696
-0.05 0.513 0.800
-0.01 0.736 0.392
0.03 0.318 0.312
0.07 0.708 0.534
0.11 0.246 0.975
0.15 0.198 0.914
0.19 0.174 0.318
0.23 0.727 0.341
300.dat
-0.17 0.527 0.658
-0.13 0.166 0.340
-0.09 0.695 0.031
-0.05 0.623 0.542
-0.01 0.996 0.674
0.03 0.816 0.365
0.07 0.286 0.433
0.11 0.069 0.381
0.15 0.719 0.621
0.19 0.516 0.701
0.23 0.248 0.659
Code:
### loop of files and extracting values
reset session
FILES = "100.dat 200.dat 300.dat"
Count = words(FILES)
CONS = 0.03
# get maxima
array NegMaxX[Count]
array NegMaxY[Count]
array PosMaxX[Count]
array PosMaxY[Count]
do for [i=1:Count] {
stats [-0.2:0] word(FILES,i) u ($1-CONS):($2*$3) nooutput
NegMaxX[i] = STATS_pos_max_y
NegMaxY[i] = STATS_max_y
stats [0:0.2] word(FILES,i) u ($1-CONS):($2*$3) nooutput
PosMaxX[i] = STATS_pos_max_y
PosMaxY[i] = STATS_max_y
}
set xrange[-0.2:0.2]
# set labels
do for [i=1:Count] {
set label i*2-1 at NegMaxX[i], NegMaxY[i] sprintf("%.3f/%.3f",NegMaxX[i],NegMaxY[i])
set label i*2 at PosMaxX[i], PosMaxY[i] sprintf("%.3f/%.3f",PosMaxX[i],PosMaxY[i])
}
plot for [i=1:Count] word(FILES,i) u ($1-CONS):($2*$3) w l lt i ti word(FILES,i), \
### end of code
Result:
I have inherited a server with some performance issues. It is running node js, nginx, basic MEAN stack. (DB on another server though)
Whenever I copied a file (log file with size of around 150MB) or vim a file with that size, the output of "iostat -x 1" will be like below
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 8137.62 0.00 49.50 0.00 29924.75 604.48 17.32 123.54 16.50 81.68
avg-cpu: %user %nice %system %iowait %steal %idle
1.59 0.00 24.34 0.00 0.00 74.07
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 39.39 0.00 36606.06 929.23 2.42 351.64 1.87 7.37
avg-cpu: %user %nice %system %iowait %steal %idle
2.78 0.00 24.44 0.00 0.00 72.78
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The main reason I am bringing this up is because sometimes a simple RESTFUL API that my nodejs is serving will respond slowly (from 10ms to 500ms) and I am not sure what to look for or check what is causing this.
The same codebase copied to another server will run smoothly without issues, the problem mentioned above is the only lead I can find that there might be something wrong with the server but I am not sure what is it.
The codes are like below:
In one file statistic/index.js:
var tracker = function (data) {
piwik.tracker(data);
};
exports.tracker = tracker;
In another file statistic/piwik.js:
exports.tracker = function (data) {
var params = {};
/** Assign params with data - just static string **/
/** API_URL is another machine in same LAN **/
needle.post(API_URL, params, function (err, response, body) {
if (err || (response.statusCode !== 200 && response.statusCode !== 204)) {
util.error(err);
}
});
};
In the file that is calling the above route/getuser.js:
exports.getUser = function (req, res) {
async.auto({
get_user: function (cb) {
/** Read user data from DB **/
cb();
},
record_statistic: ['get_user', function (cb) {
statistic.tracker({ /** Pass static string data **/});
cb();
}]
}, function (err) {
if (err) {
res.json(err);
} else {
res.json();
}
});
};
The reason I am mentioning the above codes is because when I remarked out statistic.tracker({ /** Pass static string data **/}); The function will response within 50ms, but if I include it most of the time it will respond between 100ms - 500ms. I have even put a timestamp check for the "needle" HTTP post, and it respond within 10 - 20ms.
When I copy a file (cp -p x.txt y.txt) especially when it is a > 100MB file, it will also slow down my node js. But even when I am not doing anything on the server besides nginx and node js listening for request the codes below will respond slowly. (If I didn't remark out the code)
I am suspecting IO but where else to check? or what to look out for?
Below are some info about the server:
[ec2-user#tlp-backend logs]$ uname -a
Linux tlp-backend 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user#tlp-backend logs]$ cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: VMware, Model: VMware Virtual S Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
[ec2-user#tlp-backend logs]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30502
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[ec2-user#tlp-backend logs]$ more /proc/meminfo
MemTotal: 3918960 kB
MemFree: 392260 kB
Buffers: 296116 kB
Cached: 1205652 kB
SwapCached: 364 kB
Active: 1725084 kB
Inactive: 1155564 kB
Active(anon): 949492 kB
Inactive(anon): 430528 kB
Active(file): 775592 kB
Inactive(file): 725036 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4095992 kB
SwapFree: 4092872 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 1378528 kB
Mapped: 29860 kB
Shmem: 1140 kB
Slab: 588628 kB
SReclaimable: 461108 kB
SUnreclaim: 127520 kB
KernelStack: 2296 kB
PageTables: 16940 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6055472 kB
Committed_AS: 1829320 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 288456 kB
VmallocChunk: 34359446140 kB
HardwareCorrupted: 0 kB
AnonHugePages: 507904 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10240 kB
DirectMap2M: 4184064 kB
[ec2-user#tlp-backend logs]$ more /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 2
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
top - 18:02:58 up 26 days, 18:49, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 176 total, 1 running, 175 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3918960k total, 3522368k used, 396592k free, 295340k buffers
Swap: 4095992k total, 3120k used, 4092872k free, 1201660k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 19340 1248 1040 S 0.0 0.0 0:03.45 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:02.55 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:01.57 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root RT 0 0 0 0 S 0.0 0.0 0:18.87 migration/1
7 root 20 0 0 0 0 S 0.0 0.0 0:01.34 ksoftirqd/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
9 root RT 0 0 0 0 S 0.0 0.0 0:01.90 migration/2
10 root 20 0 0 0 0 S 0.0 0.0 0:01.30 ksoftirqd/2
11 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
Apparently, the main problem is coming from the code itself. When you are mixing async.auto with needle package, you need to explicitly stating { connection: 'keep-alive' } in HTTP header
See more info here: https://github.com/tomas/needle/issues/148
I have a process that spawns some other processes,
I want to use the time command on a specific process and get the same output as the time command.
Is that possible and how?
I want to use the time command on a specific process and get the same output as the time command.
Probably it is enough just to use pidstat to get user and sys time:
$ pidstat -p 30122 1 4
Linux 2.6.32-431.el6.x86_64 (hostname) 05/15/2014 _x86_64_ (8 CPU)
04:42:28 PM PID %usr %system %guest %CPU CPU Command
04:42:29 PM 30122 706.00 16.00 0.00 722.00 3 has_serverd
04:42:30 PM 30122 714.00 12.00 0.00 726.00 3 has_serverd
04:42:31 PM 30122 714.00 14.00 0.00 728.00 3 has_serverd
04:42:32 PM 30122 708.00 16.00 0.00 724.00 3 has_serverd
Average: 30122 710.50 14.50 0.00 725.00 - has_serverd
If not then according to strace time uses wait4 system call (http://linux.die.net/man/2/wait4) to get information about a process from the kernel. The same info returns getrusage but you cannot call it for an arbitrary process according to its documentation (http://linux.die.net/man/2/getrusage).
So, I do not know any command that will give the same output. However it is feasible to create a bash script that gets PID of the specific process and outputs something like time outpus then
This script does these steps:
1) Get the number of clock ticks per second
getconf CLK_TCK
I assume it is 100 and 1 tick is equal to 10 milliseconds.
2) Then in loop do the same sequence of commands while exists the directory /proc/YOUR-PID:
while [ -e "/proc/YOUR-PID" ];
do
read USER_TIME SYS_TIME REAL_TIME <<< $(cat /proc/PID/stat | awk '{print $14, $15, $22;}')
sleep 0.1
end loop
Some explanation - according to man proc :
user time: ($14) - utime - Amount of time that this process has been scheduled in user mode, measured in clock ticks
sys time: ($15) - stime - Amount of time that this process has been scheduled in kernel mode, measured in clock ticks
starttime ($22) - The time in jiffies the process started after system boot.
3) When the process is finished get finish time
read FINISH_TIME <<< $(cat '/proc/self/stat' | awk '{print $22;}')
And then output:
the real time = ($FINISH_TIME-$REAL_TIME) * 10 - in milliseconds
user time: ($USER_TIME/(getconf CLK_TCK)) * 10 - in milliseconds
sys time: ($SYS_TIME/(getconf CLK_TCK)) * 10 - in milliseconds
I think it should give roughly the same result as time. One possible problem I see is if the process exists for a very short period of time.
This is my implementation of time:
#!/bin/bash
# Uses herestrings
print_res_jeffies()
{
let "TIME_M=$2/60000"
let "TIME_S=($2-$TIME_M*60000)/1000"
let "TIME_MS=$2-$TIME_M*60000-$TIME_S*1000"
printf "%s\t%dm%d.%03dms\n" $1 $TIME_M $TIME_S $TIME_MS
}
print_res_ticks()
{
let "TIME_M=$2/6000"
let "TIME_S=($2-$TIME_M*6000)/100"
let "TIME_MS=($2-$TIME_M*6000-$TIME_S*100)*10"
printf "%s\t%dm%d.%03dms\n" $1 $TIME_M $TIME_S $TIME_MS
}
if [ $(getconf CLK_TCK) != 100 ]; then
exit 1;
fi
if [ $# != 1 ]; then
exit 1;
fi
PROC_DIR="/proc/"$1
if [ ! -e $PROC_DIR ]; then
exit 1
fi
USER_TIME=0
SYS_TIME=0
START_TIME=0
while [ -e $PROC_DIR ]; do
read TMP_USER_TIME TMP_SYS_TIME TMP_START_TIME <<< $(cat $PROC_DIR/stat | awk '{print $14, $15, $22;}')
if [ -e $PROC_DIR ]; then
USER_TIME=$TMP_USER_TIME
SYS_TIME=$TMP_SYS_TIME
START_TIME=$TMP_START_TIME
sleep 0.1
else
break
fi
done
read FINISH_TIME <<< $(cat '/proc/self/stat' | awk '{print $22;}')
let "REAL_TIME=($FINISH_TIME - $START_TIME)*10"
print_res_jeffies 'real' $REAL_TIME
print_res_ticks 'user' $USER_TIME
print_res_ticks 'sys' $SYS_TIME
And this is an example that compares my implementation of time and real time:
>time ./sys_intensive > /dev/null
Alarm clock
real 0m10.004s
user 0m9.883s
sys 0m0.034s
In another terminal window I run my_time.sh and give it PID:
>./my_time.sh `pidof sys_intensive`
real 0m10.010ms
user 0m9.780ms
sys 0m0.030ms
I'm just looking for an easy way to divide a number (or provide other math functions). Let's say I have the following command:
find . -name '*.mp4' | wc -l
How can I take the result of wc -l and divide it by 3?
The examples I've seen don't deal with re-directed out/in.
Using bc:
$ bc -l <<< "scale=2;$(find . -name '*.mp4' | wc -l)/3"
2.33
In contrast, the bash shell only performs integer arithmetic.
Awk is also very powerful:
$ find . -name '*.mp4' | wc -l | awk '{print $1/3}'
2.33333
You don't even need wc if using awk:
$ find . -name '*.mp4' | awk 'END {print NR/3}'
2.33333
Edit 2018-02-22: Adding shell connector
There is more than 1 way:
Depending on precision required and number of calcul to be done! See shell connector further!
Using bc (binary calculator)
find . -type f -name '*.mp4' -printf \\n | wc -l | xargs printf "%d/3\n" | bc -l
6243.33333333333333333333
or
echo $(find . -name '*.mp4' -printf \\n | wc -l)/3|bc -l
6243.33333333333333333333
or using bash, result in integer only:
echo $(($(find . -name '*.mp4' -printf \\n| wc -l)/3))
6243
Using bash interger builtin math processor
res=000$((($(find . -type f -name '*.mp4' -printf "1+")0)*1000/3))
printf -v res "%.2f" ${res:0:${#res}-3}.${res:${#res}-3}
echo $res
6243.33
Pure bash
With recent 64bits bash, you could even use #glennjackman's ideas of using globstar, but computing pseudo floating could be done by:
shopt -s globstar
files=(**/*.mp4)
shopt -u globstar
res=$[${#files[*]}000/3]
printf -v res "%.2f" ${res:0:${#res}-3}.${res:${#res}-3}
echo $res
6243.33
There is no fork and $res contain a two digit rounded floating value.
Nota: Care about symlinks when using globstar and **!
Introducing shell connector
If you plan to do a lot of calculs, require high precision and use bash, you could use long running bc sub process:
mkfifo /tmp/mybcfifo
exec 5> >(exec bc -l >/tmp/mybcfifo)
exec 6</tmp/mybcfifo
rm /tmp/mybcfifo
then now:
echo >&5 '12/34'
read -u 6 result
echo $result
.35294117647058823529
This subprocess stay open and useable:
ps --sid $(ps ho sid $$) fw
PID TTY STAT TIME COMMAND
18027 pts/9 Ss 0:00 bash
18258 pts/9 S 0:00 \_ bc -l
18789 pts/9 R+ 0:00 \_ ps --sid 18027 fw
Computing $PI:
echo >&5 '4*a(1)'
read -u 6 PI
echo $PI
3.14159265358979323844
To terminate sub process:
exec 6<&-
exec 5>&-
Little demo, about The best way to divide in bash using pipes!
Computing range {1..157} / 42 ( I will let you google for answer to the ultimate question of life, the universe, and everything ;)
... and print 13 result by lines in order to reduce output:
printf -v form "%s" "%5.3f "{,}{,}{,,};form+="%5.3f\n";
By regular way
testBc(){
for ((i=1; i<157; i++)) ;do
echo $(bc -l <<<"$i/42");
done
}
By using long running bc sub process:
testLongBc(){
mkfifo /tmp/mybcfifo;
exec 5> >(exec bc -l >/tmp/mybcfifo);
exec 6< /tmp/mybcfifo;
rm /tmp/mybcfifo;
for ((i=1; i<157; i++)) ;do
echo "$i/42" 1>&5;
read -u 6 result;
echo $result;
done;
exec 6>&-;
exec 5>&-
}
Let's see without:
time printf "$form" $(testBc)
0.024 0.048 0.071 0.095 0.119 0.143 0.167 0.190 0.214 0.238 0.262 0.286 0.310
0.333 0.357 0.381 0.405 0.429 0.452 0.476 0.500 0.524 0.548 0.571 0.595 0.619
0.643 0.667 0.690 0.714 0.738 0.762 0.786 0.810 0.833 0.857 0.881 0.905 0.929
0.952 0.976 1.000 1.024 1.048 1.071 1.095 1.119 1.143 1.167 1.190 1.214 1.238
1.262 1.286 1.310 1.333 1.357 1.381 1.405 1.429 1.452 1.476 1.500 1.524 1.548
1.571 1.595 1.619 1.643 1.667 1.690 1.714 1.738 1.762 1.786 1.810 1.833 1.857
1.881 1.905 1.929 1.952 1.976 2.000 2.024 2.048 2.071 2.095 2.119 2.143 2.167
2.190 2.214 2.238 2.262 2.286 2.310 2.333 2.357 2.381 2.405 2.429 2.452 2.476
2.500 2.524 2.548 2.571 2.595 2.619 2.643 2.667 2.690 2.714 2.738 2.762 2.786
2.810 2.833 2.857 2.881 2.905 2.929 2.952 2.976 3.000 3.024 3.048 3.071 3.095
3.119 3.143 3.167 3.190 3.214 3.238 3.262 3.286 3.310 3.333 3.357 3.381 3.405
3.429 3.452 3.476 3.500 3.524 3.548 3.571 3.595 3.619 3.643 3.667 3.690 3.714
real 0m10.113s
user 0m0.900s
sys 0m1.290s
Wow! Ten seconds on my raspberry-pi!!
Then with:
time printf "$form" $(testLongBc)
0.024 0.048 0.071 0.095 0.119 0.143 0.167 0.190 0.214 0.238 0.262 0.286 0.310
0.333 0.357 0.381 0.405 0.429 0.452 0.476 0.500 0.524 0.548 0.571 0.595 0.619
0.643 0.667 0.690 0.714 0.738 0.762 0.786 0.810 0.833 0.857 0.881 0.905 0.929
0.952 0.976 1.000 1.024 1.048 1.071 1.095 1.119 1.143 1.167 1.190 1.214 1.238
1.262 1.286 1.310 1.333 1.357 1.381 1.405 1.429 1.452 1.476 1.500 1.524 1.548
1.571 1.595 1.619 1.643 1.667 1.690 1.714 1.738 1.762 1.786 1.810 1.833 1.857
1.881 1.905 1.929 1.952 1.976 2.000 2.024 2.048 2.071 2.095 2.119 2.143 2.167
2.190 2.214 2.238 2.262 2.286 2.310 2.333 2.357 2.381 2.405 2.429 2.452 2.476
2.500 2.524 2.548 2.571 2.595 2.619 2.643 2.667 2.690 2.714 2.738 2.762 2.786
2.810 2.833 2.857 2.881 2.905 2.929 2.952 2.976 3.000 3.024 3.048 3.071 3.095
3.119 3.143 3.167 3.190 3.214 3.238 3.262 3.286 3.310 3.333 3.357 3.381 3.405
3.429 3.452 3.476 3.500 3.524 3.548 3.571 3.595 3.619 3.643 3.667 3.690 3.714
real 0m0.670s
user 0m0.190s
sys 0m0.070s
Less than one second!!
Hopefully, results are same, but execution time is very different!
My shell connector
I've published a connector function: Connector-bash on GitHub.com
and shell_connector.sh on my own site.
source shell_connector.sh
newConnector /usr/bin/bc -l 0 0
myBc 1764/42 result
echo $result
42.00000000000000000000
find . -name '*.mp4' | wc -l | xargs -I{} expr {} / 2
Best used if you have multiple outputs you'd like to pipe through xargs. Use{} as a placeholder for the expression term.
Depending on your bash version, you don't even need find for this simple task:
shopt -s nullglob globstar
files=( **/*.mp4 )
dc -e "3 k ${#files[#]} 3 / p"
This method will correctly handle the bizarre edgecase of filenames containing newlines.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Tools such as MRTG provide network throughput / bandwidth graphs for the current network utilisation on specific interfaces, such as eth0. How can I return that information at the command line on Linux/UNIX?
Preferably this would be without installing anything other than what is available on the system as standard.
iftop does for network usage what top(1) does for CPU usage -- http://www.ex-parrot.com/~pdw/iftop/
I don't know how "standard" iftop is, but I was able to install it with yum install iftop on Fedora.
Got sar? Likely yes if youre using RHEL/CentOS.
No need for priv, dorky binaries, hacky scripts, libpcap, etc. Win.
$ sar -n DEV 1 3
Linux 2.6.18-194.el5 (localhost.localdomain) 10/27/2010
02:40:56 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:40:57 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:57 PM eth0 10700.00 1705.05 15860765.66 124250.51 0.00 0.00 0.00
02:40:57 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:57 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:40:58 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:58 PM eth0 8051.00 1438.00 11849206.00 105356.00 0.00 0.00 0.00
02:40:58 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:58 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:40:59 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:59 PM eth0 6093.00 1135.00 8970988.00 82942.00 0.00 0.00 0.00
02:40:59 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 8273.24 1425.08 12214833.44 104115.72 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
I wrote this dumb script a long time ago, it depends on nothing but Perl and Linux≥2.6:
#!/usr/bin/perl
use strict;
use warnings;
use POSIX qw(strftime);
use Time::HiRes qw(gettimeofday usleep);
my $dev = #ARGV ? shift : 'eth0';
my $dir = "/sys/class/net/$dev/statistics";
my %stats = do {
opendir +(my $dh), $dir;
local #_ = readdir $dh;
closedir $dh;
map +($_, []), grep !/^\.\.?$/, #_;
};
if (-t STDOUT) {
while (1) {
print "\033[H\033[J", run();
my ($time, $us) = gettimeofday();
my ($sec, $min, $hour) = localtime $time;
{
local $| = 1;
printf '%-31.31s: %02d:%02d:%02d.%06d%8s%8s%8s%8s',
$dev, $hour, $min, $sec, $us, qw(1s 5s 15s 60s)
}
usleep($us ? 1000000 - $us : 1000000);
}
}
else {print run()}
sub run {
map {
chomp (my ($stat) = slurp("$dir/$_"));
my $line = sprintf '%-31.31s:%16.16s', $_, $stat;
$line .= sprintf '%8.8s', int (($stat - $stats{$_}->[0]) / 1)
if #{$stats{$_}} > 0;
$line .= sprintf '%8.8s', int (($stat - $stats{$_}->[4]) / 5)
if #{$stats{$_}} > 4;
$line .= sprintf '%8.8s', int (($stat - $stats{$_}->[14]) / 15)
if #{$stats{$_}} > 14;
$line .= sprintf '%8.8s', int (($stat - $stats{$_}->[59]) / 60)
if #{$stats{$_}} > 59;
unshift #{$stats{$_}}, $stat;
pop #{$stats{$_}} if #{$stats{$_}} > 60;
"$line\n";
} sort keys %stats;
}
sub slurp {
local #ARGV = #_;
local #_ = <>;
#_;
}
It just reads from /sys/class/net/$dev/statistics every second, and prints out the current numbers and the average rate of change:
$ ./net_stats.pl eth0
rx_bytes : 74457040115259 4369093 4797875 4206554 364088
rx_packets : 91215713193 23120 23502 23234 17616
...
tx_bytes : 90798990376725 8117924 7047762 7472650 319330
tx_packets : 93139479736 23401 22953 23216 23171
...
eth0 : 15:22:09.002216 1s 5s 15s 60s
^ current reading ^-------- averages ---------^
nload is a great tool for monitoring bandwidth in real time and easily installed in Ubuntu or Debian with sudo apt-get install nload.
Device eth0 [10.10.10.5] (1/2):
=====================================================================================
Incoming:
. ...|
# ####|
.. |#| ... #####. .. Curr: 2.07 MBit/s
###.### #### #######|. . ## | Avg: 1.41 MBit/s
########|#########################. ### Min: 1.12 kBit/s
........ ################################### .### Max: 4.49 MBit/s
.##########. |###################################|##### Ttl: 1.94 GByte
Outgoing:
########## ########### ###########################
########## ########### ###########################
##########. ########### .###########################
########### ########### #############################
########### ###########..#############################
############ ##########################################
############ ##########################################
############ ########################################## Curr: 63.88 MBit/s
############ ########################################## Avg: 32.04 MBit/s
############ ########################################## Min: 0.00 Bit/s
############ ########################################## Max: 93.23 MBit/s
############## ########################################## Ttl: 2.49 GByte
Another excellent tool is iftop, also easily apt-get'able:
191Mb 381Mb 572Mb 763Mb 954Mb
└────────────┴──────────┴─────────────────────┴───────────┴──────────────────────
box4.local => box-2.local 91.0Mb 27.0Mb 15.1Mb
<= 1.59Mb 761kb 452kb
box4.local => box.local 560b 26.8kb 27.7kb
<= 880b 31.3kb 32.1kb
box4.local => userify.com 0b 11.4kb 8.01kb
<= 1.17kb 2.39kb 1.75kb
box4.local => b.resolvers.Level3.net 0b 58b 168b
<= 0b 83b 288b
box4.local => stackoverflow.com 0b 42b 21b
<= 0b 42b 21b
box4.local => 224.0.0.251 0b 0b 179b
<= 0b 0b 0b
224.0.0.251 => box-2.local 0b 0b 0b
<= 0b 0b 36b
224.0.0.251 => box.local 0b 0b 0b
<= 0b 0b 35b
─────────────────────────────────────────────────────────────────────────────────
TX: cum: 37.9MB peak: 91.0Mb rates: 91.0Mb 27.1Mb 15.2Mb
RX: 1.19MB 1.89Mb 1.59Mb 795kb 486kb
TOTAL: 39.1MB 92.6Mb 92.6Mb 27.9Mb 15.6Mb
Don't forget about the classic and powerful sar and netstat utilities on older *nix!
You could parse /proc/net/dev.
dstat - Combines vmstat, iostat, ifstat, netstat information and more
iftop - Amazing network bandwidth utility to analyse what is really happening on your eth
netio - Measures the net throughput of a network via TCP/IP
inq - CLI troubleshooting utility that displays info on storage, typically Symmetrix. By default, INQ returns the device name, Symmetrix ID, Symmetrix LUN, and capacity.
send_arp - Sends out an arp broadcast on the specified network device (defaults to eth0), reporting an old and new IP address mapping to a MAC address.
EtherApe - is a graphical network monitor for Unix modeled after etherman. Featuring link layer, IP and TCP modes, it displays network activity graphically.
iptraf - An IP traffic monitor that shows information on the IP traffic passing over your network.
More details:
http://felipeferreira.net/?p=1194
You can parse the output of ifconfig
I got another quick'n'dirty bash script for that:
#!/bin/bash
IF=$1
if [ -z "$IF" ]; then
IF=`ls -1 /sys/class/net/ | head -1`
fi
RXPREV=-1
TXPREV=-1
echo "Listening $IF..."
while [ 1 == 1 ] ; do
RX=`cat /sys/class/net/${IF}/statistics/rx_bytes`
TX=`cat /sys/class/net/${IF}/statistics/tx_bytes`
if [ $RXPREV -ne -1 ] ; then
let BWRX=$RX-$RXPREV
let BWTX=$TX-$TXPREV
echo "Received: $BWRX B/s Sent: $BWTX B/s"
fi
RXPREV=$RX
TXPREV=$TX
sleep 1
done
It's considering that sleep 1 will actually last exactly one second, which is not true, but good enough for a rough bandwidth assessment.
Thanks to #ephemient for the /sys/class/net/<interface>! :)
Besides iftop and iptraf, also check:
bwm-ng (Bandwidth Monitor Next Generation)
and/or
cbm (Color Bandwidth Meter)
ref: http://www.powercram.com/2010/01/bandwidth-monitoring-tools-for-ubuntu.html
If you want just to get the value, you can use simple shell oneliner like this:
S=10; F=/sys/class/net/eth0/statistics/rx_bytes; X=`cat $F`; sleep $S; Y=`cat $F`; BPS="$(((Y-X)/S))"; echo $BPS
It will show you the average "received bytes per second" for period of 10 seconds (you can change period by changing S=10 parameter, and you can measure transmitted BPS instead of received BPS by using tx_bytes instead of rx_bytes). Don't forget to change eth0 to network device you want to monitor.
Of course, you are not limited to displaying the average rate (as mentioned in other answers, there are other tools that will show you much nicer output), but this solution is easily scriptable to do other things.
For example, the following shell script (split into multiple lines for readability) will execute offlineimap process only when 5-minute average transmit speed drops below 10kBPS (presumably, when some other bandwidth-consuming process finishes):
#!/bin/sh
S=300; F=/sys/class/net/eth0/statistics/tx_bytes
BPS=999999
while [ $BPS -gt 10000 ]
do
X=`cat $F`; sleep $S; Y=`cat $F`; BPS="$(((Y-X)/S))";
echo BPS is currently $BPS
done
offlineimap
Note that /sys/class/... is Linux specific (which is ok as submitter did choose linux tag), and needs non-archaic kernel. Shell code itself is /bin/sh compatible (so not only bash, but dash and other /bin/sh implementations will work) and /bin/sh is something that is really always installed.
I like iptraf but you probably have to install it and it seems to not being maintained actively anymore.
I find dstat to be quite good. Has to be installed though. Gives you way more information than you need. Netstat will give you packet rates but not bandwith also. netstat -s
You can use iperf to benchmark network performance (maximum possible throughput).
See following links for details:
http://en.wikipedia.org/wiki/Iperf
https://iperf.fr/
https://code.google.com/p/iperf/
I couldn't get the parse ifconfig script to work for me on an AMI so got this to work measuring received traffic averaged over 10 seconds
date && rxstart=`ifconfig eth0 | grep bytes | awk '{print $2}' | cut -d : -f 2` && sleep 10 && rxend=`ifconfig eth0 | grep bytes | awk '{print $2}' | cut -d : -f 2` && difference=`expr $rxend - $rxstart` && echo "Received `expr $difference / 10` bytes per sec"
Sorry, it's ever so cheap and nasty but it worked!
ifconfig -a
ip -d link
ls -l /sys/class/net/ (physical and virtual devices)
route -n
If you want the output of (ifconfig -a) in json format you can use this (python)