I have multiple servers of Linux, where I need to test the performance of my program, here I want to output the system state when running programs. In script of linux, I use following to output:
top -b -d 5 > System.txt
iostat -d 8 > IO.txt
But unfortunately, only the system.txt can be produced, but there is no the IO.txt file, so that need to add some thing in script to make these two file exist together?
Use iostat command like this :-
iostat -d 8 1 > IO.txt.
In this 1 means the limit, how many time do you want to run the command get the result.
Related
I'm puzzled by this problem I'm having on Ubuntu 20.04 where cron is able to run a bash script but the overall outcome is different then when using the shell command.
I've look through all questions I could in here and on Google but couldn't find anyone that had the same problem.
Background:
I'm using Pushgateway to store metrics I'm generating through a bash script, and afterwards it's being imported automatically to Prometheus.
The end goal is to export a list of running processes, their CPU%, Mem% etc, similar to top command.
This is the bash script:
#!/bin/bash
z=$(top -n 1 -bi)
while read -r z
do
var=$var$(awk 'FNR>7{print "cpu_usage{process=\""$12"\", pid=\""$1"\"}", $9z} FNR>7{print "memory_usage{process=\""$12"\", pid=\""$1"\"}", $10z}')
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine
I used to have a version that used ps aux but then I found out that it only shows the average CPU% per process.
As you can see, the command I'm running is top -n 1 -bi which gives me a snapshot of active processes and their metrcis.
I'm using awk to format the data, and FNR>7 because I need to ignore the first 7 lines which is the summery presented by top.
The bash scrip is registered on /bin, /usr/bin and /usr/local/bin.
When checking http://localhost:9091/metrics, which is supposed to show me the information gathered, I'm getting this some of information when running the scrip using shell:
cpu_usage{instance="machine",job="top",pid="114468",process="php-fpm74"} 17.6
cpu_usage{instance="machine",job="top",pid="114483",process="php-fpm74"} 11.8
cpu_usage{instance="machine",job="top",pid="126305",process="ffmpeg"} 64.7
And this is the same information when cron is running the same script:
cpu_usage{instance="machine",job="top",pid="114483",process="php-fpm+"} 5
cpu_usage{instance="machine",job="top",pid="126305",process="ffmpeg"} 60
cpu_usage{instance="machine",job="top",pid="128777",process="php"} 15
So, for some reason, when I run it from cron it cuts the process name after 7 places.
I initially though it was related to the FNR>7 but even after changing it to 8 or 9 (and using exec bash to re-register the command) it gives the same results, also when I run it manually it works just fine.
Any help would be appreciated!!
I have tool trimmings content from file. I need to use it on 33 files. One file is processing 2 hours.
I want to try run it 33 Times on same tame coz one instance used one core, on my machine I have 128 cores.
So I wrote script:
#!/bin/bash
FILES=/home/ab/raw/*
for f in $FILES
do
base = ${f##*/}
nohup /home/ab/trimmer -a /home/ab/trimmer/adapters.fa -o "OUT$base" $f
done
So main line:
I run trimmer, -a it's file with patterns to delete, -o it's new file as outpu (OUT+basename) and last $f is processing file.
My intention was that the script would run separate tasks for each file.
But unfortunately, after run it, only one nohup will be launched. In htop it's still only one core working at 100%.
How can I fix it?
I am executing the "wget -o " and because the output is bigger than expected, it is split in more than one file. Is there a way to get only one file? If this is possible I would prefer to use only the command wget.
The command wget that I am executing is:
$ wget -o neighborhoods.json https://raw.githubusercontent.com/mongodb/docs-assets/geospatial/neighborhoods.json
And the multiple output is:
-rw-rw-r-- 1 ubuntu ubuntu 6652 Mar 4 01:15 neighborhoods.json
-rw-rw-r-- 1 ubuntu ubuntu 4137081 Mar 4 01:15 neighborhoods.json.1
Look well at the wget output, you will see what it is/will be doing. wget does not split files if they are long; instead, it avoids to overwrite files, if they exist (creating a new file instead of touching the already existing one).
Delete the two files neighborXXX, and start wget again; be sure it finishes without problems: it will write (create) the single file you asked for. If it is interrupted, and you restart it, it will create a new file (appending .1 and so on).
You can pass it the option -c to tell it to continue a broken download, if it was interrupted - most of the times it works well (not always tough).
I am new to Perl. I am working on a web UI where I have to give CPU and memory monitoring data, so I am using top and gnuplot command. I am able to do it through my terminal. but when I am executing same commands with a perl script its not working. The problem I am having is that whenever I am executing top command in my terminal than I have to wait for few minutes and then I have to plot it using GNUPLOT but when I am doing the same work using system command in perl I am unable to give that delay.
Here is what I am doing
system "top -p 1758 -b -d5 | tee -a stats.log";
system "./top_stats.sh -f stats.log"
Here 1758 is the app ID whose top data I have to monitor and stats.log file is the one where I am saving logs and then using this stats.log file as input to top_stats.sh script I am plotting graphs. This top_stats.sh script takes the log file and uses gnuplot to plot data
Now the problem is whenever I am executing this first system command in terminal I have to wait for some time say 2 to 3 minutes to have ample number of data points and then I have to press Ctrl+C to come out of top command and then run the script. but here as soon as this first system command is encountered it is executed and then next command is executed without any delay so I am not getting any data points to plot the graph. Is there any way I can execute my first command and wait for 3 minutes without coming out of system command and then execute the next command.??
Is there any way I can execute my first command and wait for 3 minutes without coming out of system command and then execute the next command.??
Why not specify the number of iteration for top. So your command will rougly run for (N-1)*5 seconsds. 37 iterations with an interval of 5 seconds are going to take 36*5 seconds:
system "top -p 1758 -b -d5 -n37 | tee -a stats.log && ./top_stats.sh -f stats.log";
I have seven licenses of a particular software. Therefore, I want to start 7 jobs simultaneously. I can do that using '&'. Now, 'wait' command waits till the end of all of those 7 processes to be finished to spawn the next 7. Now, I would like to write the shell script where after I start the first seven, as and when a job gets completed I would like to start another. This is because some of those 7 jobs might take very long while some others get over really quickly. I don't want to waste time waiting for all of them to finish. Is there a way to do this in linux? Could you please help me?
Thanks.
GNU parallel is the way to go. It is designed for launching multiples instances of a same command, each with a different argument retrieved either from stdin or an external file.
Let's say your licensed script is called myScript, each instance having the same options --arg1 --arg2 and taking a variable parameter --argVariable for each instance spawned, those parameters being stored in file myParameters :
cat myParameters | parallel -halt 1 --jobs 7 ./myScript --arg1 --argVariable {} --arg2
Explanations :
-halt 1 tells parallel to halt all jobs if one fails
--jobs 7 will launch 7 instances of myScript
On a debian-based linux system, you can install parallel using :
sudo apt-get install parallel
As a bonus, if your licenses allow it, you can even tell parallel to launch these 7 instances amongst multiple computers.
You could check how many are currently running and start more if you have less than 7:
while true; do
if [ "`ps ax -o comm | grep process-name | wc -l`" -lt 7 ]; then
process-name &
fi
sleep 1
done
Write two scripts. One which restarts a job everytime it is finished and one that starts 7 times the first script.
Like:
script1:
./script2 job1
...
./script2 job7
and
script2:
while(...)
./jobX
I found a fairly good solution using make, which is a part of the standard distributions. See here