I'm currently trying to measure the time a program needs to finish when I start it 8 times at the same time.
Now I would really like to write a bash or something that starts the program several times with different parameters and measures the time until all of them are finished.
I think I would manage to start my program 8 times by simply using & at the end but then I don't know how to know when they stop.
You can use wait to wait for background jobs to finish.
#!/bin/sh
program &
program &
wait
will wait until both instances of program exit.
use jobs to see whats still running, if you need 8 you can do
if jobs | wc -l < 8 then command &
Not working code but you get the idea
You can use the time command to measure the time consumption of a program, so perhaps something like
#!/bin/bash
time yourprogram withoneargument
time yourprogram with three arguments
...etc
Cheers,
Mr. Bystrup supplies the time command, which will time the execution of your programs. Mr. Politowski and user2814958 supply the & operator, which will run programs in the background, allowing you to start them at the same time. If you combine these, you're most of the way there, except the output from time for the different commands will be jumbled, and it will be hard to tell which output pertains to which command.
One way to overcome this issue is to shunt the output into different files:
/usr/bin/time program1 2>/tmp/program1.time &
/usr/bin/time program2 2>/tmp/program2.time &
Note that I'm redirecting the standard error (file descriptor 2) into the files; time writes its output on the standard error instead of the standard output. Also note my use of the full path to time. Some shells have a built-in time command that behaves differently, such as writing output directly to the terminal instead of standard error.
Related
I'm writing a Bash script to evaluate time/CPU/memory performances of commands given as input to the script.
I implemented the evaluation of time by using date command, but I have issues to evaluate CPU and memory performance related to the single command. I know that I can use top command but it shows me only runtime processes.
My issue is that if I run the script by giving as input the command, I don't know previously the assigned PID to this command, and if I want to evaluate an instant command as whoami, I cannot find it when I use top command, even if I use pipe on them.
I think for commands that needs more time, I would like to calculate an average, but for commands like whoami, ls or similar instant commands, I don't have idea how I can get the CPU and memory performance for that specific instant of time.
Thank you in advance!
I have a program that i need to collect 300 pieces of data from, but to manually do the collecting i have to run the program on my ubuntu virtual machine and record the data on excel. It takes a long time to do this whole process. I was wondering if there was a command in linux that i could use to call commands make and to kill me program.
I search watch and tried it but it doesnt work for me:
watch -n 20 make play
where make play runs my program
Yet this doesnt fo everything i want to do. I want to do this every 20 seconds so i have enough time to write my data to my excel file
1. make play (run my program so it prints what i need to record)
2. kill my program
Is there a command for this?
I think you should rethink what you are doing - I can't think of a setting where running and killing a program every 20 seconds makes any sense.
That being said, the standard way to run programs periodically in linux is a cron job. Cron has a 1 minute minimum though, so you would have to write a script that starts 3 instances of your program with 20 second delay, and run this script with cron every minute. You can combine this with the timeout utility, which will kill your program if it is still running after a given time. A quick google search should provide you with further details.
I think you Could use crontab, man crontab to get the manual of crontab. However, you may not be able to run and kill every 20s, at least every 1 min. Hope It could help.
I've been troubleshooting this issue for about a week and I am nowhere, so I wanted to reach out for some help.
I have a perl script that I execute via command like, usually in a manner of
nohup ./script.pl --param arg --param2 arg2 &
I usually have about ten of these running at once to process the same type of data from different sources (that is specified through parameters). The script works fine and I can see logs for everything in nohup.out and monitor status via ps output. This script also uses a sql database to track status of various tasks, so I can track finishes of certain sources.
However, that was too much work, so I wrote a wrapper script to execute the script automatically and that is where I am running into problems. I want something exactly the same as I have, but automatic.
The getwork.pl script runs ps and parses output to find out how many other processes are running, if it is below the configured thresh it will query the database for the most out of date source and kick off the script.
The problem is that the kicked off jobs aren't running properly, sometimes they terminate without any error messages and sometimes they just hang and sit idle until I kill them.
The getwork script queries sql and gets the entire execution command via sql concatanation, so in the sql query I am doing something like CONCAT('nohup ./script.pl --arg ',param1,' --arg2 ',param2,' &') to get the command string.
I've tried everything to get these kicked off, I've tried using system (), but again, some jobs kick off, some don't, sometimes it gets stuck, sometimes jobs start and then die within a minute. If I take the exact command I used to start the job and run it in bash, it works fine.
I've tried to also open a pipe to the command like
open my $ca, "| $command" or die ($!);
print $ca $command;
close $ca;
That works just about as well as everything else I've tried. The getwork script used to be executed through cron every 30 minutes, but I scrapped that because I needed another shell wrapper script, so now there is an infinite look in the get work script that executes a function every 30 minutes.
I've also tried many variations of the execution command, including redirecting output to different files, etc... nothing seems to be consistent. Any help would be much appreciated, because I am truly stuck here....
EDIT:
Also, I've tried to add separate logging within each script, it would start a new log file with it's PID ($$). There was a bunch of weirdness there too, all log files would get created, but then some of the processes would be running and writing to the file, others would just have an empty text file and some would just have one or two log entries. Sometimes the process would still be running and just not doing anything, other times it would die with nothing in the log. Me, running the command in shell directly always works though.
Thanks in advance
You need a kind of job managing framework.
One of the bigest one is Gearman: http://www.slideshare.net/andy.sh/gearman-and-perl
I want to know the good practice of performing a series of commands simultaneously in UNIX/Linux. Suppose that I have a program, program_a, which requires one parameter. I have stored parameters line by line in a file. So I wrote:
while read line
do
./program_a line > $line.log 2>&1
done < parameter_file
The problem is that execution of program_a takes long time. Because each executions of program_a for each parameter is independent, So I think these executions can be run simultaneously. I don't know if it regards to multithreading or other technique. The following is my thought. Use & to run each executions on the background.
while read line
do
./program_a line $line.log 2>&1 &
done < parameter_file
Is there any better way of launching multiple tasks?
Did you know that xargs can launch tasks in parallel? Check out -P -n parameters!
An example:
xargs -P 4 -n 1 ./program_a < parameter_file
That will start up to 4 (P=4) program_a instances for processing each line (n=1). You'll probably have to wrap program_a within a shell script or something so that child processes stdout & stderr can be redirected appropriately.
How this is better than putting processes to backgroud: Suppose you have 1000 lines in the input file, obviously you wouldn't want 1000 processes to be launched. Xargs allows you to look at it as a queue, with P workers each consuming and processing n items from it.
With GNU Parallel you can get a logfile for each parameter and run one job per CPU core:
parallel --results logdir ./program_a :::: parameter_file
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). You command line
with love you for it.
So I've made a small c++ binary to connect to do a command on a server to stress test it, so i started working on the following shell script:
#!/bin/bash
for (( i = 0 ; i <= 15; i++ ))
do
./mycppbinary test 1 &
done
Now, I also happen to want to time how long all the processes take to execute. I suppose I'll have to do a time command on each of these processes?
Is it possible to join those processes, as if they're a thread?
You don't join them, you wait on them. At lest in bash, and probably other shells with job control.
You can use the bash fg command to bring the last background process back into the foreground. Do it in another loop to catch them all, though some may complete before this causing you to get an error about no such process. You're not joining processes, they aren't threads, they each have their own pid and unique memory space.
1st, make the script last the same as all its children
The script you propose will die before the processes finish, due to the fact that you are launching them on the background. If you don't want this to happen, you can do as many waits as needed (as Keith suggested).
2nd, time the script
Then, you can time your script and that will give you the total execution time, as you requested.
You can time your shell script, that will give you the total execution time.