I am trying to make a bash shell script that launches some jobs on a queuing system. After a job is launched, the launch command prints the job-id to the stdout, which I would like to 'trap' and then use in the next command. The job-id digits are the only digits in the stdout message.
#!/bin/bash
./some_function
>>> this is some stdout text and the job number is 1234...
and then I would like to get to:
echo $job_id
>>> 1234
My current method is using a tee command to pipe the original command's stdout to a tmp.txt file and then making the variable by grepping that file with a regex filter...something like:
echo 'pretend this is some dummy output from a function 1234' 2>&1 | tee tmp.txt
job_id=`cat tmp.txt | grep -o '[0-9]'`
echo $job_id
>>> pretend this is some dummy output from a function 1234
>>> 1 2 3 4
...but I get the feeling this is not really the most elegant or 'standard' way of doing this. What is the better way to do this?
And for bonus points, how do I remove the spaces from the grep+regex output?
You can use grep -o when you call your script:
jobid=$(echo 'pretend this is some dummy output from a function 1234' 2>&1 |
tee tmp.txt | grep -Eo '[0-9]+$')
echo "$jobid"
1234
Something like this should work:
$ JOBID=`./some_function | sed 's/[^0-9]*\([0-9]*\)[^0-9]*/\1/'`
$ echo $JOBID
1234
Related
I've got a long running process and I want to capture a tiny bit of data from the big swath of output.
I can do this by piping it through grep, but then I can't watch it spew out all the other info.
I basically want grep to save what it finds into a variable and leave stdout alone. How can I do that?
With tee, process substitution, and I/O redirection:
{ var=$(cmd | tee >(grep regexp) >&3); } 3>&1
There isn't any way to use a variable like that. It sounds like you want to write your own filter:
./some_long_program | tee >(my_do_stuff.sh)
Then, my_do_stuff.sh is:
#!/bin/bash
while read line; do
echo "$line" | grep -q 'pattern' || continue
VARIABLE=$line # Now it's in a variable
done
If you have the storage space, this is probably more like what you want:
./some_long_program | tee /tmp/slp.log
Then, simply:
grep 'pattern' /tmp/slp.log && VARIABLE=true
or:
VARIABLE=$(grep 'pattern' /tmp/slp.log)
This will let you run the grep at any time. I don't think the variable really adds anything though.
EDIT:
#mpen Based on your last answer above, it sounds like you want to use xargs. Try:
(echo 1 ; sleep 5 ; echo 2) | xargs -L1 echo got:
The -L1 will run the command for every instance found, otherwise it grabs lots of stdin and passes them all (up to some maximum) to the command at once. You'll still want to use this via tee if you want to see all the command output as well:
./some_long_program | tee >(grep 'pattern' | xargs -L1 ./my_do_stuff.sh)
I think this can be done a little more simply if we tee to /dev/tty:
❯ BAR=$(echo foobar | tee /dev/tty | grep -Pom1 b..); echo "[$BAR]"
foobar
[bar]
I have the following simple bash script which takes input from stdin and prints the third line given as input.
#!/bin/bash
var=$(cat)
echo $var | head -n 3 | tail -n 1
The problem with this script is that it prints all the lines but here is the funny part, when I type the commands individually on the command line I am getting the expected result i.e. the third line. Why this anomaly? Am I doing something wrong here?
The aim of head -n 3 | tail -n 1 is to keep the third line into variable
It will be more efficient to use read builtin
read
read
read var
echo "${var}"
Or to keep heading white-spaces
IFS= read
and not join lines ending with \ or not give special meaning to \
read -r
You don't need $(cat) in your script. If script is reading data from stdin then just have this single line in your script:
head -n 3 | tail -n 1
And run it as:
bash myscript.sh < file.txt
This will print 3rd line from file.txt
PS: You can replace head + tail with this faster sed to print 3rd line from input:
sed '3q;d'
The shell is splitting the var variable so echo get multiple parameters. You need to quote your variable to prevent this to happen:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
This should do the trick, as far as I understand your question:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
var=$(cat) will not allow you to escape out of stdin mode. you need to specify the EOF for the script to understand to stop reading from stdin.
read -d '' var << EOF
echo "$var" | head -n 3 | tail -n 1
What I'm trying to do is:
Start a process that outputs text continously
Pipe that output to two commands:
A logger script
'head' command, so I can save the first lines the initial process output.
What I tried so far (unsuccessfuly) is:
./myProgram | tee > (myLogger log.txt) | head > firstLines.txt
The problem is that the myProgram exits as soon as head is finished.
Even if I use -i in tee command, I can't get myProgram to keep running.
Since the logger may append the incoming text to an existing file, executing
head log.txt > firstLines.txt
won't work in this case.
You can use awk as an alternative to do both:
./myProgram |
awk 'NR<=10{print > "firstLines.txt"} NR>10{close("firstLines.txt")} 1' > log.txt
Like this maybe:
yes | awk 'FNR<4 {print >>"file"; close("file")} 1' | more
where yes is your program, file is where you send the output of head to, and more is your logger.
I am beginner in unix programming and a way to automate my work
I want to run a list a grep commands and get the output of all the grep command in a in a single delimited file .
i am using the following bash script. But it's not working .
Mockup sh file:
!/bin/sh
grep -l abcd123
grep -l abcd124
grep -l abcd125
and while running i used the following command
$ ./Mockup.sh > output.txt
Is it the right command?
How can I get both the grep command and output in the output file?
how can i delimit the output after each command and result?
How can I get both the grep command and output in the output file
You can use bash -v (verbose) to print each command before execution on stderr and it's output will be as usual be available on stdout:
bash -v ./Mockup.sh > output.txt 2>&1
cat output.txt
Working Demo
A suitable shell script could be
#!/bin/sh
grep -l 'abcd123\|abcd124\|abcd125' "$#"
provided that the filenames you pass on the invocation of the script are "well behaved", that is no whitespace in them. (Edit Using the "$#" expansion takes care of generic whitespace in the filenames, tx to triplee for his/her comment)
This kind of invocation (with alternative matching strings, as per the \| syntax) has the added advantage that you have exactly one occurrence of a filename in your final list, because grep -l prints once the filename as soon as it finds the first occurrence of one of the three strings in a file.
Addendum about "$#"
% ff () { for i in "$#" ; do printf "[%s]\n" "$i" ; done ; }
% # NB "a s d" below is indeed "a SPACE s TAB d"
% ff "a s d" " ert " '345
345'
[a s d]
[ ert ]
[345
345]
%
cat myscript.sh
########################
#!/bin/bash
echo "Trying to find the file contenting the below string, relace your string with below string"
grep "string" /path/to/folder/* -R -l
########################
save above file and run it as below
sh myscript.sh > output.txt
once the command prmpt get return you can check the output.txt for require output.
Another approach, less efficient, that tries to address the OP question
How can I get both the grep command and output in the output file?
% cat Mockup
#!/bin/sh
grep -o -e string1 -e string2 -e string3 "$#" 2> /dev/null | sort -t: -k2 | uniq
Output: (mocked up as well)
% sh Mockup file{01..99}
file01:string1
file17:string1
file44:string1
file33:string2
file44:string2
file48:string2
%
looking at the output from POV of a consumer, one foresees problems with search strings and/or file names containing colons... oh well, that's another Q maybe
I need to grep both header and also particular pattern only using grep,
eg
for command "ps"
output
PID TTY TIME CMD
10280 pts/16 00:00:00 ps
32463 pts/16 00:00:00 bash
how can i grep both header and pattern like 32463 so output should be
PID TTY TIME CMD
32463 pts/16 00:00:00 bash
And One thing is that solution should be general that means it should be applicable to all commands that have headers
Try this:
ps | head -1; ps | grep bash
Like this:
ps | ( read -r head; printf '%s\n' "$head"; grep bash )
This generalizes to other commands, such as
( read -r head; printf '%s\n' "$head"; sort -k4n ) <input.csv >input-sorted-4n.csv
You could encapsulate this into a script called keepheader:
#!/bin/sh
read -r head
printf '%s\n' "$head"
exec "$#"
Use like
ps | keepheader grep bash
keepheader sort -k4n <input.csv >input-sorted-4n.csv
or maybe even
keepheader keepheader grep foo <<HERE
Header with underlines
------ ---- ----------
Cat food Whiskas
Mouse bait Cheese
HERE
(Actually maybe make the script accept an optiinal numeric parameter to specify how many header lines to keep; I leave this as an exercise for the reader.)
I suggest sed:
sed -n "1p;/$pattern/p"
how can i grep both header and pattern
You could try this
ps | grep -e 'PID\|32463'
solution should be general that means it should be applicable to all commands that have headers
This requirement is almost impossible to satisfy by grep, because different commands have different headers, it is impossible to assign a regular expression to match all of them.
But you could use the following command to achieve your goal:
command | perl -e 'while(<STDIN>) { print if $. == 1 or m/$ARGV[0]/ }' pattern
If it is too cumbersome for daily use, you can put it in a custom script, such as my-grep, and put that script in your $PATH, then you can use that script like a normal command:
command | my-grep pattern
With out using grep you can get this output for ps option
$ps -p 32463
-p Select by PID.
This selects the processes whose process ID numbers appear in pidlist. Identical to p and --pid.