I have used Sed to grab two interesting values. Now I want to send those two values as parameters to Curl. I have been successful piping Sed output to Curl with only 1 argument using xargs, however I am unable to use two arguments for one command.
echo "value1" "value2" | curl --data 'valA=$1&valB=$2' http://example.com
I am stuck in both theory and practice. I wasn't planning to use bash scripting.
[ I am running tshark, piping output to sed, and hoping to pipe that output to curl so as to record data in a remote DB. ]
As long as you're not expecting to use valA and valB in future commands, you can use read to store the the whitespace-delimited output from your exiting commands:
$ echo foo bar | { read var1 var2 ; echo $var1 $var2 ; }
foo bar
Which means you can do:
echo "value1" "value2" | { read a b ; curl --data 'valA=$a&valB=$b' http://example.com ; }
Assuming you can get sed to output something like .csv (foo\tbar):
tshark | sed ... | parallel --colsep '\t' -q curl --data 'valA={1}&valB={2}' http://example.com
You can find more about GNU Parallel at:
http://www.gnu.org/s/parallel/
Watch the intro video on
http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
Backticks sound like the best bet, if you can't run the sed twice (eg you're doing this on a live tshark stream):
curl --data $(printf "valA=%s&valB=%s" `tshark --whatever | sed -e whatever`)
Related
I'm developping a script to search for patterns within scripts executed from CRON on a bunch of remote servers through SSH.
Script on client machine -- SSH --> Remote Servers CRON/Scripts
For now I can't get the correct output.
Script on client machine
#!/bin/bash
server_list=( '172.x.x.x' '172.x.x.y' '172.x.x.z' )
for s in ${server_list[#]}; do
ssh -i /home/user/.ssh/my_key.rsa user#${s} crontab -l | grep -v '^#\|^[[:space:]]*$' | cut -d ' ' -f 6- | awk '{print $1}' | grep -v '^$\|^echo\|^find\|^PATH\|^/usr/bin\|^/bin/' | xargs -0 grep -in 'server.tld\|10.x.x.x'
done
This only gives me the paths of scripts from crontab, not the matched lines and line number plus the first line is prefixed with "grep:" keyword (example below):
grep: /opt/directory/script1.sh
/opt/directory/script2.sh
/opt/directory/script3.sh
/opt/directory/script4.sh
How to get proper output, meaning the script path plus line number plus line of matching pattern?
Remote CRON examples
OO 6 * * * /opt/directory/script1.sh foo
30 6 * * * /opt/directory/script2.sh bar
Remote script content examples
1 ) This will match grep pattern
#!/bin/bash
ping -c 4 server.tld && echo "server.tld ($1)"
2 ) This won't match grep pattern
#!/bin/bash
ping -c 4 8.x.x.x && echo "8.x.x.x ($1)"
Without example input, it's really hard to see what your script is attempting to do. But the cron parsing could almost certainly be simplified tremendously by refactoring all of it into a single Awk script. Here is a quick stab, with obviously no way to test.
#!/bin/sh
# No longer using an array for no good reason, so /bin/sh will work
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
# no -0; use grep -E and properly quote literal dot
xargs grep -Ein 'server\.tld|10.x.x.x'
done
Your command would not output null-delimited data to xargs so probably the immediate problem was that xargs -0 would receive all the file names as a single file name which obviously does not exist, and you forgot to include the ": file not found" from the end of the error message.
The use of grep -E is a minor hack to enable a more modern regex syntax which is more similar to that in Awk, where you don't have to backslash the "or" pipe etc.
This script, like your original, runs grep on the local system where you run the SSH script. If you want to run the commands on the remote server, you will need to refactor to put the entire pipeline in single quotes or a here document:
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" <<\________HERE
crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
xargs grep -Ein 'server\.tld|10.x.x.x'
________HERE
done
The refactored script contains enough complexities in the quoting that you probably don't want to pass it as an argument to ssh, which requires you to figure out how to quote strings both locally and remotely. It's easier then to pass it as standard input, which obviously just gets transmitted verbatim.
If you get "Pseudo-terminal will not be allocated because stdin is not a terminal.", try using ssh -t. Sometimes you need to add multiple -t options to completely get rid of this message.
Is there an easy way to concatenate the result of two linux commands, in one line? (i.e without using variables)
I pull the local outdoor temperature from a nearby weather station. The result today is:
5.2
I simply want to add the units, so result should look like this:
5.2°C
An example command, that almost gives me what I want is:
wget -q -O- "http://meteocentre.com/montreal/home_e.html" | grep -oP '(?<=Tn= ).*(?=°)' ; printf "°C\n"
You are already concatenating the results.
wget .. | grep .. outputs: 5.2\n
printf outputs °C\n
The concatenated result is therefore 5.2\n°C\n , exactly what you're getting.
What you want to do is strip the linefeed after 5.2. You can take advantage of the fact that $(command substitution) strips trailing linefeeds and pass it to printf:
printf '%s°C\n' "$(wget -q -O- "http://meteocentre.com/montreal/home_e.html" | grep -oP '(?<=Tn= ).*(?=°)')"
in bash, add pipe at the end:
echo $(cat) [more text]
or
echo `cat` [more text]
example:
wget -q -O- "http://meteocentre.com/montreal/home_e.html" | grep -oP '(?<=Tn= ).*(?=°)' | echo $(cat) °C
-> 5.2 °C
I have the following simple bash script which takes input from stdin and prints the third line given as input.
#!/bin/bash
var=$(cat)
echo $var | head -n 3 | tail -n 1
The problem with this script is that it prints all the lines but here is the funny part, when I type the commands individually on the command line I am getting the expected result i.e. the third line. Why this anomaly? Am I doing something wrong here?
The aim of head -n 3 | tail -n 1 is to keep the third line into variable
It will be more efficient to use read builtin
read
read
read var
echo "${var}"
Or to keep heading white-spaces
IFS= read
and not join lines ending with \ or not give special meaning to \
read -r
You don't need $(cat) in your script. If script is reading data from stdin then just have this single line in your script:
head -n 3 | tail -n 1
And run it as:
bash myscript.sh < file.txt
This will print 3rd line from file.txt
PS: You can replace head + tail with this faster sed to print 3rd line from input:
sed '3q;d'
The shell is splitting the var variable so echo get multiple parameters. You need to quote your variable to prevent this to happen:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
This should do the trick, as far as I understand your question:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
var=$(cat) will not allow you to escape out of stdin mode. you need to specify the EOF for the script to understand to stop reading from stdin.
read -d '' var << EOF
echo "$var" | head -n 3 | tail -n 1
I am trying to make a bash shell script that launches some jobs on a queuing system. After a job is launched, the launch command prints the job-id to the stdout, which I would like to 'trap' and then use in the next command. The job-id digits are the only digits in the stdout message.
#!/bin/bash
./some_function
>>> this is some stdout text and the job number is 1234...
and then I would like to get to:
echo $job_id
>>> 1234
My current method is using a tee command to pipe the original command's stdout to a tmp.txt file and then making the variable by grepping that file with a regex filter...something like:
echo 'pretend this is some dummy output from a function 1234' 2>&1 | tee tmp.txt
job_id=`cat tmp.txt | grep -o '[0-9]'`
echo $job_id
>>> pretend this is some dummy output from a function 1234
>>> 1 2 3 4
...but I get the feeling this is not really the most elegant or 'standard' way of doing this. What is the better way to do this?
And for bonus points, how do I remove the spaces from the grep+regex output?
You can use grep -o when you call your script:
jobid=$(echo 'pretend this is some dummy output from a function 1234' 2>&1 |
tee tmp.txt | grep -Eo '[0-9]+$')
echo "$jobid"
1234
Something like this should work:
$ JOBID=`./some_function | sed 's/[^0-9]*\([0-9]*\)[^0-9]*/\1/'`
$ echo $JOBID
1234
I am trying to write a short script in which I use sed to search a stream, then perform a substitution on the stream based on the results of a shell function, which requires arguments from sed, e.g.
#!/bin/sh
function test {
echo "running test"
echo $1
}
sed -n -e "s/.*\(00\).*/$(test)/p" < testfile.txt
where testfile.txt contains:
1234
2345
3006
4567
(with newlines between each; they are getting removed by your sites formatting). So ok that script works for me (output "running test"), but obviously has no arguments passed to test. I would like the sed line to be something like:
sed -n -e "s/.*\(00\).*/$(test \1)/p" < testfile.txt
and output:
running test
00
So that the pattern matched by sed is fed as an argument to test. I didn't really expect the above to work, but I have tried every combination of $() brackets, backticks, and escapes I could think of, and can find no mention of this situation anywhere. Help?
Sed won't execute commands. Perl will, however, with the /e option on a regex command.
perl -pe 'sub testit { print STDERR "running test"; return #_[0]; }; s/.*(00).*/testit($1)/e' <testfile.txt
Redirect stderr to /dev/null if you don't want to see it in-line and screw up the output.
This might work for you:
tester () { echo "running test"; echo $1; }
export -f tester
echo -e "1234\n2345\n3006\n4567" |
sed -n 's/.*\(00\).*/echo "$(tester \1)"/p' | sh
running test
00
Or if your using GNU sed:
echo -e "1234\n2345\n3006\n4567" |
sed -n 's/.*\(00\).*/echo "$(tester \1)"/ep'
running test
00
N.B. You must remember to export the function first.
try this:
sed -n -e "s/.*\(00\).*/$1$2/p" testfile.txt | sh
Note: I might have the regex wrong, but the important bit is piping to shell (sh)