I am trying to write a short script in which I use sed to search a stream, then perform a substitution on the stream based on the results of a shell function, which requires arguments from sed, e.g.
#!/bin/sh
function test {
echo "running test"
echo $1
}
sed -n -e "s/.*\(00\).*/$(test)/p" < testfile.txt
where testfile.txt contains:
1234
2345
3006
4567
(with newlines between each; they are getting removed by your sites formatting). So ok that script works for me (output "running test"), but obviously has no arguments passed to test. I would like the sed line to be something like:
sed -n -e "s/.*\(00\).*/$(test \1)/p" < testfile.txt
and output:
running test
00
So that the pattern matched by sed is fed as an argument to test. I didn't really expect the above to work, but I have tried every combination of $() brackets, backticks, and escapes I could think of, and can find no mention of this situation anywhere. Help?
Sed won't execute commands. Perl will, however, with the /e option on a regex command.
perl -pe 'sub testit { print STDERR "running test"; return #_[0]; }; s/.*(00).*/testit($1)/e' <testfile.txt
Redirect stderr to /dev/null if you don't want to see it in-line and screw up the output.
This might work for you:
tester () { echo "running test"; echo $1; }
export -f tester
echo -e "1234\n2345\n3006\n4567" |
sed -n 's/.*\(00\).*/echo "$(tester \1)"/p' | sh
running test
00
Or if your using GNU sed:
echo -e "1234\n2345\n3006\n4567" |
sed -n 's/.*\(00\).*/echo "$(tester \1)"/ep'
running test
00
N.B. You must remember to export the function first.
try this:
sed -n -e "s/.*\(00\).*/$1$2/p" testfile.txt | sh
Note: I might have the regex wrong, but the important bit is piping to shell (sh)
Related
I have looked quite a bit for answers but I am not finding any suggestions that have worked so far.
on command line, this works:
$ myvar=$( cat -n /usr/share/dict/cracklib-small | grep $myrand | sed -e "s/$myrand//" )
$ echo $myvar
$ commonness
however, inside a bash script the same exact lines just echoes out a blank line
notes - $myrand is a number, like 10340 generated with $RANDOM
cat prints out a dictionary with line numbers
grep grabs the line with $myrand in it ; e.g. 10340 commonness
sed is intended to remove the $myrand part of the line and replace it with nothing. here is my sample script
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
Your command line code is running:
grep $myrand
Your script is running:
grep myrand
These are not the same thing; the latter is looking for a word that contains "myrand" within it, not a random number.
By the way -- I'd suggest a different way to get a random line. If you have GNU coreutils, the shuf tool is built-to-purpose:
myword=$(shuf -n 1 /path/to/dict/cracklib-small)
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
where is the $ sign in grep myrand ?
you must put in some work before posting it here.
New to bash scripting. I'm getting pretty familiar with shell scripting pretty well. I wrote this text transform script for a feed for a client. And extracts the url's I want, and the titles of articles. Awesome.
echo $(var=$(curl -L website.com/news)) |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var |
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g') ; let this=0 ; echo "$result" | while read line ; do if ((this % 2 == 0 )) ; then echo website.com/news$line ; else echo $line ; fi ; let this+=1 ; done
When I try to extract it to a file and run it with bash OR sh myThing.sh, it doesn't work at all. The only thing that echo's is 'webiste.com/news', when I try to echo $this, all I get is 1. What am I doing wrong?
#!/bin/bash
echo $(var=$(curl -L website.com/news)) |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var |
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
let this=0
echo "$result" | while read line
do
if ((this % 2 == 0 ))
then
echo website.com/news$line
else
echo $line
fi
let this+=1
done
edit:
#!/bin/bash
var=$(curl -L linux.com/news)
select=$(grep -Po '<h3 class="article-list__title"><a href="\K[^<]+' <<< $var)
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
let this=0
echo "$result" | while read line
do
if ((this % 2 == 0 ))
then
echo website.com/news$line
else
echo $line
fi
let this+=1
done
This answer solves the OP's specific problem, but to address the question "Why is my shell command working at the prompt, but not as a bash script?" generally, Etan Reisner provides an excellent answer in the comments:
"You are either not running that exact command or it "works" because you have shell state that is affecting things in ways you take to be "working" and your script doesn't have that state. Try launching an entirely new shell session and see if that command, on its own, works for you there."
echo $(var=...) will assign a value to variable $var, but will not output anything, so the echo command will simply print a newline.
Furthermore, because the assignment to $var happens inside $(...) (a command substitution), it is confined to the subshell that the command inside the substitution ran in, so $var will not be defined in the calling shell.
(A subshell is a child process that contains a duplicate of the current shell's environment, without being able to modify the current shell's environment).
More generally, you cannot meaningfully define variables inside a pipeline - they will neither be visible to other pipeline segments, nor after the pipeline finishes.[1]
The only reason your [original] command could ever have worked is if $var had a preexisting value in your shell.
In fact, given that you provide input to grep via a here-string (<<<), the first segment of your pipeline (echo ...) is entirely ignored.
To pass the output of curl through the pipeline to grep and then to sed, no intermediate variables are needed at all.
Furthermore, your sed command is lacking input: you probably meant to feed it $var in your first attempt, and $select in the 2nd (your 2nd attempt came close to a correct solution).
What you were probably ultimately looking for:
result=$(curl -L website.com/news |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' |
sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
# ... processing of "$result"
Some additional notes:
You could combine the 3 sed calls into a single one.
You could feed the pipeline output directly into your while loop, without the need for intermediate variable $result.
You should generally double-quote variable references (e.g., use "$line" instead of $line to protect them from interpretation by the shell (word-splitting, globbing).
let this+=1 is better expressed as (( ++this )) in modern Bash.
This answer of mine contains links to resources for learning about bash.
[1] All commands involved in a pipeline by default run in a subshell in bash, so they all see copies of the parent shell's variables. Bash 4.2+ offers the lastpipe option (off by default) to allow you to create variables in the current shell instead of in a subshell, by running the last pipeline segment (only) in the current shell instead of in a subshell, to facilitate scenarios such as ... | while read -r line ... and have $line continue to exist after the pipeline finishes.
Note that this still doesn't enable defining a variable in an earlier pipeline segment in the hopes that a later segment will see it - this can never work, because the commands that make up a pipeline are launched at the same time, and it is only through coordination of the input and output streams that effective left-to-right processing happens.
This line is totally wrong. You are attempting to pass thru pipes the standard output of each process when none of them ever prints anything except standard error.
echo $(var=$(curl -L website.com/news)) | grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var | result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
I'll break down what I believe you are attempting to do.
echo $(var=$(curl -: website.com/news))
The above code will only print the standard error, which is a separate stream than standard output. The standard output is assigned to $var. However you are attempting to pass the standard output to the next process which is nothing but a newline at this time.
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var
The here-string <<< takes precedence over pipe. But variable $var is lost as it was defined inside a sub-shell and not in the parent shell. Thanks to #mklement0.
The proper way to accomplish all this is to not use $var. All you wanted is the value stored in $result.
result=$(curl -L website.com/news | grep -Po '<h3 class="article-link"><a href="\K[^<]+'| sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
I don't intend to optimize your script. This is more of a suggested solution. A more comprehensive answer to your question Why is my shell command working at the prompt, but not as a bash script? is answered by mklement0 here.
I am beginner in unix programming and a way to automate my work
I want to run a list a grep commands and get the output of all the grep command in a in a single delimited file .
i am using the following bash script. But it's not working .
Mockup sh file:
!/bin/sh
grep -l abcd123
grep -l abcd124
grep -l abcd125
and while running i used the following command
$ ./Mockup.sh > output.txt
Is it the right command?
How can I get both the grep command and output in the output file?
how can i delimit the output after each command and result?
How can I get both the grep command and output in the output file
You can use bash -v (verbose) to print each command before execution on stderr and it's output will be as usual be available on stdout:
bash -v ./Mockup.sh > output.txt 2>&1
cat output.txt
Working Demo
A suitable shell script could be
#!/bin/sh
grep -l 'abcd123\|abcd124\|abcd125' "$#"
provided that the filenames you pass on the invocation of the script are "well behaved", that is no whitespace in them. (Edit Using the "$#" expansion takes care of generic whitespace in the filenames, tx to triplee for his/her comment)
This kind of invocation (with alternative matching strings, as per the \| syntax) has the added advantage that you have exactly one occurrence of a filename in your final list, because grep -l prints once the filename as soon as it finds the first occurrence of one of the three strings in a file.
Addendum about "$#"
% ff () { for i in "$#" ; do printf "[%s]\n" "$i" ; done ; }
% # NB "a s d" below is indeed "a SPACE s TAB d"
% ff "a s d" " ert " '345
345'
[a s d]
[ ert ]
[345
345]
%
cat myscript.sh
########################
#!/bin/bash
echo "Trying to find the file contenting the below string, relace your string with below string"
grep "string" /path/to/folder/* -R -l
########################
save above file and run it as below
sh myscript.sh > output.txt
once the command prmpt get return you can check the output.txt for require output.
Another approach, less efficient, that tries to address the OP question
How can I get both the grep command and output in the output file?
% cat Mockup
#!/bin/sh
grep -o -e string1 -e string2 -e string3 "$#" 2> /dev/null | sort -t: -k2 | uniq
Output: (mocked up as well)
% sh Mockup file{01..99}
file01:string1
file17:string1
file44:string1
file33:string2
file44:string2
file48:string2
%
looking at the output from POV of a consumer, one foresees problems with search strings and/or file names containing colons... oh well, that's another Q maybe
I am writing a shell script for linux which takes argument as port no.
inside file following is a line which needs to be updated:
define('NO_OF_PORTS',10);
I need to replace that 10 by the argument passed.
But this should be dynamic, like next time I pass new port no it must get updated.
Using sed:
s="define('NO_OF_PORTS',10);"
n=25
sed "s/\('NO_OF_PORTS',\)[0-9]*/\1$n/" <<< "$s"
define('NO_OF_PORTS',25);
To change inline in the file use:
sed -i.bak "s/\('NO_OF_PORTS',\)[0-9]*/\1$n/" file
You can use sed in the script to edit the file
sed -i s/NO_OF_PORTS\',[0-9]*/NO_OF_PORTS\',$1/ $2
1.txt has
define('NO_OF_PORTS',19)
shell script
#!/bin/sh
echo $1
sed -i -r '/NO_OF_PORTS/ s/'[0-9]+'/'$1'/g' 1.txt
run
linux:/home/test # ./replace_port.sh 78
linux:/home/test # cat 1.txt
define('NO_OF_PORTS',78)
I looked at a very similar question but was unable to resolve the issue Replace comma with newline in sed
I am trying to convert : characters in a string. This is what I tried:
echo -e 'this:is:a:test' | sed "s/\:/'\n'/g"
but this replaces : with n. I tried tr too but had the same result. I believe the -e is not seen after being piped so new line is not recognized.
Any help is appreciated.
echo 'this:is:a:test' | tr : \\n
Any POSIX-compliant tr will support the \n escape sequence. You need to take care to quote or escape the escape sequence, however (double backslash above).
The -e argument to echo has no effect on your argument to echo.
I'll presume that you have the string in a variable already. This uses the parameter expansion substitution operator to replace every : with a newline, which is specified using a $'...'-quoted string. Both features are bash extensions to the standard, and may not work in another shell.
$ foo="this:is:a:test"
$ bar="${foo//:/$'\n'}"
$ echo "$bar"
this
is
a
test
Perhaps Perl is an option?
echo -e 'this:is:a:test' | perl -p -e 's/:/\n/g'
You do not need echo -e because you have \n in sed, not in echo statement.
So, the following should work (note that I have changed '\n' to \n):
echo -e 'this:is:a:test' | sed "s/\:/\n/g"
or
echo 'this:is:a:test' | sed "s/\:/\n/g"
Also note that you do not need to escape : so the following will work too (thanks to #anishsane)
echo 'this:is:a:test' | sed "s/:/\n/g"
Below is just to reiterate why you need -e for echo
$ echo -e "hello \n"
hello
$ echo "hello \n"
hello \n
If Python is an option:
echo 'this:is:a:test' | python3 -c 'print(input().replace(":", "\n"))'
output:
this
is
a
test