I have the following simple bash script which takes input from stdin and prints the third line given as input.
#!/bin/bash
var=$(cat)
echo $var | head -n 3 | tail -n 1
The problem with this script is that it prints all the lines but here is the funny part, when I type the commands individually on the command line I am getting the expected result i.e. the third line. Why this anomaly? Am I doing something wrong here?
The aim of head -n 3 | tail -n 1 is to keep the third line into variable
It will be more efficient to use read builtin
read
read
read var
echo "${var}"
Or to keep heading white-spaces
IFS= read
and not join lines ending with \ or not give special meaning to \
read -r
You don't need $(cat) in your script. If script is reading data from stdin then just have this single line in your script:
head -n 3 | tail -n 1
And run it as:
bash myscript.sh < file.txt
This will print 3rd line from file.txt
PS: You can replace head + tail with this faster sed to print 3rd line from input:
sed '3q;d'
The shell is splitting the var variable so echo get multiple parameters. You need to quote your variable to prevent this to happen:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
This should do the trick, as far as I understand your question:
#!/bin/bash
var=$(cat)
echo "$var" | head -n 3 | tail -n 1
var=$(cat) will not allow you to escape out of stdin mode. you need to specify the EOF for the script to understand to stop reading from stdin.
read -d '' var << EOF
echo "$var" | head -n 3 | tail -n 1
Related
Running Centos8.
Here is my current (tiny) script:
#!/usr/bin/env bash
x=$((($2-$1)+1))
head -n $2 $3 | tail -n -$x
As an example of an input:
sh script.sh 7 10 /etc/passwd
And what I am trying to get out is lines 7-10 from /etc/passwd.
It should be fairly simple, because just doing
head -n 7 /etc/passwd | tail -n -4
Returns those 4 lines perfectly fine.
However, using $x (to get the input from the second number, minus the first, plus 1), fails. Whenever I run this code I get this error:
tail: invalid number of lines: ‘4\r\r’
Which from my understanding means that the code is not accepting the $x input from the small math I did, and thinks that its 4 with two trailing spaces?
How would I go about fixing this issue, so that the four lines of /etc/password get listed?
There's problem on line 3
#!/usr/bin/env bash
x=$((($2-$1)+1))
head -n $1 $3 | tail -n -$x
That needs to say $1, not $2.
And the line endings are Mac, not Linux.
I have looked quite a bit for answers but I am not finding any suggestions that have worked so far.
on command line, this works:
$ myvar=$( cat -n /usr/share/dict/cracklib-small | grep $myrand | sed -e "s/$myrand//" )
$ echo $myvar
$ commonness
however, inside a bash script the same exact lines just echoes out a blank line
notes - $myrand is a number, like 10340 generated with $RANDOM
cat prints out a dictionary with line numbers
grep grabs the line with $myrand in it ; e.g. 10340 commonness
sed is intended to remove the $myrand part of the line and replace it with nothing. here is my sample script
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
Your command line code is running:
grep $myrand
Your script is running:
grep myrand
These are not the same thing; the latter is looking for a word that contains "myrand" within it, not a random number.
By the way -- I'd suggest a different way to get a random line. If you have GNU coreutils, the shuf tool is built-to-purpose:
myword=$(shuf -n 1 /path/to/dict/cracklib-small)
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
where is the $ sign in grep myrand ?
you must put in some work before posting it here.
I wrote a bash script where I define a variable like this:
var=$(cat $file_path | head -n $var2 | tail -n 1 | cut -f1)
Where $file_path simply contains the path to a file and $var2 is an int, e.g., 1 or 2. The variable is therefore assigned the value of the first field of line number var2 of the file.
It works perfectly fine when I run this from the command line. However, when running the script containing this command, I get the error
cat: write error: Broken pipe
Any idea why that is?
There's no need to use cat, since head takes a filename argument.
var=$(head -n $var2 $file_path | tail -n 1 | cut -f1)
Actually, there's no need to use any of those commands.
var=$(awk -v line=$var2 'NR == line { print $1; exit }' $file_path)
New to bash scripting. I'm getting pretty familiar with shell scripting pretty well. I wrote this text transform script for a feed for a client. And extracts the url's I want, and the titles of articles. Awesome.
echo $(var=$(curl -L website.com/news)) |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var |
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g') ; let this=0 ; echo "$result" | while read line ; do if ((this % 2 == 0 )) ; then echo website.com/news$line ; else echo $line ; fi ; let this+=1 ; done
When I try to extract it to a file and run it with bash OR sh myThing.sh, it doesn't work at all. The only thing that echo's is 'webiste.com/news', when I try to echo $this, all I get is 1. What am I doing wrong?
#!/bin/bash
echo $(var=$(curl -L website.com/news)) |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var |
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
let this=0
echo "$result" | while read line
do
if ((this % 2 == 0 ))
then
echo website.com/news$line
else
echo $line
fi
let this+=1
done
edit:
#!/bin/bash
var=$(curl -L linux.com/news)
select=$(grep -Po '<h3 class="article-list__title"><a href="\K[^<]+' <<< $var)
result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
let this=0
echo "$result" | while read line
do
if ((this % 2 == 0 ))
then
echo website.com/news$line
else
echo $line
fi
let this+=1
done
This answer solves the OP's specific problem, but to address the question "Why is my shell command working at the prompt, but not as a bash script?" generally, Etan Reisner provides an excellent answer in the comments:
"You are either not running that exact command or it "works" because you have shell state that is affecting things in ways you take to be "working" and your script doesn't have that state. Try launching an entirely new shell session and see if that command, on its own, works for you there."
echo $(var=...) will assign a value to variable $var, but will not output anything, so the echo command will simply print a newline.
Furthermore, because the assignment to $var happens inside $(...) (a command substitution), it is confined to the subshell that the command inside the substitution ran in, so $var will not be defined in the calling shell.
(A subshell is a child process that contains a duplicate of the current shell's environment, without being able to modify the current shell's environment).
More generally, you cannot meaningfully define variables inside a pipeline - they will neither be visible to other pipeline segments, nor after the pipeline finishes.[1]
The only reason your [original] command could ever have worked is if $var had a preexisting value in your shell.
In fact, given that you provide input to grep via a here-string (<<<), the first segment of your pipeline (echo ...) is entirely ignored.
To pass the output of curl through the pipeline to grep and then to sed, no intermediate variables are needed at all.
Furthermore, your sed command is lacking input: you probably meant to feed it $var in your first attempt, and $select in the 2nd (your 2nd attempt came close to a correct solution).
What you were probably ultimately looking for:
result=$(curl -L website.com/news |
grep -Po '<h3 class="article-link"><a href="\K[^<]+' |
sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
# ... processing of "$result"
Some additional notes:
You could combine the 3 sed calls into a single one.
You could feed the pipeline output directly into your while loop, without the need for intermediate variable $result.
You should generally double-quote variable references (e.g., use "$line" instead of $line to protect them from interpretation by the shell (word-splitting, globbing).
let this+=1 is better expressed as (( ++this )) in modern Bash.
This answer of mine contains links to resources for learning about bash.
[1] All commands involved in a pipeline by default run in a subshell in bash, so they all see copies of the parent shell's variables. Bash 4.2+ offers the lastpipe option (off by default) to allow you to create variables in the current shell instead of in a subshell, by running the last pipeline segment (only) in the current shell instead of in a subshell, to facilitate scenarios such as ... | while read -r line ... and have $line continue to exist after the pipeline finishes.
Note that this still doesn't enable defining a variable in an earlier pipeline segment in the hopes that a later segment will see it - this can never work, because the commands that make up a pipeline are launched at the same time, and it is only through coordination of the input and output streams that effective left-to-right processing happens.
This line is totally wrong. You are attempting to pass thru pipes the standard output of each process when none of them ever prints anything except standard error.
echo $(var=$(curl -L website.com/news)) | grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var | result=$(sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
I'll break down what I believe you are attempting to do.
echo $(var=$(curl -: website.com/news))
The above code will only print the standard error, which is a separate stream than standard output. The standard output is assigned to $var. However you are attempting to pass the standard output to the next process which is nothing but a newline at this time.
grep -Po '<h3 class="article-link"><a href="\K[^<]+' <<< $var
The here-string <<< takes precedence over pipe. But variable $var is lost as it was defined inside a sub-shell and not in the parent shell. Thanks to #mklement0.
The proper way to accomplish all this is to not use $var. All you wanted is the value stored in $result.
result=$(curl -L website.com/news | grep -Po '<h3 class="article-link"><a href="\K[^<]+'| sed 's/"/\n/g' | sed 's/ \//\n\//g' | sed 's/>//g')
I don't intend to optimize your script. This is more of a suggested solution. A more comprehensive answer to your question Why is my shell command working at the prompt, but not as a bash script? is answered by mklement0 here.
I am trying to make a bash shell script that launches some jobs on a queuing system. After a job is launched, the launch command prints the job-id to the stdout, which I would like to 'trap' and then use in the next command. The job-id digits are the only digits in the stdout message.
#!/bin/bash
./some_function
>>> this is some stdout text and the job number is 1234...
and then I would like to get to:
echo $job_id
>>> 1234
My current method is using a tee command to pipe the original command's stdout to a tmp.txt file and then making the variable by grepping that file with a regex filter...something like:
echo 'pretend this is some dummy output from a function 1234' 2>&1 | tee tmp.txt
job_id=`cat tmp.txt | grep -o '[0-9]'`
echo $job_id
>>> pretend this is some dummy output from a function 1234
>>> 1 2 3 4
...but I get the feeling this is not really the most elegant or 'standard' way of doing this. What is the better way to do this?
And for bonus points, how do I remove the spaces from the grep+regex output?
You can use grep -o when you call your script:
jobid=$(echo 'pretend this is some dummy output from a function 1234' 2>&1 |
tee tmp.txt | grep -Eo '[0-9]+$')
echo "$jobid"
1234
Something like this should work:
$ JOBID=`./some_function | sed 's/[^0-9]*\([0-9]*\)[^0-9]*/\1/'`
$ echo $JOBID
1234