This question already has answers here:
Syntax error in shell script with process substitution
(4 answers)
syntax error near unexpected token `<'
(2 answers)
Closed 4 months ago.
See solution at end of post.
I'm running into a frustrating problem. Thanks for any help you can give. What I'm trying to do is get all occurrences of certain values in an error file. I'm using process substitution to parse error output files. I created a test script with the following code and it works perfectly.
#--------------------
# Get all values of affiliate
echo "Building array of affiliates with errors"
while read -r line; do
AffiliateName_array+=($line)
i=$(( ${#AffiliateName_array[#]} - 1 ))
echo " ScriptName_array | index --> $i | $line"
done < <(grep Running ${LogDir}/temp_python_errors.txt | cut -d' ' -f5 | tr '[:lower:]' '[:upper:]')
The echo output looks like this:
Building array of affiliates with errors
index --> 0 | affiliate1
index --> 1 | affiliate2
Then I cut and pasted that exact code into the final script and it throws the error:
path/AR_exit_process.sh: line 166: syntax error near unexpected token `<'
path/Affiliate_Remits/AR_exit_process.sh: line 166: ` done < <(grep Running ${LogDir}/temp_python_errors.txt | cut -d' ' -f5 | tr '[:lower:]' '[:upper:]')'
Here's the code from my final script (exactly the same as the test script since it was cut and pasted):
#--------------------
# Get all values of affiliate
echo "Building array of affiliates with errors"
while read -r line; do
AffiliateName_array+=($line)
i=$(( ${#AffiliateName_array[#]} - 1 ))
echo " ScriptName_array | index --> $i | $line"
done < <(grep Running ${LogDir}/temp_python_errors.txt | cut -d' ' -f5 | tr '[:lower:]' '[:upper:]')
NOTES:
All scripts have the #!/bin/bash bang line
The scripts run on the same server
The test script is called directly from the command line but
the final script is 2 layers deep. All calling scripts use #!/bin/bash
My main reference:
https://wiki.bash-hackers.org/syntax/expansion/proc_subst
SOLUTION: As Charles suggested I added a line to both the test and final scripts with the command: set -o I found that when running the test script, I got the value posix off. However, when I ran the final script with set -o I got posix on. What this means is that the test script was running under bash while the final script was running under sh. The solution is to add the line set +o posix to turn posix off before executing the code. What I'm still not clear about is how the posix setting is different on the same server. More research ...
Related
This question already has answers here:
Conditional step in a pipeline
(2 answers)
Can I make a shell function in as a pipeline conditionally "disappear", without using cat?
(1 answer)
Closed 4 months ago.
EMPTY_VAR=''
MMDDYYYY='6.18.1997'
PIPE_VAR=' | xargs echo "1+" | bc'
echo "$MMDDYYYY" | cut -d "." -f 2${EMPTY_VAR}
>> 18
Command above would give me correct output, which is 18, but if I try to use PIPE_VAR instead it would give me bunch of errors:
echo "$MMDDYYYY" | cut -d "." -f 2${PIPE_VAR}
cut: '|': No such file or directory
cut: xargs: No such file or directory
cut: echo: No such file or directory
cut: '"1+"': No such file or directory
cut: '|': No such file or directory
cut: bc: No such file or directory
OR:
echo "$MMDDYYYY" | cut -d "." -f 2"$PIPE_VAR"
cut: invalid field value ‘| xargs echo "1+" | bc’
Try 'cut --help' for more information.
What I'm really trying to find out is that even possible to combine commands like this?
You can't put control operators like | in a variable, at least not without resorting to something like eval. Syntax parsing comes before parameter expansion when evaluating the command line, so Bash is only ever going to see that | as a literal character and not pipeline syntax. See BashParsing for more details.
Conditionally adding a pipeline is hard to do well, but having a part of the pipeline conditionally execute one command or another is more straightforward. It might look something like this:
#!/bin/bash
MMDDYYYY='6.18.1997'
echo "$MMDDYYYY" | cut -d "." -f 2 |
if some_conditional_command ; then
xargs echo "1+" | bc
else
cat
fi
It looks like you're trying to calculate the next day. That's hard to do with plain arithmetic, particularly with month/year ends.
Let date do the work. This is GNU date. It can't parse 6.18.1997 but it can parse 6/18/1997
for MMDDYYYY in '2.28.1996' '2.28.1997'; do
date_with_slashes=${MMDDYYYY//./\/}
next_day=$(date -d "$date_with_slashes + 1 day" '+%-m.%-d.%Y')
echo "$next_day"
done
2.29.1996
3.1.1997
This question already has answers here:
Difference between sh and Bash
(11 answers)
Closed 1 year ago.
I'm still figuring out some niches with bash scripting, I get the following error in my bash script:
updated_names=$(comm -23 <(echo ${current_run_names} | tr " " "\n" | sort) <(echo ${started_run_names} | tr " " "\n" | sort) )
updated_names =$(echo ${updated_names} | cut -d' ' -f1)
Not sure what I've been doing wrong, have deleted various tokens but still continue to get the same error.
How I run it
sh conversion.sh -r directory_location
Supposing that the first of these lines ...
updated_run_names=$(comm -23 <(echo ${current_run_names} | tr " " "\n" | sort) <(echo ${started_run_names} | tr " " "\n" | sort) )
update_run_names=$(echo ${new_run_names} | cut -d' ' -f1)
... is the line 72 referenced in the error message, the issue is most likely with your process substitutions. These are the two fragments of the form <( command ).
Process substitution is a Bash extension to the POSIX shell language. It is not recognized by most other shells, and it is not recognized by Bash itself when it runs in POSIX mode. There is more than one way to get Bash running in POSIX mode, but one of them is to invoke it via the name sh, which is what you are doing.
Note well that the shebang line at the top of your script has a functional role only when you launch the script directly, by making it executable and launching it by name:
./conversion.sh -r directory_location
When you instead launch it as shown in the question, by naming it as an argument to sh, the shebang line is just a comment, and Bash runs (as sh) in POSIX mode.
Note also that although it is conventional for some other language interpreters (Python, especially), it is not conventional to use env in shell script shebang lines.
I have a shell script of more than 1000 lines, i would like to check if all the commands used in the script are installed in my Linux operating system.
Is there any tool to get the list of Linux commands used in the shell script?
Or how can i write a small script which can do this for me?
The script runs successfully on the Ubuntu machine, it is invoked as a part of C++ application. we need to run the same on a device where a Linux with limited capability runs. I have identified manually, few commands which the script runs and not present on Device OS. before we try installing these commands i would like to check all other commands and install all at once.
Thanks in advance
I already tried this in the past and got to the conclusion that is very difficult to provide a solution which would work for all scripts. The reason is that each script with complex commands has a different approach in using the shells features.
In case of a simple linear script, it might be as easy as using debug mode.
For example: bash -x script.sh 2>&1 | grep ^+ | awk '{print $2}' | sort -u
In case the script has some decisions, then you might use the same approach an consider that for the "else" cases the commands would still be the same just with different arguments or would be something trivial (echo + exit).
In case of a complex script, I attempted to write a script that would just look for commands in the same place I would do it myself. The challenge is to create expressions that would help identify all used possibilities, I would say this is doable for about 80-90% of the script and the output should only be used as reference since it will contain invalid data (~20%).
Here is an example script that would parse itself using a very simple approach (separate commands on different lines, 1st word will be the command):
# 1. Eliminate all quoted text
# 2. Eliminate all comments
# 3. Replace all delimiters between commands with new lines ( ; | && || )
# 4. extract the command from 1st column and print it once
cat $0 \
| sed -e 's/\"/./g' -e "s/'[^']*'//g" -e 's/"[^"]*"//g' \
| sed -e "s/^[[:space:]]*#.*$//" -e "s/\([^\\]\)#[^\"']*$/\1/" \
| sed -e "s/&&/;/g" -e "s/||/;/g" | tr ";|" "\n\n" \
| awk '{print $1}' | sort -u
the output is:
.
/
/g.
awk
cat
sed
sort
tr
There are many more cases to consider (command substitutions, aliases etc.), 1, 2 and 3 are just beginning, but they would still cover 80% of most complex scripts.
The regular expressions used would need to be adjusted or extended to increase precision and special cases.
In conclusion if you really need something like this, then you can write a script as above, but don't trust the output until you verify it yourself.
Add export PATH='' to the second line of your script.
Execute your_script.sh 2>&1 > /dev/null | grep 'No such file or directory' | awk '{print $4;}' | grep -v '/' | sort | uniq | sed 's/.$//'.
If you have a fedora/redhat based system, bash has been patched with the --rpm-requires flag
--rpm-requires: Produce the list of files that are required for the shell script to run. This implies -n and is subject to the same limitations as compile time error checking checking; Command substitutions, Conditional expressions and eval builtin are not parsed so some dependencies may be missed.
So when you run the following:
$ bash --rpm-requires script.sh
executable(command1)
function(function1)
function(function2)
executable(command2)
function(function3)
There are some limitations here:
command and process substitutions and conditional expressions are not picked up. So the following are ignored:
$(command)
<(command)
>(command)
command1 && command2 || command3
commands as strings are not picked up. So the following line will be ignored
"/path/to/my/command"
commands that contain shell variables are not listed. This generally makes sense since
some might be the result of some script logic, but even the following is ignored
$HOME/bin/command
This point can however be bypassed by using envsubst and running it as
$ bash --rpm-requires <(<script envsubst)
However, if you use shellcheck, you most likely quoted this and it will still be ignored due to point 2
So if you want to use check if your scripts are all there, you can do something like:
while IFS='' read -r app; do
[ "${app%%(*}" == "executable" ] || continue
app="${app#*(}"; app="${app%)}";
if [ "$(type -t "${app}")" != "builtin" ] && \
! [ -x "$(command -v "${app}")" ]
then
echo "${app}: missing application"
fi
done < <(bash --rpm-requires <(<"$0" envsubst) )
If your script contains files that are sourced that might contain various functions and other important definitions, you might want to do something like
bash --rpm-requires <(cat source1 source2 ... <(<script.sh envsubst))
Based #czvtools’ answer, I added some extra checks to filter out bad values:
#!/usr/bin/fish
if test "$argv[1]" = ""
echo "Give path to command to be tested"
exit 1
end
set commands (cat $argv \
| sed -e 's/\"/./g' -e "s/'[^']*'//g" -e 's/"[^"]*"//g' \
| sed -e "s/^[[:space:]]*#.*\$//" -e "s/\([^\\]\)#[^\"']*\$/\1/" \
| sed -e "s/&&/;/g" -e "s/||/;/g" | tr ";|" "\n\n" \
| awk '{print $1}' | sort -u)
for command in $commands
if command -q -- $command
set -a resolved (realpath (which $command))
end
end
set resolved (string join0 $resolved | sort -z -u | string split0)
for command in $resolved
echo $command
end
This question already has answers here:
Displaying only single most recent line of a command's output
(2 answers)
Closed 5 years ago.
I been trying to print a variable in the same time for a scrip that pretends automatize a process the content is the output of this
sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP
so i put this in a while loop
do
out= sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP
echo -n "$out"
sleep 1
done
i read other questions here and i try with different option like echo -ne, echo -ne "$out" \r, printf "\r" or printf "%s" and no luck with no one, all the other example don't have a variable to print just counter o system variables
Update
it seems to appear that the echo -n repeat $out in the same line, if out="this is a test" the output of echo -n is "this is a test this is a test this is a test this is a test ...." maybe im missing some option ?
Update 2
sorry for the miss understood perhaps i was not very clear but what i want is overwrite the same line with the value of $out, the source of $out is the output of the aireplay-ng command that executes along with the script and i get the output with
the ouput is something like this
102415 packets (got 5 ARP requests and 15438 ACKs), sent 37085 packets...(499 pps)
but the number of ARP request is changing constantly
this code for example use echo -ne and overwrite in the same line
#!/bin/bash
for pc in $(seq 1 100); do
echo -ne "$pc%\033[0K\r"
sleep 1
done
the output of this is like a percent indicator that shows "10%" and going instead of "1% 2% 3% 4% 5% .." in the same line and i already try like this but with no luck
if you are trying to execute the sed Please use
`sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP`
First of all you are assigning a value of bash command wrongly to variable.
out=$(sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP)
Then you can print all your output in one line as you wrote:-
echo -n $out
The recent addendum to your question reads like you're miscommunicating your intent: this is a test this is a test this is a test is what a plain reading of your question indicates you to be asking for (printing this is a test over and over in a loop without newlines, after all, can be expected to do nothing else); why you'd describe this in a context that makes it sound like a bug is thus surprising.
If you want to send the cursor back to the beginning of your current line and overwrite that line, that might be something like the following:
#!/bin/bash
# ^^^^ not /bin/sh; this enables bash extensions
# ask the shell to keep $COLUMNS up-to-date
shopt -s checkwinsize
# defaults to 80-character terminal width, but uses $COLUMNS if available
printf "%-${COLUMNS:-80}s\r" "$out"`
...which prints your string, pads out to 80 characters with spaces, and then returns the cursor to the beginning of the line, such that the next thing you write will overwrite that string.
Of course, if you print that line and then return to a shell prompt, the prompt will start at the beginning of the same line and overwrite the text, so be sure to follow up with an echo.
I have a little error with a script I wrote in bash and I can't figure out what's I'm doing wrong
note that I'm using this script for thousands of calculations and this error happened only a few times (like 20 or so), but it still happened
What the script does is this: basically it takes in input a web page that I got from a site with the utility w3m and it counts all the occurrences of the words in it... After it orders them from the most common to the ones that occur only once
this is the code:
#!/bin/bash
# counts the numbers of words from specific sites #
# writes in a file the occurrences ordered from the most common #
touch check # file used to analyze the occurrences
touch distribution # final file ordered
page=$1 # the web page that needs to be analyzed
occurrences=$2 # temporary file for the occurrences
dictionary=$3 # dictionary used for another purpose (ignore this)
# write the words one by column
cat $page | tr -c [:alnum:] "\n" | sed '/^$/d' > check
# lopp to analyze the words
cat check | while read words
do
word=${words}
strlen=${#word}
# ignores blacklisted words or small ones
if ! grep -Fxq $word .blacklist && [ $strlen -gt 2 ]
then
# if the word isn't in the file
if [ `egrep -c -i "^$word: " $occurrences` -eq 0 ]
then
echo "$word: 1" | cat >> $occurrences
# else if it is already in the file, it calculates the occurrences
else
old=`awk -v words=$word -F": " '$1==words { print $2 }' $occurrences`
### HERE IS THE ERROR, EITHER THE LET OR THE SED ###
let "new=old+1"
sed -i "s/^$word: $old$/$word: $new/g" $occurrences
fi
fi
done
# orders the words
awk -F": " '{print $2" "$1}' $occurrences | sort -rn | awk -F" " '{print $2": "$1}' > distribution
# ignore this, not important
grep -w "1" distribution | awk -F ":" '{print $1}' > temp_dictionary
for line in `cat temp_dictionary`
do
if ! grep -Fxq $line $dictionary
then
echo $line >> $dictionary
fi
done
rm check
rm temp_dictionary
this is the error: (I'm translating it, so it could be different in english)
./wordOccurrences line:30 let:x // where x is a number, usually 9 or 10 (but also 11, 13, etc)
1: syntax error in the espression (the error token is 1)
sed: expression -e #1, character y: command 's' not terminated // where y is another number (this one is also usually 9 or 10) with y being different from x
EDIT:
Talking with kev it looks like it's a newline problem
I added an echo between let and sed to print the sed and it worked perfectly for like 5 to 10 minutes until that error. Usually the sed without error looked like this:
s/^CONSULENTI: 6$/CONSULENTI: 7/g
but when I got the error it was like this:
s/^00145: 1
1$/00145: 4/g
how to fix this?
If you get a new line in $old, it means awk prints two lines so there is a duplicate in $occurences.
The script seems complicated to count words, and not efficient because it launches many processes and process file in a loop ;
maybe you can do something similar with
sort | uniq -c
You should also consider that your case-insensitivity is not consistent throughout the program. I created a page with just "foooo" in it and ran the program, then created one with "Foooo" in it and ran the program again. The 'old=`awk...' line sets 'old' to the empty string because awk is matching case sensitively. This results in the occurrences file not being updated. The subsequent sed and possibly some of the greps are also case sensitive.
This may not be the only error since it doesn't explain the error message you saw, but it is an indication that the same word with different capitalization will be handled erroneously by your script.
The following would separate the words, lowercase them, and then remove the ones smaller than three characters:
tr -cs '[:alnum:]' '\n' <foo | tr '[:upper:]' '[:lower:]' | egrep -v '^.{0,2}$'
Using this at the front of your script would mean that the rest of the script would not have to be case insensitive to be correct.