I'm using ssh to connect to a remote machine and read a log file there. From that log file, based on some tokens, I extract specific logs and store it in a variable. Every log is in new line in the log file and the data can contain any character including white space.
array=("$(egrep "UserComments/propagateBundle-2013-10-19--04:42:13|UserComments/propagateBundle-2013-10-19--04:38:36|UserComments/propagateBundle-2013-10-19--04:34:24" <path>/propagateBundle.log)")
echo ${array[0]}
echo "$array"
First echo prints complete output in one line separated by white space while the other prints outputs in new line. Problem, is, I'm not able to save this output as an array. I tried this:
newArray=("$array")
max=${#newArray[#]}
echo $max
But echoing 'max' yields '1' on the screen. How can I save the output in an array? I also tried using
IFS=\`\n`
but could not get the data in an array.
EDIT
I used the solution given by Anubhav and it worked like charm. Now I faced a second issue. Since my data contains white spaces, so the array broke at white spaces and wrongly contained the one comments as multiple arrays. So, I used
IFS=\`\n`
and also used a $ symbol before backticks. Although this solves my problem, I still get an exception in the logs:
test.sh: line 11: n: command not found
Any suggestions?
Don't put quotes in the command substitution:
array=( $(egrep "UserComments/propagateBundle-2013-10-19--04:42:13|UserComments/propagateBundle-2013-10-19--04:38:36|UserComments/propagateBundle-2013-10-19--04:34:24" <path>/propagateBundle.log) )
With quotes as in your code whole output is treated as single string in the array.
I've used IFS=('\n') otherwise all "n" chars disappears from results and sort command doesn't work properly. See bellow, it is a customized llq output.
#!/bin/bash
IFS=('\n')
raw=(`llq -f %id %o %gu %gl %st %BS %c`)
echo
echo ${raw[*]} | grep "step(s)"
echo
echo ${raw[*]} | grep "Step"
echo ${raw[*]} | grep "\---*"
echo ${raw[*]} | grep "bgp-fn*" | sort -k5 -r
echo ${raw[*]} | grep "\---*"
echo ${raw[*]} | grep "Step"
echo
echo ${raw[*]} | grep "step(s)"
echo
Related
I need to assign the output of a command to a variable. The command I tried is:
#!/bin/bash
JAVA_PROCESSES=`jps -vl | grep -v 'sun.tools.jps.Jps' | grep -v 'hudson.remoting.jnlp.Main' | grep -v grep`
NUMBER_OF_JAVA_PROCESSES=`echo $JAVA_PROCESSES | wc -l`
echo $NUMBER_OF_JAVA_PROCESSES
echo $JAVA_PROCESSES
..
When I tried as in above, all java processes grepped are assigned to JAVA_PROCESSES variable in one line. Processes are not separated by new line. Therefore $NUMBER_OF_JAVA_PROCESSES always give 1 for me.
Also $NUMBER_OF_JAVA_PROCESSES show 1 even no processes are assigned to JAVA_PROCESSES due to the empty line in $JAVA_PROCESSES.
Please suggest a way to assign grepped processes separated by new line.
If the main thing you want is to know whether or not you got any at all, you could just test if the variable is empty:
java_procs=$(jps -vl | grep -v 'sun.tools.jps.Jps' | grep -v 'hudson.remoting.jnlp.Main' | grep -v grep)
if [ -z "$java_procs" ]; then
echo "No processes"
fi
Also, we can simplify the grep by using extended regex and just needing a single processes:
java_procs=$(jps -vl | grep -Ev 'sun.tools.jps.Jps|hudson.remoting.jnlp.Main|grep')
Assuming none of the lines output by jps can contain linebreaks themselves, we could get the count after that if we need it:
num_procs=$(printf '%s\n' "$java_procs" | wc -l)
The main problem you were running into is that you weren't quoting your variable, so echo $JAVA_PROCESSES was being expanded and then subject to word splitting, so your newlines were being "eaten" by the shell. You'd always have only one line which would be a space separated list of all the words in your JAVA_PROCESSES variable. To protect from word splitting you can quote the variable, as I did in my code above.
echo will also always add a line break at the end, which is good sometimes, and not so good sometimes, but you should be aware of it happening (that's why you would always get a count of 1 even when there were no processes).
This question already has answers here:
Displaying only single most recent line of a command's output
(2 answers)
Closed 5 years ago.
I been trying to print a variable in the same time for a scrip that pretends automatize a process the content is the output of this
sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP
so i put this in a while loop
do
out= sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP
echo -n "$out"
sleep 1
done
i read other questions here and i try with different option like echo -ne, echo -ne "$out" \r, printf "\r" or printf "%s" and no luck with no one, all the other example don't have a variable to print just counter o system variables
Update
it seems to appear that the echo -n repeat $out in the same line, if out="this is a test" the output of echo -n is "this is a test this is a test this is a test this is a test ...." maybe im missing some option ?
Update 2
sorry for the miss understood perhaps i was not very clear but what i want is overwrite the same line with the value of $out, the source of $out is the output of the aireplay-ng command that executes along with the script and i get the output with
the ouput is something like this
102415 packets (got 5 ARP requests and 15438 ACKs), sent 37085 packets...(499 pps)
but the number of ARP request is changing constantly
this code for example use echo -ne and overwrite in the same line
#!/bin/bash
for pc in $(seq 1 100); do
echo -ne "$pc%\033[0K\r"
sleep 1
done
the output of this is like a percent indicator that shows "10%" and going instead of "1% 2% 3% 4% 5% .." in the same line and i already try like this but with no luck
if you are trying to execute the sed Please use
`sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP`
First of all you are assigning a value of bash command wrongly to variable.
out=$(sed "s/Read/\n/g" /tmp/Air/test.txt | tail -1 test.txt | grep ARP)
Then you can print all your output in one line as you wrote:-
echo -n $out
The recent addendum to your question reads like you're miscommunicating your intent: this is a test this is a test this is a test is what a plain reading of your question indicates you to be asking for (printing this is a test over and over in a loop without newlines, after all, can be expected to do nothing else); why you'd describe this in a context that makes it sound like a bug is thus surprising.
If you want to send the cursor back to the beginning of your current line and overwrite that line, that might be something like the following:
#!/bin/bash
# ^^^^ not /bin/sh; this enables bash extensions
# ask the shell to keep $COLUMNS up-to-date
shopt -s checkwinsize
# defaults to 80-character terminal width, but uses $COLUMNS if available
printf "%-${COLUMNS:-80}s\r" "$out"`
...which prints your string, pads out to 80 characters with spaces, and then returns the cursor to the beginning of the line, such that the next thing you write will overwrite that string.
Of course, if you print that line and then return to a shell prompt, the prompt will start at the beginning of the same line and overwrite the text, so be sure to follow up with an echo.
I was trying to do these few operations/commands on a single line and assign it to a variable. I have it working about 90% of the way except for one part of it.
I was unaware you could do this, but I read that you can nest $(..) inside other $(..).... So I was trying to do that to get this working, but can't seem to get it the rest of the way.
So basically, what I want to do is:
1. Take the output of a file and assign it to a variable
2. Then pre-pend some text to the start of that output
3. Then append some text to the end of the output
4. And finally remove newlines and replace them with "\n" character...
I can do this just fine in multiple steps but I would like to try and get this working this way.
So far I have tried the following:
My 1st attempt, before reading about nested $(..):
MY_VAR=$(echo -n "<pre style=\"display:inline;\">"; cat input.txt | sed ':a;N;$!ba;s/\n/\\n/g'; echo -n "</pre>")
This one worked 99% of the way except there was a newline being added between the cat command's output and the last echo command. I'm guessing this is from the cat command since sed removed all newlines except for that one, maybe...?
Other tries:
MY_VAR=$( $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo -n "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g')
MY_VAR="$( echo $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g' )"
MY_VAR="$( echo "$(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>")" | sed ':a;N;$!ba;s/\n/\\n/g' )"
*Most these others were tried with and without the extra double-quotes surrounding the different $(..) parts...
I had a few other attempts, but they didn't have any luck either... On a few of the other attempts above, it seemed to work except sed was NOT inserting the replacement part of it. The output was correct for the most part, except instead of seeing "\n" between lines it just showed each of the lines smashed together into one line without anything to separate them...
I'm thinking there is something small I am missing here if anyone has any idea..?
*P.S. Does Bash have a name for the $(..) structure? It's hard trying to Google for that since it doesn't really search symbols...
You have no need to nest command substitutions here.
your_var='<pre style="display:inline;">'"$(<input.txt)"'</pre>'
your_var=${your_var//$'\n'/'\n'}
"$(<input.txt)" expands to the contents of input.txt, but without any trailing newline. (Command substitution always strips trailing newlines; printf '%s' "$(cat ...)" has the same effect, albeit less efficiently as it requires a subshell, whereas cat ... alone does not).
${foo//bar/baz} expands to the contents of the shell variable named foo, with all instances of bar replaced with baz.
$'\n' is bash syntax for a literal newline.
'\n' is bash syntax for a two-character string, beginning with a backslash.
Thus, tying all this together, it first generates a single string with the prefix, the contents of the file, and the suffix; then replaces literal newlines inside that combined string with '\n' two-character sequences.
Granted, this is multiple lines as implemented above -- but it's also much faster and more efficient than anything involving a command substitution.
However, if you really want a single, nested command substitution, you can do that:
your_var=$(printf '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
The printf %s combines its arguments without any delimiter between them
The sed operation adds a literal \n to the end of each line except the last
The tr -d '\n' operation removes literal newlines from the file
However, even this approach could be done more efficiently without the nesting:
printf -v your_var '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
...which has the printf assign its results directly to your_var, without any outer command substitution required (and thus saving the expense of the outer subshell).
SSH from Host A to a few hosts (only one listed below right now) using the SSH Key I generated and then go to a specific file, grep for a specific word with a date of yesterday .. then I want to email this output to myself.
It is sending an email but it is giving me the command as opposed to the output from the command.
#!/bin/bash
HOST="XXXXXXXXXXXXXXXXXX, XXXXXXXXXXXXX"
DATE=$(date -d "yesterday")
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
COUNT=$(echo "$INVALID" | wc -c)
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
This isn't properly an attempt to answer your question, but I think you should be aware of some fundamental problems with your code.
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
This assigns a simple string to the variable INVALID. Because of quoting issues, s/$/.\n/g is not quoted at all, and will probably be mangled by the shell. (You cannot nest single quotes -- the first single-quoted string extends from the first quote to the next one, and then WORD is outside of any quotes, followed by the next single-quoted string, etc.)
If your intent is to execute this as a command at this point, you are looking for a command substitution; with the multiple layers of uselessness peeled off, perhaps something like
INVALID=$(sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx)
which looks for a line matching WORD and $DATE and prints the match with a dot appended at the end -- I believe that's what your code boils down to, but without further insights into what this code is supposed to do, it's impossible to tell if this is what you actually need.
COUNT=$(echo "$INVALID" | wc -c)
This assigns a number to $COUNT. With your static definition of INVALID, the number will always be 62; but I guess that's not actually what you want here.
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
This attempts to execute that number as a command on a number of remote hosts (except the loop is over HOSTS and the variable containing the hosts is named just HOST). This cannot possibly be useful, unless you have a battery of commands named as natural numbers which do something useful on these remote hosts; but I think it's safe to assume that that is not what is supposed to be going on here (and if it was, it would absolutely be necessary to explain this in your question).
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
So EMAILTEXT is either an empty string or the value of INVALID. You assigned it to be a static string above, which is probably the source of your immediate question. But even if it was somehow assigned to a command on the local host, why do you need to visit remote hosts and execute something there? Or is your intent actually to execute the command on each remote host and obtain the output?
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
Piping into echo makes no sense at all, because it does not read its standard input. You should probably just have a newline after done; though a possibly more useful arrangement would be to have your loop produce output which we then pipe to mail.
Purely speculatively, perhaps something like the following is what you actually want.
for host in $HOSTS; do
ssh BLA#"$host" sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx |
grep . || echo INVALID
done | mail XXXXXXXXXXX.com
If you want to check that there is strictly more than one line of output (which is what the -gt 1 suggests) then this may need to be a little bit more complicated.
Your command substitution is not working. You should read up on how it works but here are the problem lines:
COUNT=$(echo "$INVALID" | wc -c)
[...]
ssh BLA#"$x" $COUNT
should be:
COUNT_CMD="'${INVALID} | wc -c'"
[...]
COUNT=$(ssh BLA#"$x" $COUNT_CMD)
This inserts the value of $INVALID into the string, and puts the whole thing in single quotes. The single quotes are necessary for the ssh call so the pipes aren't evaluated in the script but on the remote host. (COUNT is changed to COUNT_CMD for readability/clarity.)
EDIT:
I misread the question and have corrected my answer.
I have a little error with a script I wrote in bash and I can't figure out what's I'm doing wrong
note that I'm using this script for thousands of calculations and this error happened only a few times (like 20 or so), but it still happened
What the script does is this: basically it takes in input a web page that I got from a site with the utility w3m and it counts all the occurrences of the words in it... After it orders them from the most common to the ones that occur only once
this is the code:
#!/bin/bash
# counts the numbers of words from specific sites #
# writes in a file the occurrences ordered from the most common #
touch check # file used to analyze the occurrences
touch distribution # final file ordered
page=$1 # the web page that needs to be analyzed
occurrences=$2 # temporary file for the occurrences
dictionary=$3 # dictionary used for another purpose (ignore this)
# write the words one by column
cat $page | tr -c [:alnum:] "\n" | sed '/^$/d' > check
# lopp to analyze the words
cat check | while read words
do
word=${words}
strlen=${#word}
# ignores blacklisted words or small ones
if ! grep -Fxq $word .blacklist && [ $strlen -gt 2 ]
then
# if the word isn't in the file
if [ `egrep -c -i "^$word: " $occurrences` -eq 0 ]
then
echo "$word: 1" | cat >> $occurrences
# else if it is already in the file, it calculates the occurrences
else
old=`awk -v words=$word -F": " '$1==words { print $2 }' $occurrences`
### HERE IS THE ERROR, EITHER THE LET OR THE SED ###
let "new=old+1"
sed -i "s/^$word: $old$/$word: $new/g" $occurrences
fi
fi
done
# orders the words
awk -F": " '{print $2" "$1}' $occurrences | sort -rn | awk -F" " '{print $2": "$1}' > distribution
# ignore this, not important
grep -w "1" distribution | awk -F ":" '{print $1}' > temp_dictionary
for line in `cat temp_dictionary`
do
if ! grep -Fxq $line $dictionary
then
echo $line >> $dictionary
fi
done
rm check
rm temp_dictionary
this is the error: (I'm translating it, so it could be different in english)
./wordOccurrences line:30 let:x // where x is a number, usually 9 or 10 (but also 11, 13, etc)
1: syntax error in the espression (the error token is 1)
sed: expression -e #1, character y: command 's' not terminated // where y is another number (this one is also usually 9 or 10) with y being different from x
EDIT:
Talking with kev it looks like it's a newline problem
I added an echo between let and sed to print the sed and it worked perfectly for like 5 to 10 minutes until that error. Usually the sed without error looked like this:
s/^CONSULENTI: 6$/CONSULENTI: 7/g
but when I got the error it was like this:
s/^00145: 1
1$/00145: 4/g
how to fix this?
If you get a new line in $old, it means awk prints two lines so there is a duplicate in $occurences.
The script seems complicated to count words, and not efficient because it launches many processes and process file in a loop ;
maybe you can do something similar with
sort | uniq -c
You should also consider that your case-insensitivity is not consistent throughout the program. I created a page with just "foooo" in it and ran the program, then created one with "Foooo" in it and ran the program again. The 'old=`awk...' line sets 'old' to the empty string because awk is matching case sensitively. This results in the occurrences file not being updated. The subsequent sed and possibly some of the greps are also case sensitive.
This may not be the only error since it doesn't explain the error message you saw, but it is an indication that the same word with different capitalization will be handled erroneously by your script.
The following would separate the words, lowercase them, and then remove the ones smaller than three characters:
tr -cs '[:alnum:]' '\n' <foo | tr '[:upper:]' '[:lower:]' | egrep -v '^.{0,2}$'
Using this at the front of your script would mean that the rest of the script would not have to be case insensitive to be correct.