Using columns in bash - linux

I've used the column command to split some of my output into 3 different columns. Problem is with the final column, the filetype output is being split into a 4th and 5th column because of the spaces.
Can somebody tell me how to change my code so that output stays under the Filetype column?
list_files()
{
if [ "$(ls -A ~/.junkdir)" ]
then
filesdir=/home/student/.junkdir/*
echo "Listing files in Junk Directory"
output="FILENAME SIZE(BYTES) TYPE \n\n---------------- ---------------- ------------------- "
for listed_file in $filesdir
do
file_name=$(basename "file $listed_file" | cut -d ' ' -f1)
file_size=$(du --bytes $listed_file | awk '{print $1}')
file_type=$(file $listed_file | cut -d ' ' -f2-)
output="$output\n${file_name} ${file_size} ${file_type}\n"
done
echo -ne $output | column -t
else
echo 'Junk directory is empty'
fi
}
The output at the moment..
Listing files in Junk Directory
FILENAME SIZE(BYTES) TYPE
---------------- ---------------- -------------------
files.txt 216 ASCII text
forLoop 401 Bourne-Again shell script,
ASCII text executable

I rarely give full solution, but it seems you are really stuck.
list_files2()
{
filesdir=/home/student/.junkdir/*
printf "FILENAME\1SIZE(BYTES)\1TYPE\1\n\n----------------\1----------------\1-------------------\n"
for listed_file in $filesdir
do
file_name=$(basename "file $listed_file" | cut -d ' ' -f1)
file_size=$(du --bytes $listed_file | awk '{print $1}')
file_type=$(file $listed_file | cut -d ' ' -f2-)
printf "%s\1%s\1%s\n" "${file_name}" "${file_size}" "${file_type}"
done
}
list_files()
{
if [ "$(ls -A ~/.junkdir)" ]
then
echo "Listing files in Junk Directory"
list_files2 | column -t -s $'\1'
else
echo 'Junk directory is empty'
fi
}
I slightly reorganized your code and made some other changes as well. I will explain what I did.
$'\1' is the 0x01 char. Even though I proposed to use $'\0', it's weird that my version of column has weird interaction with it appearing in the input. But in shell scripting practice, it's generally a bad idea to assume blank separator. In your case, you got caught by space, which is reasonable, because white space has so much overloaded meaning, you cannot prevent it from appearing in a human readable text. The solution to this is to consider using exotic chars like 0x00 or 0x01 as separator instead, which is almost always the case that it won't show up as a part of the text. So it's very safe to use and in fact, it's common in portable shell scripting to use 0x00 as separator.
Do not concatenate string like you did. In fact, there are couple problems with it.
one is you keep concatenating string while you don't really need the intermediate result.
another biteback is what if inside of your text, strings like \n exist and it should be interpreted as literal? echo -e is not going to distinguish that. yet another SO question on the road.
using printf in fact is more flavorable in shell, though I use echo a lot myself as well. Here the benefit of using printf is evident.
I don't have your files so I didn't try it, though I would expect it works. Let me know if there're glitches here and there.

Perhaps you can try
output="$output\n${file_name}\t${file_size}\t${file_type}\n"
...
echo -ne $output

Related

Linux Scripting with Spaces in Filenames

I am currently working with a vendor-provided software that is trying to handle sending attachment files to another script that will text-extract from the listed file. The script fails when we receive files from an outside source that contain spaces, as the vendor-supplied software does not surround the filename in quotes - meaning when the text-extraction script is run, it receives a filename that will split apart on the space and cause an error on the extractor script. The vendor-provided software is not editable by us.
This whole process is designed to be an automated transfer, so having this wrench that could be randomly thrown into the gears is an issue.
What we're trying to do, is handle the spaced name in our text extractor script, since that is the piece we have some control over. After a quick Google, it seems like changing the IFS value for the script would be the quick solution, but unfortunately, that script would take effect after the extensions have already mutilated the incoming data.
The script I'm using takes in a -e value, a -i value, and a -o value. These values are sent from the vendor supplied script, which I have no editing control over.
#!/bin/bash
usage() { echo "Usage: $0 -i input -o output -e encoding" 1>&2; exit 1; }
while getopts ":o:i:e:" o; do
case "${o}" in
i)
inputfile=${OPTARG}
;;
o)
outputfile=${OPTARG}
;;
e)
encoding=${OPTARG}
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
...
...
<Uses the inputfile, outputfile, and encoding variables>
I admit, there may be pieces to this I don't fully understand, and it could be a simple fix, but my end goal is to be able to extract -o, -i, and -e that all contain 1 value, regardless of the spaces within each section. I can handle quoting the script after I can extract the filename value
The script fragment that you have posted does not have any issues with spaces in the arguments.
The following, for example, does not need quoting (since it's an assignment):
inputfile=${OPTARG}
All other uses of $inputfile in the script should be double quoted.
What matters is how this script is called.
This would fail and would assign only hello to the variable inputfile:
$ ./script.sh -i hello world.txt
The string world.txt would prompt the getopts function to stop processing the command line and the script would continue with the shift (world.txt would be left in $1 afterwards).
The following would correctly assign the string hello world.txt to inputfile:
$ ./script.sh -i "hello world.txt"
as would
$ ./script.sh -i hello\ world.txt
The following script uses awk to split the arguments while including spaces in the file names. The arguments can be in any order. It does not handle multiple consecutive spaces in an argument, it collapses them to one.
#!/bin/bash
IFS=' '
str=$(printf "%s" "$*")
istr=$(echo "${str}" | awk 'BEGIN {FS="-i"} {print $2}' | awk 'BEGIN {FS="-o"} {print $1}' | awk 'BEGIN {FS="-e"} {print $1}')
estr=$(echo "${str}" | awk 'BEGIN {FS="-e"} {print $2}' | awk 'BEGIN {FS="-o"} {print $1}' | awk 'BEGIN {FS="-i"} {print $1}')
ostr=$(echo "${str}" | awk 'BEGIN {FS="-o"} {print $2}' | awk 'BEGIN {FS="-e"} {print $1}' | awk 'BEGIN {FS="-i"} {print $1}')
inputfile=""${istr}""
outputfile=""${ostr}""
encoding=""${estr}""
# call the jar
There was an issue when calling the jar where Java threw a MalformedUrlException on a filename with a space.
So after reading through the commentary, we decided that although it may not be the right answer for every scenario, the right answer for this specific scenario was to extract the pieces manually.
Because we are building this for a pre-built script passing to it, and we aren't updating that script any time soon, we can accept with certainty that this script will always receive a -i, -o, and -e flag, and there will be spaces between them, which causes all the pieces passed in to be stored in different variables in $*.
And we can assume that the text after a flag is the response to the flag, until another flag is referenced. This leaves us 3 scenarios:
The variable contains one of the flags
The variable contains the first piece of a parameter immediately after the flag
The variable contains part 2+ of a parameter, and the space in the name was interpreted as a split, and needs to be reinserted.
One of the other issues I kept running into was trying to get string literals to equate to variables in my IF statements. To resolve that issue, I pre-stored all relevant data in array variables, so I could test $variable == $otherVariable.
Although I don't expect it to change, we also handled what to do if the three flags appear in a different order than we anticipate (Our assumption was that they list as i,o,e... but we can't see excatly what is passed). The parameters are dumped into an array in the order they were read in, and a parallel array tracks whether the items in slots 0,1,2 relate to i,o,e.
The final result still has one flaw: if there is more than one consecutive space in the filename, the whitespace is trimmed before processing, and I can only account for one space. But saying as we processed over 4000 files before encountering one with a space, I find it unlikely with the naming conventions that we would encounter something with more than one space.
At that point, we would have to be stepping in for a rare intervention anyways.
Final code change is as follows:
#!/bin/bash
IFS='|'
position=-1
ioeArray=("" "" "")
previous=""
flagArr=("-i" "-o" "-e" " ")
ioePattern=(0 1 2)
#echo "for loop:"
for i in $*; do
#printf "%s\n" "$i"
if [ "$i" == "${flagArr[0]}" ] || [ "$i" == "${flagArr[1]}" ] || [ "$i" == "${flagArr[2]}" ]; then
((position += 1));
previous=$i;
case "$i" in
"${flagArr[0]}")
ioePattern[$position]=0
;;
"${flagArr[1]}")
ioePattern[$position]=1
;;
"${flagArr[2]}")
ioePattern[$position]=2
;;
esac
continue;
fi
if [[ $previous == "-"* ]]; then
ioeArray[$position]=${ioeArray[$position]}$i;
else
ioeArray[$position]=${ioeArray[$position]}" "$i;
fi
previous=$i;
done
echo "extracting (${ioeArray[${ioePattern[0]}]}) to (${ioeArray[${ioePattern[1]}]}) with (${ioeArray[${ioePattern[2]}]}) encoding."
inputfile=""${ioeArray[${ioePattern[0]}]}"";
outputfile=""${ioeArray[${ioePattern[1]}]}"";
encoding=""${ioeArray[${ioePattern[2]}]}"";

redirect output from one function to another

I'm trying to create a pipeline from user input, but when I redirect the output I'm getting a output with no newlines and it's just one huge single line.Here's the code :
42 function stack(){
43 echo $(history|tail -1|cut -d" " -f5-|cut -d "|" -f1) >> ~/commands
44 local last=$(tail -1 ~/commands)
45 echo $(eval $last) >> ~/output
46 }
Is there a better way to pipe the output from this to a file ? Echo seems to corrupt the output.
I'm not sure to understand the purpose of cuts, but quote are missing around $() so the output is split into words with IFS
echo "$(eval "$last")"
maybe cut -c8- is safer than cut -d" " -f5- for history entries with a number with number of digits different from 3.
also cut -d"|" -f1 can fail if | is used as literal for example echo '|'.
Maybe you can look at Even designators in bash manual : in interactive bash following will run the last command
$ !-1

Mail output with Bash Script

SSH from Host A to a few hosts (only one listed below right now) using the SSH Key I generated and then go to a specific file, grep for a specific word with a date of yesterday .. then I want to email this output to myself.
It is sending an email but it is giving me the command as opposed to the output from the command.
#!/bin/bash
HOST="XXXXXXXXXXXXXXXXXX, XXXXXXXXXXXXX"
DATE=$(date -d "yesterday")
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
COUNT=$(echo "$INVALID" | wc -c)
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
This isn't properly an attempt to answer your question, but I think you should be aware of some fundamental problems with your code.
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
This assigns a simple string to the variable INVALID. Because of quoting issues, s/$/.\n/g is not quoted at all, and will probably be mangled by the shell. (You cannot nest single quotes -- the first single-quoted string extends from the first quote to the next one, and then WORD is outside of any quotes, followed by the next single-quoted string, etc.)
If your intent is to execute this as a command at this point, you are looking for a command substitution; with the multiple layers of uselessness peeled off, perhaps something like
INVALID=$(sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx)
which looks for a line matching WORD and $DATE and prints the match with a dot appended at the end -- I believe that's what your code boils down to, but without further insights into what this code is supposed to do, it's impossible to tell if this is what you actually need.
COUNT=$(echo "$INVALID" | wc -c)
This assigns a number to $COUNT. With your static definition of INVALID, the number will always be 62; but I guess that's not actually what you want here.
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
This attempts to execute that number as a command on a number of remote hosts (except the loop is over HOSTS and the variable containing the hosts is named just HOST). This cannot possibly be useful, unless you have a battery of commands named as natural numbers which do something useful on these remote hosts; but I think it's safe to assume that that is not what is supposed to be going on here (and if it was, it would absolutely be necessary to explain this in your question).
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
So EMAILTEXT is either an empty string or the value of INVALID. You assigned it to be a static string above, which is probably the source of your immediate question. But even if it was somehow assigned to a command on the local host, why do you need to visit remote hosts and execute something there? Or is your intent actually to execute the command on each remote host and obtain the output?
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
Piping into echo makes no sense at all, because it does not read its standard input. You should probably just have a newline after done; though a possibly more useful arrangement would be to have your loop produce output which we then pipe to mail.
Purely speculatively, perhaps something like the following is what you actually want.
for host in $HOSTS; do
ssh BLA#"$host" sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx |
grep . || echo INVALID
done | mail XXXXXXXXXXX.com
If you want to check that there is strictly more than one line of output (which is what the -gt 1 suggests) then this may need to be a little bit more complicated.
Your command substitution is not working. You should read up on how it works but here are the problem lines:
COUNT=$(echo "$INVALID" | wc -c)
[...]
ssh BLA#"$x" $COUNT
should be:
COUNT_CMD="'${INVALID} | wc -c'"
[...]
COUNT=$(ssh BLA#"$x" $COUNT_CMD)
This inserts the value of $INVALID into the string, and puts the whole thing in single quotes. The single quotes are necessary for the ssh call so the pipes aren't evaluated in the script but on the remote host. (COUNT is changed to COUNT_CMD for readability/clarity.)
EDIT:
I misread the question and have corrected my answer.

Bash loop is not working — cannot find command "[0%"

I just wrote a ping sweep script in Bash this morning, and guess what: it's not working. Can you please check what it is that I'm missing.
Here's the script:
for i in `seq 1 255`
do
if ["$(ping -c1 -W1 -n 192.168.1.$i | grep '%' | cut -d',' -f3 | cut -d' ' -f2)" -eq "0%"]
then echo "Host live"
else echo "Host down"
fi
done
And here's the error:
bash: [0%: command not found
Host down
bash: [100%: command not found
Host down
My purpose is to make a ping sweep program which scans the range 192.168.1.1-255 and it notifies the host's status. I know about nmap but just wanted to learn skills in Bash so I made this one. Please try to tell what the error meant. I mean to what command it's referring "command not found"?
The ping command returns error code if there was any problem, so you do not need to parse the output:
for i in {1..255}
do
if ping -c1 -W1 -n "192.168.1.$i"
then
echo 'Host live'
else
echo 'Host down'
fi
done
Primary diagnosis
The [ command needs a space after its name, just like the rm command needs a space after its name and the ls command does, and … The [ command also requires its last argument to be ], spelled thus, so there needs to be a space before that, too.
You have:
if ["$(ping -c1 -W1 -n 192.168.1.$i | grep '%' | cut -d',' -f3 | cut -d' ' -f2)" -eq "0%"]
At minimum, you need:
if [ "$(ping -c1 -W1 -n 192.168.1.$i | grep '%' | cut -d',' -f3 | cut -d' ' -f2)" -eq "0%" ]
Secondary issues
Note that 'at minimum' means, amongst other things, that I've not spent time analyzing why you are executing the complex sequence of 4 commands in the test condition, or looked for ways to cut that down to two (using grep and cut twice suggests that sed or a more powerful tool would be better). I griped about the formatting in the original version of the question, where the loop (it isn't a nested loop, incidentally — or it isn't in the code shown) was all on one line thanks to Bash flattening it in history. My version of the code would have far fewer semicolons in it, for example. The -eq operator in [ is for testing the equality of numbers (the converse convention applies in Perl, where eq is for testing strings and == tests numbers). Note that POSIX standard [ (aka test) does not support == as a synonym for =, though Bash does. It isn't entirely clear that "0%" is OK as an argument for numeric comparison. Many programs would not object — the zero can be converted and the residue doesn't matter; others might decide legitimately to complain that the whole string could not be converted, so it is erroneous. Careful code wouldn't risk the disconnect.
See Steven Penny's answer for a more thorough rewrite of the code. My answer remains a valid diagnosis of the immediate problem of not being able to find commands named [0% and [100%.

Line from bash command output stored in variable as string

I'm trying to find a solution to a problem analog to this one:
#command_A
A_output_Line_1
A_output_Line_2
A_output_Line_3
#command_B
B_output_Line_1
B_output_Line_2
Now I need to compare A_output_Line_2 and B_output_Line_1 and echo "Correct" if they are equal and "Not Correct" otherwise.
I guess the easiest way to do this is to copy a line of output in some variable and then after executing the two commands, simply compare the variables and echo something.
This I need to implement in a bash script and any information on how to get certain line of output stored in a variable would help me put the pieces together.
Also, it would be cool if anyone can tell me not only how to copy/store a line, but probably just a word or sequence like : line 1, bytes 4-12, stored like string in a variable.
I am not a complete beginner but also not anywhere near advanced linux bash user. Thanks to any help in advance and sorry for bad english!
An easier way might be to use diff, no?
Something like:
command_A > command_A.output
command_B > command_B.output
diff command_A.output command_B.output
This will work for comparing multiple strings.
But, since you want to know about single lines (and words in the lines) here are some pointers:
# first line of output of command_A
command_A | head -n 1
The -n 1 option says only to use the first line (default is 10 I think)
# second line of output of command_A
command_A | head -n 2 | tail -n 1
that will take the first two lines of the output of command_A and then the last of those two lines. Happy times :)
You can now store this information in a variable:
export output_A=`command_A | head -n 2 | tail -n 1`
export output_B=`command_B | head -n 1`
And then compare it:
if [ "$output_A" == "$output_B" ]; then echo 'Correct'; else echo 'Not Correct'; fi
To just get parts of a string, try looking into cut or (for more powerful stuff) sed and awk.
Also, just learing a good general purpose scripting language like python or ruby (even perl) can go a long way with this kind of problem.
Use the IFS (internal field separator) to separate on newlines and store the outputs in an array.
#!/bin/bash
IFS='
'
array_a=( $(./a.sh) )
array_b=( $(./b.sh) )
if [ "${array_a[1]}" = "${array_b[0]}" ]; then
echo "CORRECT"
else
echo "INCORRECT"
fi

Resources