I have a shell script, which gets bunch of values. I am segregating it based on a Delimiter(,). Now i want to print it one by one inside a for loop.
For E.g
var=/a/b/c,d/e/f/,x/y/z
for i in $(echo $var | sed "s/,/ /g")
do
echo $i
done
Output is coming empty, Expected output is
/a/b/c
d/e/f/
x/y/z
You don't need a loop.
sed 's/,/\n/g' <<< "$var"
sed 'y/,/\n/' <<< "$var"
tr ',' '\n' <<< "$var"
echo "${var//,/$'\n'}"
they all yield the desired output.
You could read it into an array. This is quite readable:
var=/a/b/c,d/e/f/,x/y/z
IFS=, read -a paths <<<"$var"
for p in "${paths[#]}"; do echo "$p"; done
/a/b/c
d/e/f/
x/y/z
Just to add another option -
var="/a/b/c,d/e/f/,x/y/z"
while IFS=, read a b c
do printf "a=$a b=$b c=$c\n"
done <<< "$var"
a=/a/b/c b=d/e/f/ c=x/y/z
Doesn't need a loop - works fine as
$: IFS=, read a b c <<< "$var"
$: printf "a=$a b=$b c=$c\n"
a=/a/b/c b=d/e/f/ c=x/y/z
... just wanted to show the loop structure in case it helped.
Related
I am trying to write bash script and how can i take values defined in variable in pipe delimiter format and print them.
Below is the algorithm i am thinking, but not sure about bash commands.
#!/bin/bash
variable1="A|B|C"
if [ ! -z "$variable1" ]
count=#should return 3 as there are 3 values (A|B|C)
fi
while count>0; do
derivedvariable=#should get A, B, C in loop
print $derivedvariable
done
output should be:
A
B
C
Use array to store elements of your input delimited by a character:
s='A|B|C'
IFS='|' read -ra arr <<< "$s"
# length:
echo "${#arr[#]}"
# looping array:
for i in "${arr[#]}"; do
echo "$i"
done
Output:
3
A
B
C
Use IFS to specify the field separator when splitting words after variable expansion.
variable1='A|B|C'
IFS='|'
for i in $variable1
do
echo "$i"
done
Use grep or tr:
grep -o "[^|]*" <<< "${variable1}"
tr '|' '\n' <<< "${variable1}"
I have a variable diagnosisCodes having value as shown below :
echo $diagnosisCodes
{"code":"some code1","codeSet":"some set1","description":"some description1"} {"code":"some code2","codeSet":"some set2","description":"some description2"} {"code":"some code3","codeSet":"some set3","description":"some description3"}
I want to split these using delimiter "} {", so that I get 3 different values and then store them in an array (in bash).
I found the solution using the below command (suggested by Incrivel Monstro Verde in the comments):
IFS='|' read -r -a array <<< $(echo $string | sed 's/{"code/|"code/g;s/\"}/"/g')
But this solution would fail if my string has "codeSet" attribute which comes before the "code" attribute.
You can use read:
#!/bin/bash
string='{"code":"some code1","codeSet":"some set1","description":"some description1"} {"code":"some code2","codeSet":"some set2","description":"some description2"} {"code":"some code3","codeSet":"some set3","description":"some descripti
IFS='{' read -r -a array <<< $(echo $string | sed 's/}//g')
echo "${array[1]}"
echo "${array[2]}"
echo "${array[3]}"
Try this out
#!/bin/bash
string='{"code":"some code1","codeSet":"some set1","description":"some description1"} {"code":"some code2","codeSet":"some set2","description":"some description2"} {"code":"some code3","codeSet":"some set3","description":"some description3"}'
var="${string//\} {/$'|'}" #replace '} {' with '|' , works for one space only between '} {'
IFS='|' read -r -a array <<< $(echo $var | sed 's/{//g;s/}//g')
for i in "${array[#]}"
do
echo $i
done
This is my problem. In bash 3:
$ test='One "This is two" Three'
$ set -- $test
$ echo $2
"This
How to get bash to understand the quotes and return $2 as This is two and not "This? Unfortunately I cannot alter the construction of the variable called test in this example.
The reason this happens is because of the order in which the shell parses the command line: it parses (and removes) quotes and escapes, then replaces variable values. By the time $test gets replaced with One "This is two" Three, it's too late for the quotes to have their intended effect.
The simple (but dangerous) way to do this is by adding another level of parsing with eval:
$ test='One "This is two" Three'
$ eval "set -- $test"
$ echo "$2"
This is two
(Note that the quotes in the echo command are not necessary, but are a good general practice.)
The reason I say this is dangerous is that it doesn't just go back and reparse for quoted strings, it goes back and reparses everything, maybe including things you didn't want interpreted like command substitutions. Suppose you had set
$ test='One `rm /some/important/file` Three'
...eval will actually run the rm command. So if you can't count on the contents of $test to be "safe", do not use this construct.
BTW, the right way to do this sort of thing is with an array:
$ test=(One "This is two" Three)
$ set -- "${test[#]}"
$ echo "$2"
This is two
Unfortunately, this requires control of how the variable is created.
Now we have bash 4 where it's possible to do something like that:
#!/bin/bash
function qs_parse() {
readarray -t "$1" < <( printf "%s" "$2"|xargs -n 1 printf "%s\n" )
}
tab=' ' # tabulation here
qs_parse test "One 'This is two' Three -n 'foo${tab}bar'"
printf "%s\n" "${test[0]}"
printf "%s\n" "${test[1]}"
printf "%s\n" "${test[2]}"
printf "%s\n" "${test[3]}"
printf "%s\n" "${test[4]}"
Outputs, as expected:
One
This is two
Three
-n
foo bar # tabulation saved
Actually, I am not sure but it's probably possible to do that in older bash like that:
function qs_parse() {
local i=0
while IFS='' read -r line || [[ -n "$line" ]]; do
parsed_str[i]="${line}"
let i++
done < <( printf "%s\n" "$1"|xargs -n 1 printf "%s\n" )
}
tab=' ' # tabulation here
qs_parse "One 'This is two' Three -n 'foo${tab}bar'"
printf "%s\n" "${parsed_str[0]}"
printf "%s\n" "${parsed_str[1]}"
printf "%s\n" "${parsed_str[2]}"
printf "%s\n" "${parsed_str[3]}"
printf "%s\n" "${parsed_str[4]}"
The solution to this problem is to use xargs (eval free).
It retains double quoted strings together:
$ test='One "This is two" Three'
$ IFS=$'\n' arr=( $(xargs -n1 <<<"$test") )
$ printf '<%s>\n' "${arr[#]}"
<One>
<This is two>
<Three>
Of course, you can set the positional arguments with that array:
$ set -- "${arr[#]}"
$ echo "$2"
This is two
I wrote a couple native bash functions to do this: https://github.com/mblais/bash_ParseFields
You can use the ParseFields function like this:
$ str='field1 field\ 2 "field 3"'
$ ParseFields -d "$str" a b c d
$ printf "|%s|\n|%s|\n|%s|\n|%s|\n" "$a" "$b" "$c" "$d"
|field1|
|field 2|
|field 3|
||
The -d option to ParseFields removes any surrounding quotes and interprets backslashes from the parsed fields.
There is also a simpler ParseField function (used by ParseFields) that parses a single field at a specific offset within a string.
Note that these functions cannot parse a stream, only a string. The IFS variable can also be used to specify field delimiters besides whitespace.
If you require that unescaped apostrophes may appear in unquoted fields, that would require a minor change - let me know.
test='One "This is two" Three'
mapfile -t some_args < <(xargs -n1 <<<"$test")
echo "'${some_args[0]}'" "'${some_args[1]}'" "'${some_args[2]}'"
output:
'One' 'This is two' 'Three'
Following is the shell script to read all the DSF present in the box. But since the line is having spaces, it is displaying them in different lines.
For those of you who dont understand ioscan -m dsf, replace it by ls -ltr, then the output is such that the permission and names are displayed in different line, but i want them in the same line.
#!/usr/bin/ksh
for a in `ioscan -m dsf`
do
echo $a
done
The for loop is not designed to loop over "lines". Instead it loops over "words".
Short simplified terminology: "lines" are things separated by newlines. "words" are things separated by spaces. in bash lingo "words" are called "fields".
The idiomatic way to loop over lines is to use a while loop in combination with read.
ioscan -m dsf | while read -r line
do
printf '%s\n' "$line"
done
Note that the while loop is in a subshell because of the pipe. This can cause some confusion with variable scope. In bash you can work around this by using process substitution.
while read -r line
do
printf '%s\n' "$line"
done < <(ioscan -m dsf)
But now the "generator" (ioscan in this example) is in a subshell.
For more information about the subshell problematic in loops see http://mywiki.wooledge.org/BashFAQ/024
If you insist on using a for loop to loop over lines you have to change the value of $IFS to only newline. IFS is short for Internal Field Separator. Usually $IFS contains a space, a tab, and a newline.
Here is the typical way to do so:
OLDIFS="$IFS"
IFS=$'\n' # bash specific
for line in $(ioscan -m dsf)
do
printf '%s\n' "$line"
done
IFS="$OLDIFS"
(the bash specific part ($'\n') is called ANSI-C Quoting)
But beware many commands depends on some sane setting for $IFS. I do not recommend changing $IFS. Too often it will cause an endless nightmare of obscure bug hunting.
See also:
http://wiki.bash-hackers.org/syntax/ccmd/classic_for
http://wiki.bash-hackers.org/commands/builtin/read
http://mywiki.wooledge.org/IFS
http://mywiki.wooledge.org/SubShell
http://mywiki.wooledge.org/ProcessSubstitution
Using for
for l in $() performs word splitting based on IFS:
$ for l in $(printf %b 'a b\nc'); do echo "$l"; done
a
b
c
$ IFS=$'\n'; for l in $(printf %b 'a b\nc'); do echo "$l"; done
a b
c
IFS doesn't have to be set back if it is not used later.
for l in $() also performs pathname expansion:
$ printf %b 'a\n*\n' > file.txt
$ IFS=$'\n'
$ for l in $(<file.txt); do echo "$l"; done
a
file.txt
$ set -f; for l in $(<file.txt); do echo "$l"; done; set +f
a
*
If IFS=$'\n', linefeeds are stripped and collapsed:
$ printf %b '\n\na\n\nb\n\n' > file.txt
$ IFS=$'\n'; for l in $(<file.txt); do echo "$l"; done
a
b
$(cat file.txt) (or $(<file.txt)) also reads the whole file to memory.
Using read
Without -r backslashes are used for line continuation and removed before other characters:
$ cat file.txt
\1\\2\
3
$ cat file.txt | while read l; do echo "$l"; done
1\23
$ cat file.txt | while read -r l; do echo "$l"; done
\1\\2\
3
Characters in IFS are stripped from the start and end of lines but not collapsed:
$ printf %b '1 2 \n\t3\n' | while read -r l; do echo "$l"; done
1 2
3
$ printf %b ' 1 2 \n\t3\n' | while IFS= read -r l; do echo "$l"; done
1 2
3
If the last line doesn't end with a newline, read assigns l to it but exits before the body of the loop:
$ printf 'x\ny' | while read l; do echo $l; done
x
$ printf 'x\ny' | while read l || [[ $l ]]; do echo $l; done
x
y
If a while loop is in a pipeline, it is also in a subshell, so variables are not visible outside it:
$ x=0; seq 3 | while read l; do let x+=l; done; echo $x
0
$ x=0; while read l; do let x+=l; done < <(seq 3); echo $x
6
$ x=0; x=8 | x=9; echo $x
0
you need to use this basically IFS=$'\n' and grep -x instead of grep as it will work like a equal to operator instead of like operator.
I want to split a text with comma , not space in for foo in list. Suppose I have a CSV file CSV_File with following text inside it:
Hello,World,Questions,Answers,bash shell,script
...
I used following code to split it into several words:
for word in $(cat CSV_File | sed -n 1'p' | tr ',' '\n')
do echo $word
done
It prints:
Hello
World
Questions
Answers
bash
shell
script
But I want it to split the text by commas not spaces:
Hello
World
Questions
Answers
bash shell
script
How can I achieve this in bash?
Set IFS to ,:
sorin#sorin:~$ IFS=',' ;for i in `echo "Hello,World,Questions,Answers,bash shell,script"`; do echo $i; done
Hello
World
Questions
Answers
bash shell
script
sorin#sorin:~$
Using a subshell substitution to parse the words undoes all the work you are doing to put spaces together.
Try instead:
cat CSV_file | sed -n 1'p' | tr ',' '\n' | while read word; do
echo $word
done
That also increases parallelism. Using a subshell as in your question forces the entire subshell process to finish before you can start iterating over the answers. Piping to a subshell (as in my answer) lets them work in parallel. This matters only if you have many lines in the file, of course.
I think the canonical method is:
while IFS=, read field1 field2 field3 field4 field5 field6; do
do stuff
done < CSV.file
If you don't know or don't care about how many fields there are:
IFS=,
while read line; do
# split into an array
field=( $line )
for word in "${field[#]}"; do echo "$word"; done
# or use the positional parameters
set -- $line
for word in "$#"; do echo "$word"; done
done < CSV.file
kent$ echo "Hello,World,Questions,Answers,bash shell,script"|awk -F, '{for (i=1;i<=NF;i++)print $i}'
Hello
World
Questions
Answers
bash shell
script
Create a bash function
split_on_commas() {
local IFS=,
local WORD_LIST=($1)
for word in "${WORD_LIST[#]}"; do
echo "$word"
done
}
split_on_commas "this,is a,list" | while read item; do
# Custom logic goes here
echo Item: ${item}
done
... this generates the following output:
Item: this
Item: is a
Item: list
(Note, this answer has been updated according to some feedback)
Read: http://linuxmanpages.com/man1/sh.1.php
& http://www.gnu.org/s/hello/manual/autoconf/Special-Shell-Variables.html
IFS The Internal Field Separator that is used for word splitting
after expansion and to split lines into words with the read
builtin command. The default value is ``''.
IFS is a shell environment variable so it will remain unchanged within the context of your Shell script but not otherwise, unless you EXPORT it. ALSO BE AWARE, that IFS will not likely be inherited from your Environment at all: see this gnu post for the reasons and more info on IFS.
You're code written like this:
IFS=","
for word in $(cat tmptest | sed -n 1'p' | tr ',' '\n'); do echo $word; done;
should work, I tested it on command line.
sh-3.2#IFS=","
sh-3.2#for word in $(cat tmptest | sed -n 1'p' | tr ',' '\n'); do echo $word; done;
World
Questions
Answers
bash shell
script
You can use:
cat f.csv | sed 's/,/ /g' | awk '{print $1 " / " $4}'
or
echo "Hello,World,Questions,Answers,bash shell,script" | sed 's/,/ /g' | awk '{print $1 " / " $4}'
This is the part that replace comma with space
sed 's/,/ /g'
For me, use array split is simpler ref
IN="bla#some.com;john#home.com"
arrIN=(${IN//;/ })
echo ${arrIN[1]}
Using readarray(mapfile):
$ cat csf
Hello,World,Questions,Answers,bash shell,script
$ readarray -td, arr < csf
$ printf '%s\n' "${arr[#]}"
Hello
World
Questions
Answers
bash shell
script