I have comma separated (sometimes tab) text file as below:
parameters.txt:
STD,ORDER,ORDER_START.xml,/DML/SOL,Y
STD,INSTALL_BASE,INSTALL_START.xml,/DML/IB,Y
with below code I try to loop through the file and do something
while read line; do
if [[ $1 = "$(echo "$line" | cut -f 1)" ]] && [[ "$(echo "$line" | cut -f 5)" = "Y" ]] ; then
//do something...
if [[ $? -eq 0 ]] ; then
// code to replace the final flag
fi
fi
done < <text_file_path>
I wanted to update the last column of the file to N if the above operation is successful, however below approaches are not working for me:
sed 's/$f5/N/'
'$5=="Y",$5=N;{print}'
$(echo "$line" | awk '$5=N')
Update: Few considerations which need to be considered to give more clarity which i missed at first, apologies!
The parameters file may contain lines with last field flag as "N" as well.
Final flag needs to be update only if "//do something" code has successfully executed.
After looping through all lines i.e, after exiting "while loop" flags for all rows to be set to "Y"
perhaps invert the operations do processing in awk.
$ awk -v f1="$1" 'BEGIN {FS=OFS=","}
f1==$1 && $5=="Y" { // do something
$5="N"}1' file
not sure what "do something" operation is, if you need to call another command/script it's possible as well.
with bash:
(
IFS=,
while read -ra fields; do
if [[ ${fields[0]} == "$1" ]] && [[ ${fields[4]} == "Y" ]]; then
# do something
fields[4]="N"
fi
echo "${fields[*]}"
done < file | sponge file
)
I run that in a subshell so the effects of altering IFS are localized.
This uses sponge to write back to the same file. You need the moreutils package to use it, otherwise use
done < file > tmp && mv tmp file
Perhaps a bit simpler, less bash-specific
while IFS= read -r line; do
case $line in
"$1",*,Y)
# do something
line="${line%Y}N"
;;
esac
echo "$line"
done < file
To replace ,N at the end of the line($) with ,Y:
sed 's/,N$/,Y/' file
Related
why this code not work? What's the problem of passing $line to a function?
function a {
echo $1 | grep $2
}
while read -r line; do
a $line "LAN"
done < database.txt
Another question, i have to overwrite line by line a txt file possibly using sed command.But not all the line, only the part to change. Something like this:
while read -r line; do
echo $line | sed "s/STRING1/STRING2/"
done < namefile
EDIT
I give you an example for my second question.
input file:
LAN 1:
[text]11111[text]
[text]22222[text]
[text]33333[text]
LAN 2:
[text]11111[text]
[text]22222[text]
[text]33333[text]
output file:
LAN 1:
[text]44444[text]
[text]22222[text]
[text]33333[text]
LAN 2:
[text]11111[text]
[text]22222[text]
[text]33333[text]
I have to overwrite database.txt so i think to do this line by line using a counter for LAN. This is my code:
while read -r line; do
echo "$line" | grep -q LAN
if [ $? = "0" ]; then
net_count=$((net_count+1))
fi
if [ $net_count = <lan choose before> ]; then # variable that contains lan number chosen by user
echo "$line" | fgrep -q "11111"
if [ "$?" = "0" ]; then
echo $line | sed "s/11111/44444/" > database.txt
break
fi
fi
done < database.txt
Thank you all
Running sed once for every line is typically over a thousand times slower than running just one sed instance processing your whole file of input. Don't do it.
If you want to do string manipulation on a line-by-line basis, use native bash primitives for the purpose, as documented in BashFAQ #100:
a() {
local line=$1 regex=$2
if [[ $line =~ $regex ]]; then
printf '%s\n' "$line"
fi
}
while IFS= read -r line; do
a "$line" LAN
done <database.txt
Likewise, for substring replacements, an appropriate parameter expansion primitive exists:
while read -r line; do
printf '%s\n' "${line//STRING1/STRING2}"
done < namefile
That said, those approaches are only appropriate if you need to iterate line-by-line. Typically, it makes more sense to use a single grep or sed operation, and iterate over the results of those calls if you need to do native bash operations. For instance, the following iterates over the output from grep, as emitted by a process substitution:
regex=LAN
while IFS= read -r line; do
echo "Read line from grep: $line"
done < <(grep -e "$regex" <database.txt)
You need to put $line in quotes for the whole line to be considered as a single parameter. Else, every space character splits the strings as multiple parameter to the function:
#!/bin/bash
function a {
echo "$1" | grep "$2"
}
while read -r line; do
a "$line" "LAN"
done < database.txt
And for the second question, if you want to print only the lines that you modify, you can use the following code:
while read -r line; do
echo "$line" | sed -n "s/STRING1/STRING2/p"
done < namefile
Here, -n will omit the lines that do not match the string; and the flag p makes sure to print the lines that match.
I'm writing a bash script to read a set of files line by line and perform some edits. To begin with, I'm simply trying to move the files to backup locations and write them out as-is, to test the script is working. However, it is failing to copy the last line of each file. Here is the snippet:
while IFS= read -r line
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
I obviously want to preserve whitespace when I copy the files, which is why I have set the IFS to null. I can see from the output that the last line of each file is being read, but it never appears in the output.
I've also tried an alternative variation, which does print the last line, but adds a newline to it:
while IFS= read -r line || [ -n "$line" ]
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
What is the best way to do this do this read-write operation, to write the files exactly as they are, with the correct whitespace and no newlines added?
The command that is adding the line feed (LF) is not the read command, but the echo command. read does not return the line with the delimiter still attached to it; rather, it strips the delimiter off (that is, it strips it off if it was present in the line, IOW, if it just read a complete line).
So, to solve the problem, you have to use echo -n to avoid adding back the delimiter, but only when you have an incomplete line.
Secondly, I've found that when providing read with a NAME (in your case line), it trims leading and trailing whitespace, which I don't think you want. But this can be solved by not providing a NAME at all, and using the default return variable REPLY, which will preserve all whitespace.
So, this should work:
#!/bin/bash
inFile=in;
outFile=out;
rm -f "$outFile";
rc=0;
while [[ $rc -eq 0 ]]; do
read -r;
rc=$?;
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
echo "$REPLY" >>"$outFile";
elif [[ -n "$REPLY" ]]; then ## incomplete line
echo "incomplete=\"$REPLY\"";
echo -n "$REPLY" >>"$outFile";
fi;
done <"$inFile";
exit 0;
Edit: Wow! Three excellent suggestions from Charles Duffy, here's an updated script:
#!/bin/bash
inFile=in;
outFile=out;
while { read -r; rc=$?; [[ $rc -eq 0 || -n "$REPLY" ]]; }; do
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
printf '%s\n' "$REPLY" >&3;
else ## incomplete line
echo "incomplete=\"$REPLY\"";
printf '%s' "$REPLY" >&3;
fi;
done <"$inFile" 3>"$outFile";
exit 0;
After review i wonder if :
{
line=
while IFS= read -r line
do
echo "$line"
line=
done
echo -n "$line"
} <$INFILE >$OUTFILE
is juts not enough...
Here my initial proposal :
#!/bin/bash
INFILE=$1
if [[ -z $INFILE ]]
then
echo "[ERROR] missing input file" >&2
exit 2
fi
OUTFILE=$INFILE.processed
# a way to know if last line is complete or not :
lastline=$(tail -n 1 "$INFILE" | wc -l)
if [[ $lastline == 0 ]]
then
echo "[WARNING] last line is incomplete -" >&2
fi
# we add a newline ANYWAY if it was complete, end of file will be seen as ... empty.
echo | cat $INFILE - | {
first=1
while IFS= read -r line
do
if [[ $first == 1 ]]
then
echo "First Line is ***$line***" >&2
first=0
else
echo "Next Line is ***$line***" >&2
echo
fi
echo -n "$line"
done
} > $OUTFILE
if diff $OUTFILE $INFILE
then
echo "[OK]"
exit 0
else
echo "[KO] processed file differs from input"
exit 1
fi
Idea is to always add a newline at the end of file and to print newlines only BETWEEN lines that are read.
This should work for quite all text files given they are not containing 0 byte ie \0 character, in which case 0 char byte will be lost.
Initial test can be used to decided whether an incomplete text file is acceptable or not.
Add a new line if line is not a line. Like this:
while IFS= read -r line
do
echo "Line is ***$line***";
printf '%s' "$line" >&3;
if [[ ${line: -1} != '\n' ]]
then
printf '\n' >&3;
fi
done < $POM.backup 3>$POM
I am trying to read lines within a file and if the line contains a tag, the text within the tag is used to replace the tag with a value from a different propertiesfile, before the line, with all tags replaced is written to a different file.
So the initial file being read have lines that would adhere to the following format:
testkey "TEST-KEY" "[#key_location#]:///[#key_name#]"
Where [# and #] house the tag text.
The propertied file would then contain lines like:
key_location=location_here
key_name=test_key_name
So the end result I am trying to achieve is that the line is written to a new file, but the tags are replaced with the values from the property file, so using the above content:
testkey "TEST-KEY" "loaction_here:///test_key_name"
I am not sure how best to handle the tags and deal with multiple tags in one line and am pretty lost. Any help would be greatly appreciated.
Skeleton code:
while read line
if [[ $line == *[#* ]]
then
#echo found a tag and need to deal with it
else
echo "$line">> $NEW_FILE
fi
done < $INITIAL_FILE
EDIT
Lines within the file could contain one or more tags, not always two like in the example given.
You'll have to do some looping and global sed replacements. The following is probably not optimal but it will get you started:
#!/bin/bash
declare -A props
while read line ; do
key=$(echo $line | sed -r 's/^(.*)=.*/\1/')
value=$(echo $line | sed -r 's/^.*=(.*)/\1/')
props[$key]=$value
done < values.properties
replace() {
line=$1
for key in "${!props[#]}"; do
line=$(echo $line | sed "s/\[#$key#\]/${props[$key]}/g")
done
echo $line
}
while read line ; do
while [[ $line == *"[#"*"#]"* ]] ;do
line=$(replace "$line")
echo Iter: $line
done
echo DONE: $line
done < $INITIAL_FILE
The snippet prints to stdout and it includes intermediate results so that you can check how it works. I think you will easily be able to modify it to write to a file, etc.
There are a number of ways to do this (e.g. count #). But simply, you can just use * expansion outside just the start and end of the string so
if [[ $line == *"[#"*"#]"*"[#"*"#]"* ]]; then
echo "has tags"
else
echo "does not have tags"
fi
Would work in your case, e.g.
$ echo "$line"
testkey "TEST-KEY" "[#key_location#]:///[#key_name#]"
$ if [[ $line == *"[#"*"#]"*"[#"*"#]"* ]]; then echo "has tags"; fi
has tags
I need to find strings matching some regexp pattern and represent the search result as array for iterating through it with loop ), do I need to use sed ? In general I want to replace some strings but analyse them before replacing.
Using sed and diff:
sed -i.bak 's/this/that/' input
diff input input.bak
GNU sed will create a backup file before substitutions, and diff will show you those changes. However, if you are not using GNU sed:
mv input input.bak
sed 's/this/that/' input.bak > input
diff input input.bak
Another method using grep:
pattern="/X"
subst=that
while IFS='' read -r line; do
if [[ $line = *"$pattern"* ]]; then
echo "changing line: $line" 1>&2
echo "${line//$pattern/$subst}"
else
echo "$line"
fi
done < input > output
The best way to do this would be to use grep to get the lines, and populate an array with the result using newline as the internal field separator:
#!/bin/bash
# get just the desired lines
results=$(grep "mypattern" mysourcefile.txt)
# change the internal field separator to be a newline
IFS=$'/n'
# populate an array from the result lines
lines=($results)
# return the third result
echo "${lines[2]}"
You could build a loop to iterate through the results of the array, but a more traditional and simple solution would just be to use bash's iteration:
for line in $lines; do
echo "$line"
done
FYI: Here is a similar concept I created for fun. I thought it would be good to show how to loop a file and such with this. This is a script where I look at a Linux sudoers file check that it contains one of the valid words in my valid_words array list. Of course it ignores the comment "#" and blank "" lines with sed. In this example, we would probably want to just print the Invalid lines only but this script prints both.
#!/bin/bash
# -- Inspect a sudoer file, look for valid and invalid lines.
file="${1}"
declare -a valid_words=( _Alias = Defaults includedir )
actual_lines=$(cat "${file}" | wc -l)
functional_lines=$(cat "${file}" | sed '/^\s*#/d;/^\s*$/d' | wc -l)
while read line ;do
# -- set the line to nothing "" if it has a comment or is empty line.
line="$(echo "${line}" | sed '/^\s*#/d;/^\s*$/d')"
# -- if not set to nothing "", check if the line is valid from our list of valid words.
if ! [[ -z "$line" ]] ;then
unset found
for each in "${valid_words[#]}" ;do
found="$(echo "$line" | egrep -i "$each")"
[[ -z "$found" ]] || break;
done
[[ -z "$found" ]] && { echo "Invalid=$line"; sleep 3; } || echo "Valid=$found"
fi
done < "${file}"
echo "actual lines: $actual_lines funtional lines: $functional_lines"
Hi I am trying to compare two strings in a directory. the format is as follows.
{sametext}difference{sametext}.
Note: {sametext} is not static for each file
for example
myfile_1_exercise.txt compared to myfile_2_exercise.txt
Can you tell me how I would match the above strings in an if statement.
Basically I need to know how I would ignore the number in the two strings so that these would be same.
Some example code is shown below:
My example code looks like this:
for g in `ls -d */`;
do
if [ -d $g ]; then
cd $g # down 1 directories
for h in `ls *root`;
do
printf "${Process[${count}]} = ${basedir}/${f}${g}${h}\n"
h1=${h}
if [ "${h1}" = "${h2}" ]; then # NEED to MATCH SOME HOW??????????
echo we have a match
fi
h2=${h1}
let count+=1
done
cd ../
#printf "\n\n\n\n"
fi
done
What should be the test to determine this instead of "${h1}" = "${h2}"?
Cheers,
Mike
"myfile_1_exercise.txt" == "myfile_2_exercise.txt"
You mean the above test should return true (ignoring the numbers) right?
This is what I would have done:
h1="myfile_1_exercise.txt"
h2="myfile_2_exercise.txt"
if [ $( echo ${h1} | sed 's/[0-9]*//g' ) == $( echo ${h2} | sed 's/[0-9]*//g' ) ] ; then
# do something here.
fi
sed comes in handy here.
This basically goes through every file in the directory, extracts the two strings from the filename, and keeps a list of all the unique combinations thereof.
Then, it walks through this list, and uses bash's wildcard expansion to allow you to loop over each collection.
EDIT: Got rid of an ugly hack.
i=0
for f in *_*_*.txt
do
a=`echo "$f" | sed 's/\(.*\)_.*_\(.*\).txt/\1/g'`
b=`echo "$f" | sed 's/\(.*\)_.*_\(.*\).txt/\2/g'`
tmp=${all[#]}
expr match "$tmp" ".*$a:$b.*" >/dev/null
if [ "$?" == "1" ]
then
all[i]="$a:$b"
let i+=1
fi
done
for f in ${all[#]}
do
a=`echo "$f" | sed 's/\(.*\):\(.*\)/\1/g'`
b=`echo "$f" | sed 's/\(.*\):\(.*\)/\2/g'`
echo $a - $b
for f2 in $a_*_$b.txt
do
echo " $f2"
# ...
done
done
Of course, this assumes that all the files you care about follow the *_*_*.txt pattern.
Disclaimer:
Mileage can vary and you may have to adjust and debug the script for corner cases.
You may be better off using Perl for your task.
There could be better solutions even in Bash. This one is not very efficient but it seems to work.
Said that, here is a script that compares two strings according to your requirements. I am sure you can figure how to use it in your directory listing script (for which you may want to consider find by the way)
This script takes two strings and prints match! if they match
$ bash compare.sh myfile_1_exercise.txt myfile_2_exercise.txt
match!
$ bash compare.sh myfile_1_exercise.txt otherfile_2_exercise.txt
$
The script:
#!/bin/bash
fname1=$1
fname2=$2
findStartMatch() {
match=""
rest1=$1 ;
rest2=$2 ;
char1=""
char2=""
while [[ "$rest1" != "" && "$rest2" != "" && "$char1" == "$char2" ]] ; do
char1=$(echo $rest1 | sed 's/\(.\).*/\1/');
rest1=$(echo $rest1 | sed 's/.\(.*\)/\1/') ;
char2=$(echo $rest2 | sed 's/\(.\).*/\1/');
rest2=$(echo $rest2 | sed 's/.\(.*\)/\1/') ;
if [[ "$char1" == "$char2" ]] ; then
match="${match}${char1}"
fi
done
}
findEndMatch() {
match=""
rest1=$1 ;
rest2=$2 ;
char1=""
char2=""
while [[ "$rest1" != "" && "$rest2" != "" && "$char1" == "$char2" ]] ; do
char1=$(echo $rest1 | sed 's/.*\(.\)/\1/');
rest1=$(echo $rest1 | sed 's/\(.*\)./\1/') ;
char2=$(echo $rest2 | sed 's/.*\(.\)/\1/');
rest2=$(echo $rest2 | sed 's/\(.*\)./\1/') ;
if [[ "$char1" == "$char2" ]] ; then
match="${char1}${match}"
fi
done
}
findStartMatch $fname1 $fname2
startMatch=$match
findEndMatch $fname1 $fname2
endMatch=$match
if [[ "$startMatch" != "" && "$endMatch" != "" ]] ; then
echo "match!"
fi
If you are actually comparing two files like you mentiond ... probably you can use diff command like
diff myfile_1_exercise.txt myfile_2_exercise.txt