I am writing a shell script in which I will read a file and will modify it.
there will be occurrence of some string "ABC_1" in multiple lines.
I need to replace it with "XYZ_1" only when there is "OPQ_3" also present in the line else there should be no modification in the line.
please help how can I do replacement if I read a file liken by line.
for FILE in $FILES
do
echo $FILE
while read line
do
if grep -n "OPQ_3" $line
then
sed -i 's/ABC_1/XYZ_2/'
fi
done < $FILE
done
You can use this sed:
sed -i '/OPQ_3\|OPQ_4/s/ABC_1/XYZ_2/' file
Anubhava has the better answer. Here's how you'd write it in bash
for file in $FILES; do
echo "$file"
tmpfile=$(mktemp)
while IFS= read -r line; do
[[ $line == *OPQ_3* ]] && line=${line/ABC_1/XYZ_2/}
echo "$line"
done < "$file" > "$tmpfile"
mv "$tmpfile" "$file"
done
Note IFS= read -r line is the only way to read a line from stdin exactly, without losing any whitespace or special characters.
Related
For files like :
_aaa.txt
_bbb.txt
_ccc.txt
I want to convert them to :
_aAa.txt
_bBb.txt
Any idea how to do this ?
In plain bash, using only shell parameter expansions to perform the conversion:
#!/bin/bash
n=3
for file in *; do
[[ -f $file ]] || continue
suffix=${file:n-1}
mv -i "$file" "${file:0:n-1}${suffix^}"
done
Check the output of the following
#!/bin/bash
for filename in *; do
newname=$(sed 's/./\U&/3' <<< "$filename")
echo "$filename --> $newname"
# mv $filename $newname
done
Then remove the #, if the filename printed is correct
If rename command is available, please try:
rename 's/^(..)(.)/$1\U$2/' *.txt
I am trying to write a bash script to scan for authorized_keys files and remove the keys of a couple previous employees if found. I am having one heck of a time figuring out the escaping for the sed command at the end. I am using commas instead of / since / can show up in the ssh-key. Any help would be appreciated
#!/bin/bash
declare -A keys
keys["employee1"]='AAAAB3NzaC1yc2EAAAABJQAAAIEAxoZ7ZdpJkL98n8cSTkFBwaAeSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+AHWZLGRHfXT2XBg2SRaUnubAKp0w6qNIbej0MsA/ifAs8AfVGdj0pUPLtKpo6XVZkB8vEZSIQ+xNk1n5HJrGJnFGWKWeY3z1/KOLxcLHU='
keys["employee2"]='AAAAB3NzaC1yc2EAAAABIwAAAQEAwHYNAVhb319OBVXPhYF8cSTkFBwaAekr7UcKjfLPCHMpz19W0L/C0g+75Hn8COxOQILDUhIPhYHXOduQjGD/6NXgJDWxgyT00Azg5BREUnBd58WqZPlEvTZYlAgmdMIbnWPPGdJwzqKH/k7/STK6vTKxL6rxBo4lSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+NIbej0MsA/ifAs8AfAkfO2JjgeQpJMyZ7B02XVN5iSLAyC3Cb5FXIjJuk4LPhcApuVyszH2lgve0r5bt/nFgVujJTvJTHPlGrqkYDcDJVUtfbjoLqGPrnpijp6rGIC7aFDDe7bk0ygHYMXDFWcjJBerfLGUWTYWFFLY3bfiO/h/9oEycmQHyB2co4a0IyyDnaYn9OY6xsRRATVlk4Q=='
files=`find / -name authorized_keys`
echo "Checking Authorized_Keys files on: " `hostname`
echo ""
echo "Located files: "
for file in $files; do
echo " $file"
done
echo""
for file in $files; do
for key in "${!keys[#]}"; do
if grep -q ${keys[$key]} $file; then
echo " *** Removing $key from $file"
sed "s,${keys[$key]},d" $file
fi
done
done
You've made it a bit complicated I think.
You can do this using grep -vf and process substitution:
# array to hold the value you want to remove
keys=(
'AAAAB3NzaC1yc2EAAAABJQAAAIEAxoZ7ZdpJkL98n8cSTkFBwaAeSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+AHWZLGRHfXT2XBg2SRaUnubAKp0w6qNIbej0MsA/ifAs8AfVGdj0pUPLtKpo6XVZkB8vEZSIQ+xNk1n5HJrGJnFGWKWeY3z1/KOLxcLHU=',
'AAAAB3NzaC1yc2EAAAABIwAAAQEAwHYNAVhb319OBVXPhYF8cSTkFBwaAekr7UcKjfLPCHMpz19W0L/C0g+75Hn8COxOQILDUhIPhYHXOduQjGD/6NXgJDWxgyT00Azg5BREUnBd58WqZPlEvTZYlAgmdMIbnWPPGdJwzqKH/k7/STK6vTKxL6rxBo4lSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+NIbej0MsA/ifAs8AfAkfO2JjgeQpJMyZ7B02XVN5iSLAyC3Cb5FXIjJuk4LPhcApuVyszH2lgve0r5bt/nFgVujJTvJTHPlGrqkYDcDJVUtfbjoLqGPrnpijp6rGIC7aFDDe7bk0ygHYMXDFWcjJBerfLGUWTYWFFLY3bfiO/h/9oEycmQHyB2co4a0IyyDnaYn9OY6xsRRATVlk4Q=='
)
while IFS= read -d '' -r file; do
grep -vf <(printf "%s\n" "${keys[#]}") "$file" > "$file.tmp"
mv "$file.tmp" "$file"
done < <(find / -name authorized_keys -print0)
In your case, it's easy, just need to use a sign which not contained in base64 code as the delimiter, eg |:
sed "\|${keys[$key]}|d" $file
Explanation in the sed manual:
\%regexp%
(The % may be replaced by any other single character.)
This also matches the regular expression regexp, but allows one to use a different delimiter than /.
This question already has answers here:
Looping through the content of a file in Bash
(16 answers)
Closed 2 years ago.
I have the following .txt file:
Marco
Paolo
Antonio
I want to read it line-by-line, and for each line I want to assign a .txt line value to a variable. Supposing my variable is $name, the flow is:
Read first line from file
Assign $name = "Marco"
Do some tasks with $name
Read second line from file
Assign $name = "Paolo"
The following reads a file passed as an argument line by line:
while IFS= read -r line; do
echo "Text read from file: $line"
done < my_filename.txt
This is the standard form for reading lines from a file in a loop. Explanation:
IFS= (or IFS='') prevents leading/trailing whitespace from being trimmed.
-r prevents backslash escapes from being interpreted.
Or you can put it in a bash file helper script, example contents:
#!/bin/bash
while IFS= read -r line; do
echo "Text read from file: $line"
done < "$1"
If the above is saved to a script with filename readfile, it can be run as follows:
chmod +x readfile
./readfile filename.txt
If the file isn’t a standard POSIX text file (= not terminated by a newline character), the loop can be modified to handle trailing partial lines:
while IFS= read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
done < "$1"
Here, || [[ -n $line ]] prevents the last line from being ignored if it doesn't end with a \n (since read returns a non-zero exit code when it encounters EOF).
If the commands inside the loop also read from standard input, the file descriptor used by read can be chanced to something else (avoid the standard file descriptors), e.g.:
while IFS= read -r -u3 line; do
echo "Text read from file: $line"
done 3< "$1"
(Non-Bash shells might not know read -u3; use read <&3 instead.)
I encourage you to use the -r flag for read which stands for:
-r Do not treat a backslash character in any special way. Consider each
backslash to be part of the input line.
I am citing from man 1 read.
Another thing is to take a filename as an argument.
Here is updated code:
#!/usr/bin/bash
filename="$1"
while read -r line; do
name="$line"
echo "Name read from file - $name"
done < "$filename"
Using the following Bash template should allow you to read one value at a time from a file and process it.
while read name; do
# Do what you want to $name
done < filename
#! /bin/bash
cat filename | while read LINE; do
echo $LINE
done
Use:
filename=$1
IFS=$'\n'
for next in `cat $filename`; do
echo "$next read from $filename"
done
exit 0
If you have set IFS differently you will get odd results.
Many people have posted a solution that's over-optimized. I don't think it is incorrect, but I humbly think that a less optimized solution will be desirable to permit everyone to easily understand how is this working. Here is my proposal:
#!/bin/bash
#
# This program reads lines from a file.
#
end_of_file=0
while [[ $end_of_file == 0 ]]; do
read -r line
# the last exit status is the
# flag of the end of file
end_of_file=$?
echo $line
done < "$1"
If you need to process both the input file and user input (or anything else from stdin), then use the following solution:
#!/bin/bash
exec 3<"$1"
while IFS='' read -r -u 3 line || [[ -n "$line" ]]; do
read -p "> $line (Press Enter to continue)"
done
Based on the accepted answer and on the bash-hackers redirection tutorial.
Here, we open the file descriptor 3 for the file passed as the script argument and tell read to use this descriptor as input (-u 3). Thus, we leave the default input descriptor (0) attached to a terminal or another input source, able to read user input.
For proper error handling:
#!/bin/bash
set -Ee
trap "echo error" EXIT
test -e ${FILENAME} || exit
while read -r line
do
echo ${line}
done < ${FILENAME}
Use IFS (internal field separator) tool in bash, defines the character using to separate lines into tokens, by default includes <tab> /<space> /<newLine>
step 1: Load the file data and insert into list:
# declaring array list and index iterator
declare -a array=()
i=0
# reading file in row mode, insert each line into array
while IFS= read -r line; do
array[i]=$line
let "i++"
# reading from file path
done < "<yourFullFilePath>"
step 2: now iterate and print the output:
for line in "${array[#]}"
do
echo "$line"
done
echo specific index in array: Accessing to a variable in array:
echo "${array[0]}"
The following will just print out the content of the file:
cat $Path/FileName.txt
while read line;
do
echo $line
done
I want to search for patterns in a file and remove the lines containing the pattern. To do this, am using:
originalLogFile='sample.log'
outputFile='3.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
temp=$outputFile
done <$whiteListOfErrors
This works fine for the first iteration. For the second run, it throws :
grep: input file ‘3.txt’ is also the output
Any solutions or alternate methods?
The following should be equivalent
grep -v -f "$whiteListOfErrors" "$originalLogFile" > "$outputFile"
originalLogFile='sample.log'
outputFile='3.txt'
tmpfile='tmp.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
cp $outputfile $tmpfile
temp=$tmpfile
done <$whiteListOfErrors
Use sed for this:
sed '/.*pattern.*/d' file
If you have multiple patterns you may use the -e option
sed -e '/.*pattern1.*/d' -e '/.*pattern2.*/d' file
If you have GNU sed (typical on Linux) the -i option is comfortable as it can modify the original file instead of writing to a new file. (But handle with care, in order to not overwrite your original)
Used this to fix the problem:
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp | tee $outputFile
temp=$outputFile
done <$falseFailures
Trivial solution might be to work with alternating files; e.g.
idx=0
while ...
let next='(idx+1) % 2'
grep ... $file.$idx > $file.$next
idx=$next
A more elegant might be the creation of one large grep command
args=( )
while read line; do args=( "${args[#]}" -v "$line" ); done < $whiteList
grep "${args[#]}" $origFile
I received a advice for do not parse ls, like describes in this website: Don't parse ls.
I was looking for DAILY files in my directory so that's what I did then:
for f in *.DA*; do
[[ -e $f ]] || continue
for file in $f; do
echo "The file that you are working on: "$file
archiveContent=$( sed -n -e 1p $file )
echo $archiveContent
done
done
Ok, that's works well, I've two files A.DAILY and B.DAILY, with the both archives I can get what is inside it, but when I changed a little bit the loop, it doesn't iterated with all files with .DAILY extension in my directory.
for f in *.DA*; do
[[ -e $f ]] || continue
for file in $f; do
echo "The file that you are working on: "$file
archiveContent=$( sed -n -e 1p $file )
echo $archiveContent
COMPRESS $archiveContent;
done
done
when I called a function inside the loop, the loop just does for the first file, but not to the second.
Since the outer loop sets f to each file in turn, your inner loop doesn't seem to serve any purpose.
for f in *.DA*; do
[[ -e $f ]] || continue
echo "The file that you are working on: $f"
archiveContent=$( sed -n -e 1p "$f" )
echo "$archiveContent"
COMPRESS "$archiveContent"
done