Using bash I was wondering how I can find all instances of a word which starts with text, store it as a variable and print it out.
For example if this was my file
test.conf
$temp_test
test
$1234_$temp
$temp_234
My output would be as follows:
$temp_test
$temp_234
Can anyone tell me how this might be possible? This is the closest I could get so far.
while read NAME
do
echo "$NAME"
done < test.conf
You can just use grep with right regex:
grep '^ *$temp' test.conf
$temp_test
$temp_234
UPDATE: As per comments:
while read -r l; do
echo "$l"
done < <(sed -n '/^ *$temp_/s/^ *\$temp_//p' t.conf)
test
234
Related
I'm sorry for my poor English, first.
I want to read a file (tel.txt) that contains many tel numbers (a number per line) and use that line to grep command to search about the specific number in the source file (another file)!
I wrote this code :
dir="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A"
file="$dir/tel.txt"
datafile="$dir/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.txt"
while IFS= read -r line
do
current="$line"
echo `grep -F $current "$datafile" >> output.txt`
done < $file
the tel file sample :
44001547
44001478
55421487
but that code returns nothing!
when I declare 'current' variable with literals it works correctly!
what happened?!
Your grep command is redirected to write its output to a file, so you don't see it on the terminal.
Anyway, you should probably be using the much simpler and faster
grep -Ff "$file" "$datafile"
Add | tee -a output.txt if you want to save the output to a file and see it at the same time.
echo `command` is a buggy and inefficient way to write command. (echo "`command`" would merely be inefficient.) There is no reason to capture standard output into a string just so that you can echo that string to standard output.
Why don't you search for the line var directly? I've done some tests, this script works on my linux (CentOS 7.x) with bash shell:
#!/bin/bash
file="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A/tel.txt"
while IFS= read -r line
do
echo `grep "$line" /home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.tx >> output.txt`
done < $file
Give it a try... It shows nothing on the screen since you're redirecting the output to the file output.txt so the matching results are saved there.
You should use file descriptors when reading with while loop.instead use for loop to avoid false re-directions
dir="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A"
file="$dir/tel.txt"
datafile="$dir/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.txt"
for line in `cat $file`
do
current="$line"
echo `grep -F $current "$datafile" >> output.txt`
done
My objective here is to read each revision present in revisions.txt and extract the text against that revision number from logs.txt file.
For example revisions.txt file has the following inputs,
APP-1001
APP-1002
APP-1004
And logs.txt file has the following inputs,
APP-999 : Bug Fix for XYZ issues
APP-1001 : Bug Fix for XYZ issues
APP-1002 : Bug Fix for XYZ issues
APP-1003 : Bug Fix for XYZ issues
APP-1004 : Bug Fix for XYZ issues
I want to get all the lines against all revisions in the revisions.txt file. I came up with the following code,
#!/bin/bash
echo "Start!"
input=logs.txt
revision=revisions.txt
IFS=$'\n' read -d '' -r -a revs < $revision
for rev in "${revs[#]}"
do
echo $rev
while IFS='' read -r line || [[ -n "$line" ]]; do
regex='^'$rev'(.*)'
if [[ $line =~ $regex ]];
then
echo $line
fi
done < "$input"
done
Output as of now,
APP-1001
APP-1002
APP-1004
APP-1004 : Bug Fix for XYZ issues
It's only printing text for the last revision i.e. APP-1004. I read on the web that it's not possible to read two files in nested loop.
Please suggest a different way of doing this.
Thanks in advance.
You can nest the loops and hence, you can iterate from both files at the same time.
while read revisions_line
do
while read logs_line
do
echo "This line comes from revisions : $revisions_line"
echo "This line comes from logs : $logs_line"
done << "$( cat logs.txt)"
done << "$( cat revisions.txt)"
Regards!
You can simply use below script for the same:-
#!/bin/bash
while read line
do
#echo "$line logs.txt"
log_info=$(grep $line logs.txt)
if [ log_info != "" ]; then
echo "$log_info" | awk '{$1=$2=""; print $0}' | sed 's/^[ \t]*//'
fi
done < revisions.txt
How will it work? While loop will read each line from file revisions.txt and grep will search for that in logs.txt file and if found it will store that line in variable log_info. Now if log_info variable is not empty the line will be redirect to awk command and awk command will remove column1 and column2 from the line and redirect the output to sed command and finally sed will remove the leading blank space from it. The final out put will be:-
Bug Fix for XYZ issues
Bug Fix for XYZ issues
Bug Fix for XYZ issues
If you want the whole string just modify like below:-
echo "$log_info"
If you don't bother about the leading blank space you can go ahead with below:-
echo "$log_info" | awk '{$1=$2=""; print $0}'
I'm really new to Linux scripting. I am sure this is simple, but I cannot figure it out.
As part of a script, I am trying to pass the content of a file as arguments of a command in a script:
while read i
do $COMMAND $i
done < file.lst
I want to pass every line of the file.lst as the argument of the command except the very first line of the file. How to I do this?
EDIT:
Here is the section of the script:
while read i
do cp --recursive --preserve=all $i $DIR
done < $DIR/file.lst
while read -r i
do
"$COMMAND" "$i"
done < <(sed -n '2,$p' file.lst)
This solutions does not use a while so I am not entirely sure if it solves your problem, but based on your code sample. you can do the following
tail -n +2 input | xargs echo
This will read all lines from input starting at line 2 and execute echo using the value of the line
the file input contains:
skip
1
2
3
executing that command gives
1
2
3
Just substitute input for the file you want and echo for the command you want
Add an extra read to consume the first line before the while loop begins.
{
read -r;
while read -r i; do
"$COMMAND" "$i"
done
} < file.lst
I am trying to find a clever way to figure out if the file passed to sed has been altered successfully or not.
Basically, I want to know if the file has been changed or not without having to look at the file modification date.
The reason why I need this is because I need to do some extra stuff if sed has successfully replaced a pattern.
I currently have:
grep -q $pattern $filename
if [ $? -eq 0 ]
then
sed -i s:$pattern:$new_pattern: $filename
# DO SOME OTHER STUFF HERE
else
# DO SOME OTHER STUFF HERE
fi
The above code is a bit expensive and I would love to be able to use some hacks here.
A bit late to the party but for the benefit of others, I found the 'w' flag to be exactly what I was looking for.
sed -i "s/$pattern/$new_pattern/w changelog.txt" "$filename"
if [ -s changelog.txt ]; then
# CHANGES MADE, DO SOME STUFF HERE
else
# NO CHANGES MADE, DO SOME OTHER STUFF HERE
fi
changelog.txt will contain each change (ie the changed text) on it's own line. If there were no changes, changelog.txt will be zero bytes.
A really helpful sed resource (and where I found this info) is http://www.grymoire.com/Unix/Sed.html.
I believe you may find these GNU sed extensions useful
t label
If a s/// has done a successful substitution since the last input line
was read and since the last t or T command, then branch to label; if
label is omitted, branch to end of script.
and
q [exit-code]
Immediately quit the sed script without processing any more input, except
that if auto-print is not disabled the current pattern space will be printed.
The exit code argument is a GNU extension.
It seems like exactly what are you looking for.
This might work for you (GNU sed):
sed -i.bak '/'"$old_pattern"'/{s//'"$new_pattern"'/;h};${x;/./{x;q1};x}' file || echo changed
Explanation:
/'"$old_pattern"'/{s//'"$new_pattern"'/;h} if the pattern space (PS) contains the old pattern, replace it by the new pattern and copy the PS to the hold space (HS).
${x;/./{x;q1};x} on encountering the last line, swap to the HS and test it for the presence of any string. If a string is found in the HS (i.e. a substitution has taken place) swap back to the original PS and exit using the exit code of 1, otherwise swap back to the original PS and exit with the exit code of 0 (the default).
You can diff the original file with the sed output to see if it changed:
sed -i.bak s:$pattern:$new_pattern: "$filename"
if ! diff "$filename" "$filename.bak" &> /dev/null; then
echo "changed"
else
echo "not changed"
fi
rm "$filename.bak"
You could use awk instead:
awk '$0 ~ p { gsub(p, r); t=1} 1 END{ exit (!t) }' p="$pattern" r="$repl"
I'm ignoring the -i feature: you can use the shell do do redirections as necessary.
Sigh. Many comments below asking for basic tutorial on the shell. You can use the above command as follows:
if awk '$0 ~ p { gsub(p, r); t=1} 1 END{ exit (!t) }' \
p="$pattern" r="$repl" "$filename" > "${filename}.new"; then
cat "${filename}.new" > "${filename}"
# DO SOME OTHER STUFF HERE
else
# DO SOME OTHER STUFF HERE
fi
It is not clear to me if "DO SOME OTHER STUFF HERE" is the same in each case. Any similar code in the two blocks should be refactored accordingly.
In macos I just do it as follows:
changes=""
changes+=$(sed -i '' "s/$to_replace/$replacement/g w /dev/stdout" "$f")
if [ "$changes" != "" ]; then
echo "CHANGED!"
fi
I checked, and this is faster than md5, cksum and sha comparisons
I know it is a old question and using awk instead of sed is perhaps the best idea, but if one wants to stick with sed, an idea is to use the -w flag. The file argument to the w flag only contains the lines with a match. So, we only need to check that it is not empty.
perl -sple '$replaced++ if s/$from/$to/g;
END{if($replaced != 0){ print "[Info]: $replaced replacement done in $ARGV(from/to)($from/$to)"}
else {print "[Warning]: 0 replacement done in $ARGV(from/to)($from/$to)"}}' -- -from="FROM_STRING" -to="$DESIRED_STRING" </file/name>
Example:
The command will produce the following output, stating the number of changes made/file.
perl -sple '$replaced++ if s/$from/$to/g;
END{if($replaced != 0){ print "[Info]: $replaced replacement done in $ARGV(from/to)($from/$to)"}
else {print "[Warning]: 0 replacement done in $ARGV(from/to)($from/$to)"}}' -- -from="timeout" -to="TIMEOUT" *
[Info]: 5 replacement done in main.yml(from/to)(timeout/TIMEOUT)
[Info]: 1 replacement done in task/main.yml(from/to)(timeout/TIMEOUT)
[Info]: 4 replacement done in defaults/main.yml(from/to)(timeout/TIMEOUT)
[Warning]: 0 replacement done in vars/main.yml(from/to)(timeout/TIMEOUT)
Note: I have removed -i from the above command , so it will not update the files for the people who are just trying out the command. If you want to enable in-place replacements in the file add -i after perl in above command.
check if sed has changed MANY files
recursive replace of all files in one directory
produce a list of all modified files
workaround with two stages: match + replace
g='hello.*world'
s='s/hello.*world/bye world/g;'
d='./' # directory of input files
o='modified-files.txt'
grep -r -l -Z -E "$g" "$d" | tee "$o" | xargs -0 sed -i "$s"
the file paths in $o are zero-delimited
$ echo hi > abc.txt
$ sed "s/hi/bye/g; t; q1;" -i abc.txt && (echo "Changed") || (echo "Failed")
Changed
$ sed "s/hi/bye/g; t; q1;" -i abc.txt && (echo "Changed") || (echo "Failed")
Failed
https://askubuntu.com/questions/1036912/how-do-i-get-the-exit-status-when-using-the-sed-command/1036918#1036918
Don't use sed to tell if it has changed a file; instead, use grep to tell if it is going to change a file, then use sed to actually change the file. Notice the single line of sed usage at the very end of the Bash function below:
# Usage: `gs_replace_str "regex_search_pattern" "replacement_string" "file_path"`
gs_replace_str() {
REGEX_SEARCH="$1"
REPLACEMENT_STR="$2"
FILENAME="$3"
num_lines_matched=$(grep -c -E "$REGEX_SEARCH" "$FILENAME")
# Count number of matches, NOT lines (`grep -c` counts lines),
# in case there are multiple matches per line; see:
# https://superuser.com/questions/339522/counting-total-number-of-matches-with-grep-instead-of-just-how-many-lines-match/339523#339523
num_matches=$(grep -o -E "$REGEX_SEARCH" "$FILENAME" | wc -l)
# If num_matches > 0
if [ "$num_matches" -gt 0 ]; then
echo -e "\n${num_matches} matches found on ${num_lines_matched} lines in file"\
"\"${FILENAME}\":"
# Now show these exact matches with their corresponding line 'n'umbers in the file
grep -n --color=always -E "$REGEX_SEARCH" "$FILENAME"
# Now actually DO the string replacing on the files 'i'n place using the `sed`
# 's'tream 'ed'itor!
sed -i "s|${REGEX_SEARCH}|${REPLACEMENT_STR}|g" "$FILENAME"
fi
}
Place that in your ~/.bashrc file, for instance. Close and reopen your terminal and then use it.
Usage:
gs_replace_str "regex_search_pattern" "replacement_string" "file_path"
Example: replace do with bo so that "doing" becomes "boing" (I know, we should be fixing spelling errors not creating them :) ):
$ gs_replace_str "do" "bo" test_folder/test2.txt
9 matches found on 6 lines in file "test_folder/test2.txt":
1:hey how are you doing today
2:hey how are you doing today
3:hey how are you doing today
4:hey how are you doing today hey how are you doing today hey how are you doing today hey how are you doing today
5:hey how are you doing today
6:hey how are you doing today?
$SHLVL:3
Screenshot of the output:
References:
https://superuser.com/questions/339522/counting-total-number-of-matches-with-grep-instead-of-just-how-many-lines-match/339523#339523
https://unix.stackexchange.com/questions/112023/how-can-i-replace-a-string-in-a-files/580328#580328
I'm trying to execute a command for each line coming from a cat command. I'm basing this on sample code I got from a vendor.
Here's the script:
for tbl in 'cat /tmp/tables'
do
echo $tbl
done
So I was expecting the output to be each line in the file. Instead I'm getting this:
cat
/tmp/tables
That's obviously not what I wanted.
I'm going to replace the echo with an actual command that interfaces with a database.
Any help in straightening this out would be greatly appreciated.
You are using the wrong type of quotes.
You need to use the back-quotes rather than the single quote to make the argument being a program running and piping out the content to the forloop.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done
Also for better readability (if you are using bash), you can write it as
for tbl in $(cat /tmp/tables)
do
echo "$tbl"
done
If your expectations are to get each line (The for-loops above will give you each word), then you may be better off using xargs, like this
cat /tmp/tables | xargs -L1 echo
or as a loop
cat /tmp/tables | while read line; do echo "$line"; done
The single quotes should be backticks:
for tbl in `cat /etc/tables`
Although, this will not get you output/input by line, but by word. To process line by line, you should try something like:
cat /etc/tables | while read line
echo $line
done
With while loop:
while read line
do
echo "$line"
done < "file"
while IFS= read -r tbl; do echo "$tbl" ; done < /etc/tables
read this.
You can do a lot of parsing in bash by redefining the IFS (Input Field Seperator), for example
IFS="\t\n" # You must use double quotes for escape sequences.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done