I am trying to grep some info from a directory for each line in a file. I am using while loop to grep each line of a file. The grep alone works perfectly. I tested my while loop with echo and it works perfectly but when I use grep inside it gives me no output.
while IFS= read -r LINE; do
grep --include=\requests-definition.const.ts -rnwH $DIR -e "$LINE";
echo $LINE;
done < key_list
my key_list is a text file having a key on each line.
when I use the grep alone it works but it may not work In a while loop.
Thanks !
Make sure you loop is not exiting at the first Line without match.
From grep Man
EXIT STATUS
Normally the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if
an error occurred. However, if the -q or --quiet or --silent is used and a line is
selected, the exit status is 0 even if an error occurred.
To aviod exiting you have several ways to go:
Use a subshell with $(grep ...)
use the flag set +e
while IFS= read -r LINE; do
set +x # don't exit if exit code is different from 0
grep --include=\requests-definition.const.ts -rnwH $DIR -e "$LINE";
set -e # exit if exit code is different from 0
echo $LINE;
done < key_list
the problem was solved by removing special characters from key_list using dos2unix command.
Related
I wanna move a file after the grep command but as I execute my script, I noticed that there are no results coming back. regardless of that, I want to move the file/s to another directory.
this is what I've been doing:
for file in *.sup
do
grep -iq "$file" '' /desktop/list/varlogs.txt || mv "$file" /desktop/first;
done
but I am getting this error:
mv: 0653-401 Cannot rename first /desktop/first/first
suggestions would be very helpful
I am not sure what the two single quotes are for in between ..."$file" '' /desktop.... With them there, grep is looking also for $file in a file called '', so grep will throw the grep: : No such file or directory error with that there.
Also pay attention to the behavior change of adding the -q or --quiet flags, as it affects the returned value of grep and will impact whether the command to the || is run or not (see man grep for more).
I can't make out exactly what you are trying to do, but you can add a couple statements to help figure out what is going on. You could run your script with bash -x ./myscript.sh to display everything that runs as it runs, or add set -x before and set +x after the for loop in the script to show what is happening.
I added some debugging to your script and changed th || to an if/then statement to expose what is happening. Try this and see if you can find where things are going awry.
echo -e "============\nBEFORE:\n============"
echo -e "\n## The files in current dir '$(pwd)' are: ##\n$(ls)"
echo -e "\n## The files in '/desktop/first' are: ##\n$(ls /desktop/first)"
echo -e "\n## Looking for '.sup' files in '$(pwd)' ##"
for file in *.sup; do
echo -e "\n## == look for '${file}' in '/desktop/list/varlogs.txt' == ##"
# let's change this to an if/else
# the || means try the left command for success, or try the right one
# grep -iq "$file" '' /desktop/list/varlogs.txt || mv -v "$file" /desktop/first
# based on `man grep`: EXIT STATUS
# Normally the exit status is 0 if a line is selected,
# 1 if no lines were selected, and 2 if an error occurred.
# However, if the -q or --quiet or --silent is used and a line
# is selected, the exit status is 0 even if an error occurred.
# note that --ignore-case and --quiet are long versions of -i and -q/ -iq
if grep --ignore-case --quiet "${file}" '' /desktop/list/varlogs.txt; then
echo -e "\n'${file}' found in '/desktop/list/varlogs.txt'"
else
echo -e "\n'${file}' not found in '/desktop/list/varlogs.txt'"
echo -e "\nmove '${file}' to '/desktop/first'"
mv --verbose "${file}" /desktop/first
fi
done
echo -e "\n============\nAFTER:\n============"
echo -e "\n## The files in current dir '$(pwd)' are: ##\n$(ls)"
echo -e "\n## The files in '/desktop/first' are: ##\n$(ls /desktop/first)"
|| means try the first command, and if it is not successful (i.e. does not return 0), then do the next command. In your case, it appears you are looking in /desktop/list/varlogs.txt to see if any .sup files in the current directory match any in the varlogs file and if not, then move them to the /desktop/first/ directory. If matches were found, leave them in the current dir. (according to the logic you have currently)
mv --verbose explain what is being done
echo -e enables interpretation of backslash escapes
set -x shows the commands that are being run/ debugging
Please respond and clarify if anything is different. I am trying to raise in the ranks to be more helpful so I would appreciate comments, and upvotes if this was helpful.
Suggesting to avoid repeated scans of /desktop/list/varlogs.txt, and remove duplicats:
mv $(grep -o -f <<<$(ls -1 *.sup) /desktop/list/varlogs.txt|sort|uniq) /desktop/first
Suggesting to test step 1. in explanation below to list the files to be moved.
Explanation
1. grep -o -f <<<$(ls -1 *.sup) /desktop/list/varlogs.txt| sort| uniq
List all the files selected in ls -1 *.sup mentioned in /desktop/list/varlogs.txt in a single scan.
-o list only matched filenames.
<<<$(ls -1 *.sup) prepare a temporary redirected input file containing all the pattern match strings. From the output of ls -1 *.sup
|sort|uniq Than, sort the list and remove duplicates (we can move the file only once).
2. mv <files-list-output-from-step-1> /desktop/first
Move all the files found in step 1 to directory /desktop/first
I have the following code but it doesnt work when i execute the code, the file th2.csv its empty.
The function of the sed is replace two words. I dont know how to make the script work correctly.
It must be done with the while.
bash th1.csv > th2.csv
Script bash
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" th1.csv
done < th1.csv
Given the requirements that you must loop and apply regex, line by line, then consider:
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" >> th2.csv
done < th1.csv
This reads, line by line, via a while loop. Each line is passed as stdin to sed. Note we remove the th1.csv at the end of your original sed attempt, as that will override sed reading from stdin (causing it to ignore it and instead process the file over and over again, every iteration). Lastly we append >> to your th2.csv file each iteration.
Guessing a step ahead, that you may want to pass the two files in as parameters to the script (just based on your first code snippet) then you can change this to:
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" >> "$2"
done < "$1"
And, assuming this script is called myscript.sh you can call it like:
/bin/bash myscript.sh 'th1.csv' 'th2.csv'
Or, if you make it executable with chmod +x myscript.sh then:
./myscript.sh 'th1.csv' 'th2.csv'.
I have multiple files with an insanely long list of commands. I can't run them all in one go, so I need a smart way to read and execute from file as well as delete the command after completion.
So far I have tried
for i in filename.txt ; do ; execute $i ; sed -s 's/$i//' ; done ;
but it doesn't work. Before I introduced sed, $i was executing. Now even that is not working.
I thought of a workaround where I will read first line and delete first line till file is empty.
Any better ideas or commands?
This should work for you, list.txt is your file containing commands.
Make sure you backup the command file before running.
while read line; do $line;sed -i '1d' list.txt;done < "list.txt"
sed -i edits in-place so list.txt will be changed along the loop and you will end up with a empty file.
I think what you want to do is something like this:
while read -r -- i; do $i; sed -i "0,/$i/s/$i//;/^$/d" filename.txt; done < filename.txt
The file is read into the loop. Each line is executed, and the sed command will delete only the first entry it finds, then delete the empty line.
I think that one way to do it is to have the source file of all the commands to be executed, and the script that executes the commands also writes a second log file that lists the files as they are executed.
If you need to resume the process, you work on the lines in the source file that are not present in the log file.
logfile=commands.log
srcfile=commands.src
oldfile=commands.old
trap "mv $oldfile $logfile; exit 1" 0 1 2 3 13 15
[ -f $logfile ] || cp /dev/null $logfile
cp $logfile $oldfile
comm -23 $srcfile $logfile |
while read -r line
do
echo "$line" >> $oldfile
($line) < /dev/null
done
mv $oldfile $logfile
trap 0
I am trying to do an if/then statement, where if there is non-empty output from a ls | grep something command then I want to execute some statements. I am do not know the syntax I should be using. I have tried several variations of this:
if [[ `ls | grep log ` ]]; then echo "there are files of type log";
Well, that's close, but you need to finish the if with fi.
Also, if just runs a command and executes the conditional code if the command succeeds (exits with status code 0), which grep does only if it finds at least one match. So you don't need to check the output:
if ls | grep -q log; then echo "there are files of type log"; fi
If you're on a system with an older or non-GNU version of grep that doesn't support the -q ("quiet") option, you can achieve the same result by redirecting its output to /dev/null:
if ls | grep log >/dev/null; then echo "there are files of type log"; fi
But since ls also returns nonzero if it doesn't find a specified file, you can do the same thing without the grep at all, as in D.Shawley's answer:
if ls *log* >&/dev/null; then echo "there are files of type log"; fi
You also can do it using only the shell, without even ls, though it's a bit wordier:
for f in *log*; do
# even if there are no matching files, the body of this loop will run once
# with $f set to the literal string "*log*", so make sure there's really
# a file there:
if [ -e "$f" ]; then
echo "there are files of type log"
break
fi
done
As long as you're using bash specifically, you can set the nullglob option to simplify that somewhat:
shopt -s nullglob
for f in *log*; do
echo "There are files of type log"
break
done
Or without if; then; fi:
ls | grep -q log && echo 'there are files of type log'
Or even:
ls *log* &>/dev/null && echo 'there are files of type log'
The if built-in executes a shell command and selects the block based on the return value of the command. ls returns a distinct status code if it does not find the requested files so there is no need for the grep part. The [[ utility is actually a built-in command from bash, IIRC, that performs arithmetic operations. I could be wrong on that part since I rarely stray far from Bourne shell syntax.
Anyway, if you put all of this together, then you end up with the following command:
if ls *log* > /dev/null 2>&1
then
echo "there are files of type log"
fi
I'm trying to get the output of the ps command to output to a file, then to use that file to populate a radiolist. So far I'm having problems.
eval "ps -o pid,command">/tmp/process$$
more /tmp/process$$
sed -e '1d' /tmp/process$$ > /tmp/process2$$
while IFS= read -r pid command
do
msgboxlist="$msgboxlist" $($pid) $($command) "off"
done</tmp/process2$$
height=`wc -l "/tmp/process$$" | awk '{print $1}'`
width=`wc --max-line-length "/tmp/process$$" | awk '{print $1}'`
echo $height $width
dialog \
--title "Directory Listing" \
--radiolist "Select process to terminate" "$msgboxlist" $(($height+7)) $(($width+4))
So far not only does the while read not split the columns into 2 variables ($pid is the whole line and $command is blank) but when I try to run this the script is trying to run the line as a command. For example:
+ read -r pid command
++ 7934 bash -x assessment.ba
assessment.ba: line 322: 7934: command not found
+ msgboxlist=
+ off
assessment.ba: line 322: off: command not found
Basically I have no idea where I'm supposed to be putting quotes, double quotes and backslashes. It's driving me wild.
tl;dr Saving a command into a variable without running it, how?
You're trying to execute $pid and $command as commands:
msgboxlist="$msgboxlist" $($pid) $($command) "off"
Try:
msgboxlist="$msgboxlist $pid $command off"
Or use an array:
msgboxlist=() # do this before the while loop
msgboxlist+=($pid $command "off")
# when you need to use the whole list:
echo "${msgboxlist[#]}"
Your script can be refactored by removing some unnecessary calls like this:
ps -o pid=,command= > /tmp/process$$
msgboxlist=""
while read -r pid command
do
msgboxlist="$msgboxlist $pid $command off"
done < /tmp/process2$$
height=$(awk 'END {print NR}' "/tmp/process$$")
width=$(awk '{if (l<length($0)) l=length($0)} END{print l}' "/tmp/process$$")
dialog --title "Directory Listing" \
--radiolist "Select process to terminate" "$msgboxlist" $(($height+7)) $(($width+4))
I have to admit, I'm not 100% clear on what you're doing; but I think you want to change this:
msgboxlist="$msgboxlist" $($pid) $($command) "off"
to this:
msgboxlist+=("$pid" "$command" off)
which will add the PID, the command, and "off" as three new elements to the array named msgboxlist. You'd then change "$msgboxlist" to "${msgboxlist[#]}" in the dialog command, to include all of those elements as arguments to the command.
Use double quotes when you want variables to be expanded. Use single quotes to disable variable expansion.
Here's an example of a command saved for later execution.
file="readme.txt"
cmd="ls $file" # $file is expanded to readme.txt
echo "$cmd" # ls readme.txt
$cmd # lists readme.txt
Edit adressing the read:
Using read generally reads an entire line. Consider this instead (tested):
ps o pid=,command= | while read line ; do
set $line
pid=$1
command=$2
echo $pid $command
done
Also note the different usage of 'ps o pid=,command=' to skip displaying headers.