echo 'Sum'
awk '!found[$1]++' file.txt file2.txt | sort -n
echo 'A\B:'
sort file2.txt file2.txt file.txt | uniq -u
It just prints like 3 errors, file not found, command not found, 'ort wrong option', 'niq wrong option', whatever 'niq' and 'ort' are. Does it have some syntax errors or i need to install some packages? It all worked fine on cygwin on Windows.
Related
I have a file testtns.txt which has numbers like below :
123
456
I am then passing the input of the file folder path like "/var/www/batchfiles/files/test.csv" . The test.csv has following records :
1,123,On its way to warehouse,20230131
2,456,On its way to warehouse,20230201
3,777,Pickedup,20230201
4,888,Pickedup,20230202
I have created the script printgrep.bash to read the numbers from the file testtns.txt and then grep the csv files in folder "/var/www/batchfiles/files/". I am running the script using command .\printgrep.bash /var/www/batchfiles/files/* and excepting the output to be in output.txt as follows :
1,123,On its way to warehouse,20230131
2,456,On its way to warehouse,20230201
However when I run the above command, output.txt is empty and doesnt have any results. However if I run the grep command as it is, it does return results as excepted.
Can someone let me know why the grep command not working in the below script printgrep.bash :
#!/bin/bash
cat testtns.txt | while read line
do
grep -i "^ $line" $1 >> output.txt
done
Tried even below grep command but still didnt work :
#!/bin/bash
cat testtns.txt | while read line
do
grep -i "$line" $1 >> output.txt
done
I like to use the cat command to join several *.txt files under mac osx.
my first file1.txt looks like:
a;b;c;d
1;2;3;4
second file2.txt:
a;b
5;6
7;8
what I want:
a;b;c;d
1;2;3;4
5;6;;
7;8;;
my question: can I skip the header from the second file in the output file? And how is cat dealing with the missing columns? writing NaNs?
maybe this command could do it?
head -1 file1.txt > all.txt;
tail -n +2 -q file*.txt >> all.txt
I don't think the cat command alone will deal with removing the headers or mark any missing columns, since all it does is concatenate files. But if you know the highest possible number of columns, you can do something like this:
cat file1.txt <( tail -n+2 file2.txt ) | gawk -F';' -v OFS=';' '{NF=4}1'
Where NF=4 is the highest number of columns (in your example, 4).
The command above is concatenating file1.txt with a header-less version of file2.txt, using the output of a subcommand as input (operator <( ) ). You can use the <( ) as many times you want for each file you're wanting to concatenate. The final command, gawk, was adapted from this answer) and it's padding out the column delimiters for you.
(note: use brew install gawk if gawk isn't found; Mac OS X's awk won't work)
If not having the first header doesn't bother you and you don't want to use cat, you could do:
gawk -F';' -v OFS=';' '{NF=4}1' file*.txt | egrep -v '^a;b'
I am running a script that has been working fine. However, yesterday, I got a couple errors. These errors are after several loops of the script:
sed: cant read file3.txt: No such file or directory
grep: file3.txt: No such file or directory
grep: file3.txt: No such file or directory
sed: cant read file3.txt: No such file or directory
grep: file3.txt: No such file or directory
Keep in mind, these errors do not happen consistently. It's occurring once in a while somewhere near this part of the script. File3.txt is the file not being found:
cat file1.txt | while read LINE; do grep -m 1 $LINE file2.txt >> file3.txt; done
sed -i 's/string//g' file3.txt
grep 'string' file3.txt | cut -d '|' -f1-2 > file4.txt
grep -v 'string' file3.txt | cut -d '|' -f1-2 >> file5.txt
sed -i 's/string//' file3.txt
grep -Fvf file3.txt file1.txt > file6.txt
Now, I'm thinking that since file3.txt is being appended, or later operated on by SED, sometimes the next command starts too soon and it can't find the file? Should I put a wait command in between?
I have looked up many pages with this error, but was unable to find anything:
cat file_name | grep "something" results "cat: grep: No such file or directory" in shell scripting
Pipe multiple commands to a single command with no EOF signal wait
grep command works in command line, but not in bash script: get no such file or directory erro
https://serverfault.com/questions/169539/sed-cant-find-a-file-that-obviously-exists
"No such file or directory" but it exists
If you think that putting a wait or sleep command will help, please let me know. Or, if you think there's a better solution, that would be great too. I'm running on Cygwin terminal. Any insight is greatly appreciated.
Instead of redirecting to file3.txt inside the while loop, redirect the whole loop. Then the file will be created even if the loop never runs because the input file is empty.
while read LINE; do
grep -m 1 $LINE file2.txt
done < file1.txt > file3.txt
If file1.txt is ever empty then file3.txt won't be created.
Also do grep -m 1 $LINE file2.txt will cause problems if there are crucial characters (space is the easiest of them).
Let's assume that the $LINE variable contains more than one word separated by spaces: hello world.
Now the command looks like this: grep -m 1 hello world file2.txt - grep interpretation will look something like this: let's find all hello in file named world and file named file2.txt in current folder.
Using "$LINE" instead of $LINE will lead you to a whole different scenario.
Look at the difference between the following two:
grep -m 1 $LINE file2.txt
grep -m 1 "$LINE" file2.txt
So here is the task which I can't solve. I have a directory with .h files and a directory with .i files, which have the same names as the .h files. I want just by typing a command to have all .h files which are not found as .i files. It's not a hard problem, I can do it in some programming language, but I'm just curious how it will look like in cmd :). To be more specific here is the algo:
get file names without extensions from ls *.h
get file names without extensions from ls *.i
compare them
print all names from 1 that are not met in 2
Good luck!
diff \
<(ls dir.with.h | sed 's/\.h$//') \
<(ls dir.with.i | sed 's/\.i$//') \
| grep '$<' \
| cut -c3-
diff <(ls dir.with.h | sed 's/\.h$//') <(ls dir.with.i | sed 's/\.i$//') executes ls on the two directories, cuts off the extensions, and compares the two lists. Then grep '$<' finds the files that are only in the first listing, and cut -c3- cuts off the "< " characters that diff inserted.
ls ./dir_h/*.h | sed -r -n 's:.*dir_h/([^.]*).h$:dir_i/\1.i:p' | xargs ls 2>&1 | \
grep "No such file or directory" | awk '{print $4}' | sed -n -r 's:dir_i/([^:]*).*:dir_h/\1:p'
ls -1 dir1/*.hh dir2/*.ii | awk -F"/" '{print $NF}' |awk -F"." '{a[$1]++;b[$0]}END{for(i in a)if(a[i]==1 && b[i".hh"]) print i}'
explanation:
ls -1 dir1/*.hh dir2/*.ii
above will list all the files *.hh and *.ii files in both the directories.
awk -F"/" '{print $NF}'
above will just print the file name excluding the complete path of the file.
awk -F"." '{a[$1]++;b[$0]}END{for(i in a)if(a[i]==1 && b[i".hh"]) print i}'
above will create two associative arrays one with file name and one with excluding the extension.
if both hh and ii files exist the value in the assosciative array will 2 if there is only one file then the value will be 1.so we need array item whose value is 1 and it should be a header file (.hh).
this can be checked using the asso..array b which is done in the END block.
Assuming bash is your shell:
for file in $( ls dir_with_h/*.h ); do
name=${file%\.h}; # trim trailing ".h" file extension
name=${name#dir_with_h/}; # trim leading folder name
if [ ! -e dir_with_i/${name}.i ]; then
echo ${name};
fi
done
Undoubtedly this can be ported to virtually all other shells. I find this less cryptic than some other approaches (although this is surely my problem) but it is a little wordy. As such. a shell script might help recall it.
I'm having some rather unusual problems using grep in a bash script. Below is an example of the bash script code that I'm using that exhibits the behaviour:
UNIQ_SCAN_INIT_POINT=1
cat "$FILE_BASENAME_LIST" | uniq -d >> $UNIQ_LIST
sed '/^$/d' $UNIQ_LIST >> $UNIQ_LIST_FINAL
UNIQ_LINE_COUNT=`wc -l $UNIQ_LIST_FINAL | cut -d \ -f 1`
while [ -n "`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`" ]; do
CURRENT_LINE=`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`
CURRENT_DUPECHK_FILE=$FILE_DUPEMATCH-$CURRENT_LINE
grep $CURRENT_LINE $FILE_LOCTN_LIST >> $CURRENT_DUPECHK_FILE
MATCH=`grep -c $CURRENT_LINE $FILE_BASENAME_LIST`
CMD_ECHO="$CURRENT_LINE matched $MATCH times," cmd_line_echo
echo "$CURRENT_DUPECHK_FILE" >> $FILE_DUPEMATCH_FILELIST
let UNIQ_SCAN_INIT_POINT=UNIQ_SCAN_INIT_POINT+1
done
On numerous occasions, when grepping for the current line in the file location list, it has put no output to the current dupechk file even though there have definitely been matches to the current line in the file location list (I ran the command in terminal with no issues).
I've rummaged around the internet to see if anyone else has had similar behaviour, and thus far all I have found is that it is something to do with buffered and unbuffered outputs from other commands operating before the grep command in the Bash script....
However no one seems to have found a solution, so basically I'm asking you guys if you have ever come across this, and any idea/tips/solutions to this problem...
Regards
Paul
The `problem' is the standard I/O library. When it is writing to a terminal
it is unbuffered, but if it is writing to a pipe then it sets up buffering.
try changing
CURRENT_LINE=`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`
to
CURRENT LINE=`sed "$UNIQ_SCAN_INIT_POINT"'q;d' $UNIQ_LIST_FINAL`
Are there any directories with spaces in their names in $FILE_LOCTN_LIST? Because if they are, those spaces will need escaped somehow. Some combination of find and xargs can usually deal with that for you, especially xargs -0
A small bash script using md5sum and sort that detects duplicate files in the current directory:
CURRENT="" md5sum * |
sort |
while read md5sum filename;
do
[[ $CURRENT == $md5sum ]] && echo $filename is duplicate;
CURRENT=$md5sum;
done
you tagged linux, some i assume you have tools like GNU find,md5sum,uniq, sort etc. here's a simple example to find duplicate files
$ echo "hello world">file
$ md5sum file
6f5902ac237024bdd0c176cb93063dc4 file
$ cp file file1
$ md5sum file1
6f5902ac237024bdd0c176cb93063dc4 file1
$ echo "blah" > file2
$ md5sum file2
0d599f0ec05c3bda8c3b8a68c32a1b47 file2
$ find . -type f -exec md5sum "{}" \; |sort -n | uniq -w32 -D
6f5902ac237024bdd0c176cb93063dc4 ./file
6f5902ac237024bdd0c176cb93063dc4 ./file1