Hello stack overflow -
I would like to take an input text file (one line per file to find) to do two things: copy the files that are found to a different directory and provide a message whether the file is not found. The message does not have to exclusively say a file is not found; it can also include the location of files that are found like the output displayed below. I have not been able to combine the two commands below. Is this possible? I am sure there are alternative solutions and am open to those
#will tell you if a file is not found or the location of the file if found:
command:
for i in $(cat toGet.txt); do find . -name "$i" | grep . || echo "$i - file not found" ; done
output:
file1_L001_R*_001.fastq.gz - file not found
./file2_S13_L001_R2_001.fastq.gz
./file2_S13_L001_R1_001.fastq.gz
file3_L001_R*_001.fastq.gz - file not found
#will copy files found to new directory
for i in $(cat toGet.txt); do find . -name "$i" -exec cp {} /path/to/directory \; ; done
Any suggestions
Write a script that receives the filename to copy on its standard input. If the input is empty, it reports that the file is not found, otherwise it copies it. Then pipe the find output to it.
copy_to.sh:
#!/bin/sh
looking_for=$1
dest_dir=$2
found=$(cat)
if [ -z "$found" ]
then echo "$looking_for - file not found"
else cp "$found" "$dest_dir"
while read -r i; do
find . -name "$i" | ./copy_to.sh "$i" /path/to/directory
done < toGet.txt
Related
I have several folders inside a directory called freesurfer like:
freesurfer/sub-001/mri
freesurfer/sub-001/label
freesurfer/sub-001/surf
freesurfer/sub-001/stats
freesurfer/sub-002/mri
freesurfer/sub-002/label
freesurfer/sub-002/surf
freesurfer/sub-002/stats
I would like to rename only 'stats' folder and move it to a different location:
freesurfer/sub-001/mri
freesurfer/sub-001/label
freesurfer/sub-001/surf
freesurfer/sub-001/sub-001_stats
freesurfer/sub-002/mri
freesurfer/sub-002/label
freesurfer/sub-002/surf
freesurfer/sub-002/sub-002_stats
I have managed to write the following code:
while read dir ; do
new_dir=$(rev <<< "$dir" | sed 's~/~_~' | rev);
echo "mv $dir $new_dir";
done < <(find . -type d -name 'stats')
The above code seems to be working fine when I run it with 'echo' before 'mv' command and appends the folder name in front of 'stats' directory but when I take the 'echo' off, it gives an error saying: "No such file or directory".
What am I doing wrong?
Your code to create new_dir isn't correct. It's creating freesurfer/sub-002_stats instead of freesurfer/sub-002/sub-002_stats. But this doesn't explain the error you're getting, since it will just rename the file incorrectly (moving all of them into the parent directory).
Try:
while read dir ; do
new_dir=$(dirname "$dir")/$(basename $(dirname "$dir"))_stats
mv "$dir" "$new_dir"
done < <(find . -type d -name 'stats')
I have a directory with files in the following strcuture:
HomeTransaction1/Date1/transactionfile1.txt
HomeTransaction1/Date1/transactionfile1.xls
HomeTransaction1/Date1/transactionfile2.xls
HomeTransaction1/Date1/transactionfile2.txt
HomeTransaction1/Date1/transactionfile3.txt
HomeTransaction1/Date2/transactionfile1.txt
HomeTransaction1/Date3/transactionfile2.txt
HomeTransaction1/Date3/transactionfile3.txt
HomeTransaction2/Date1/transactionfile1.txt
HomeTransaction2/Date1/transactionfile2.txt
HomeTransaction3/Date1/transactionfile3.txt
I'm trying to get for a specific thing in the transaction files that end in .txt so I'm trying to come up with a bash script to achieve this. Conceptually, this is my thought process.
A - List each folder in the current directory. I this example, it'll be HomeTransaction1, HomeTransaction2 and HomeTransaction3
B - for each folder in B list all the folders(the Date folders)
C - for each folder in step B, run "grep " for files with .txt extension
This is what I have come up with so far:
#!/bin/bash
for FILE in `ls -l`
do
if test -d $FILE && (startswith "HomeTrasaction") //I want to add a condition to check that the directory name and starts with "HomeTrasaction"
then
cd $FILE // example cd to 'HomeTransaction1' directory
echo "In HomeTransaction directory $FILE"
for SUB_FILE in `ls -l`
do
cd $SUB_FILE //example cd to 'Date1'
echo "In Date directory $FILE"
for TS_FILES in ($find . -print | grep .txt)
grep "text-to-search" $SUB_FILE
fi
done
I appreciate any help in finalizing my script. Thank you.
The solution is actually pretty simple
find ./HomeTrasaction* -iname "*.txt" -exec grep -i "phrase" {} \;
find ./HomeTrasaction* - search each directory that start with this phrase, in the current directory.
-iname "*.txt" - for each file that ends with .txt
-exec grep -i "phrase" {} ; - grep for the word "phrase"
If this is still not clear "man find" :)
I'd like to search in the directory structure recursively for files of the specific file types. But I need to pass the file types from the external file. The output should be list where each line is absolute path to the file. I will use the output for further processing.
The external file where is the list of file types looks for example like this (filter.lst):
*.properties
I've tried this (searchfiles.sh):
while read line
do
echo "$(find $1 -type f -name $line)"
done < $2
Echo command inside the script is only for the test purpose. I ran the script:
./searchfiles.sh test_scripting filter.lst
The the output of the echo of the find command was empty. Why? I tried to alter the script in the following way to test if the command is built correctly and the files *.properties exist:
while read line
do
echo "find $1 -type f -name $line"
echo "$(find $1 -type f -name $line)"
done < $2
I've got output:
./searchfiles.sh test_scripting filter.lst
find test_scripting -type f -name *.properties
If I copy manualy "find test_scripting -type f -name *.properties" and paste it to the shell the files are correctly found:
find test_scripting -type f -name *.properties
test_scripting/dir1/audit.properties
test_scripting/audit.properties
test_scripting/dir2/audit.properties
Why does not "find" command process correctly the variables?
The cause of the strange behaviour were hidden characters in the input filter.lst file. The filter.lst was created in the Windows OS and then copied to the Linux OS. Hence the find command didn't find expected files. Test the input file if it contains hidden characters:
od -c filter.lst
0000000 * . p r o p e r t i e s \r \n
0000016
The hidden character is "\r". Edit the script to remove hidden characters using sed command in each line.
while read line
do
echo "$(find $1 -type f -name $(echo $line | sed -e 's/\r$//'))"
done < $2
More about removing hidden characters is in this thread.
Notice: The best way is to run the script in the empty directory. If there is the file with the name e.g. example.properties in the directory where you run the script, the "echo $line" (executed as echo *.properties) will only display the list of .properties files - in this case only file example.properties.
I need a Bash Script to Execute a program for all directories that do not have a specific file and create the output file on the same directory.This program needs an input file which exist in every directory with the name *.DNA.fasta.Suppose I have the following directories that may contain sub directories also
dir1/a.protein.fasta
dir2/b.protein.fasta
dir3/anyfile
dir4/x.orf.fasta
I have started by finding the directories that don't have that specific file whic name is *.protein.fasta
in this case I want the dir3 and dir4 to be listed (since they do not contain *.protein.fasta)
I have tried this code:
find . -maxdepth 1 -type d \! -exec test -e '{}/*protein.fasta' \; -print
but it seems I missed some thing it does not work.
also I do not know how to proceed for the whole story.
This is a tricky one.
I can't think of a good solution. But here's a solution, nevertheless. Note that this is guaranteed not to work if your directory or file names contain newlines, and it's not guaranteed to work if they contain other special characters. (I've only tested with the samples in your question.)
Also, I haven't included a -maxdepth because you said you need to search subdirectories too.
#!/bin/bash
# Create an associative array
declare -A excludes
# Build an associative array of directories containing the file
while read line; do
excludes[$(dirname "$line")]=1
echo "excluded: $(dirname "$line")" >&2
done <<EOT
$(find . -name "*protein.fasta" -print)
EOT
# Walk through all directories, print only those not in array
find . -type d \
| while read line ; do
if [[ ! ${excludes[$line]} ]]; then
echo "$line"
fi
done
For me, this returns:
.
./dir3
./dir4
All of which are directories that do not contain a file matching *.protein.fasta. Of course, you can replace the last echo "$line" with whatever you need to do with these directories.
Alternately:
If what you're really looking for is just the list of top-level directories that do not contain the matching file in any subdirectory, the following bash one-liner may be sufficient:
for i in *; do test -d "$i" && ( find "$i" -name '*protein.fasta' | grep -q . || echo "$i" ); done
#!/bin/bash
for dir in *; do
test -d "$dir" && ( find "$dir" -name '*protein.fasta' | grep -q . || Programfoo"$dir/$dir.DNA.fasta");
done
i have directory that has 2 sub-directories and that again has few sub-directory and they have some files. I need to rename all the files to append an html extension to the filenames.
the directory structure looks like this
main-directory
sub-directory
sub-directory
sub-directory
file1
file2
and so on to lot of files
now i could not use something like this
for file in main-directory/*
do
if [ -f "$file" ]
then `mv "$file" "$file.html"`
fi
done
because the for loop wont use the path recursively. so i used something like this
for file in `ls -1R main-directory` // -1 for showing file and directory names separated by new lines and -R for recursive travel
do
if [ -f "$file" ]
then `mv "$file" "$file.html"`
fi
done
the above code is not able to rename files. to check whether the line
for file in `ls -1R main-directory`
is working i wrote something like this
for file in `ls -1R main-directory`
do
if [ -f "$file" ]
echo $file
done
this doesn't show anything. what can be wrong?
you can use find and look into of type file and then -exec to change all the file and then appending the .html.
find main-directory -type f -exec mv -v '{}' '{}'.html \;
In your first for loop, the mv command should not be in back-ticks.
In your second for loop, the if-statement has incorrect syntax. There is no then or fi. It should be:
for file in `ls -1R main-directory`
do
if [ -f "$file" ]
then
echo $file
fi
done
But even then, this won't work because ls -1R main-directory gives you just the file names, not the absolute paths to the file. Move your echo outside the if-statement to test:
for file in `ls -1R main-directory`
do
echo $file
done
Therefor ls -1R main-directory is not a good way to get all files in the current directory. Use find . -type f instead.
For some reason, I can never remember the find ... -exec syntax off the top of my head with the {} and the \;. Instead, I've fallen into the habit of just using a loop fed from find:
find main-directory -type f | while read file
do
mv "$file" "$file.html"
done
find outputs each file to stdout and the read file will consume one line at a time and set the contents of that line to the $file environment variable. You can then use that anywhere in the body of your loop.
I use this approach to solve lots of little problems like this where I need to loop over a bunch of output and do something useful. Because it is more versatile, I use it more than the esoteric find ... -exec {} \; approach.
Another trick is to prepend you command with echo to do a quick sanity check before doing potentially damaging things to your system:
find find main-directory -type f | while read file
do
echo mv "$file" "$file.html"
done
here is the answer to my question. people responded by 1 liners which are a neat approach but i didnt get much out of those 1 liners so here is something that i wanted
IFS=$'\n' // this is for setting field separator to new line because the default is whitespace
dir=main-directory
for file in `ls -1R main-directory | sed 's/:$//'` // -1 for showing file and directory names separated by new lines and -R for recursive travel
do
if [ -f "$dir/$file" ]
then `mv "$dir/$file" "$dir/$file.html"`
elif [ -d "$file" ]
then dir=$file
fi
done
here the sed 's/:$//' detects a : at the end of line and removes it. this was one of the things that prevented my code to run because whenever ls -1R main-directory detected a directory it appended a : at the end