I am trying to find all text files in my directory and copy a list to text file. I am using a for loop to get the names. Then the output is being placed in the text file. However I am getting an indication of a syntax error near unexpected token$'do\r''`. What would be the best way to do this? Also for the text file name how could I save it with a date/timestamp name?
FILE_PATH="/my_folder/"
for f in $(find $FILE_PATH -type f -name '*.txt'); do
echo "file '$f'" >> "$FILE_PATH"mylist.txt;
done
Don't use for loop. In order for that for loop to execute, it first has to find all of the files that meet your criteria, then substitutes that list into your for loop. It means waiting for that ``findto process before yourfor` loop can even start to process the names. Plus, if there are lots of files, you can overload the command line and lose files you want to see.
Why not:
FILE_PATH="/my_folder"
find $FILE_PATH -type f -name '*.txt' > "$FILE_PATH/mylist.txt"
No need for a loop. No need to echo. Simple and clean.
Plus, you don't have issues with file names that may contain spaces.
You can avoid find using pathname expansion:
FILE_PATH="/my_folder/"
for f in "$FILE_PATH"/* ; do
test -f "$f" || continue # not a regular file
echo "$f"
done >| "$FILE_PATH/mylist.txt" # >| force overwrite
Related
experts, i have many folders and inside the folder there are many sub-folders and the sub-folders contain many files.However, in all the sub-folders one file name is same that is input.ps.Now i want to rename the same input.ps with full path plus file name
so input.ps in all directories should be renamed to home_wuan_data_filess_input.ps
i tried
#!/bin/sh
for file in /home/wuan/data/filess/*.ps
do
mv $file $file_
done
But it does not do the same as i expect,i hope experts will help me.Thanks in advance.
so input.ps in all directories should be renamed to home_wuan_data_filess_input.ps
You may use this find solution:
find /home/wuan/data/filess -type f -name 'input*.ps' -exec bash -c '
for f; do fn="${f#/}"; echo mv "$f" "${fn//\//_}"; done' _ {} +
while read line;
do
fil=${line//\//_}; # Converts all / characters to _ to form the file name
fil=${fil:1}; # Remove the first -
dir=${line%/*}; # Extract the directory
echo "mv $line $dir/$fil"; # Echo the move command
# mv "$line" "$dir/$fil"; # Remove the comment to perform the actual command
done <<< "$(find /path/to/dir -name "input.ps")"
Ok, so file will end up being
/home/wuan/data/filess/input.ps
What we need here is the path, and the full, snake-cased name. Let's start by getting the path:
for f in /home/wuan/data/filess/*.ps; do
path="${f%*/}";
This will match the substring of f up until the last occurrence of /, effectively giving us the path.
Next, we want to snake_case all the things, which is even easier:
for f in /home/wuan/data/filess/*.ps; do
path="${f%*/}";
newname="${f//\//_}"
This replaces all instances of / with _, giving the name you want the new file to have. Now let's put all of this together, and move the file f to path/newname:
for f in /home/wuan/data/filess/*.ps; do
path="${f%*/}";
newname="${f//\//_}"
mv "${f}" "${path}/${newname}"
done
That should do the trick
Here's one of many sites listing some of the bash string manipulations you can use.
Sorry for the delayed update, the power in my building just cut out :)
I have the following script and it has an error. I am trying to merge all the files into one large file. From the command line the cat commant works fine and the content is printed to the redirected file. From script it is working sometime but not the other time. I dont know why its behaving abnormally. Please help.
#!/bin/bash
### For loop starts ###
for D in `find . -type d`
do
combo=`find $D -maxdepth 1 -type f -name "combo.txt"`
cat $combo >> bigcombo.tsv
done
Here is the output of bash -x app.sh
++ find . -type d
+ for D in '`find . -type d`'
++ find . -maxdepth 1 -type f -name combo.txt
+ combo=
+ cat
^C
UPDATE:
The following worked for me. There was issue with the path. I still dont know what was the issue so answer is welcome.
#!/bin/bash
### For loop starts ###
rm -rf bigcombo.tsv
for D in `find . -type d`
do
psi=`find $D -maxdepth 1 -type f -name "*.psi_filtered"`
# This will give us only the directory path from find result i.e. removing filename.
directory=$(dirname "${psi}")
cat $directory"/selectedcombo.txt" >> bigcombo.tsv
done
The obvious problem is that you are attempting to cat a file which doesn't exist.
Secondary problems are related to efficiency and correctness. Running two nested loops is best avoided, though splitting the action into two steps is merely inelegant here; the inner loop will only execute once, at most. Capturing command results into variables is a common beginner antipattern; a variable which is only used once can often be avoided, and avoids littering the shell's memory with cruft (and coincidentally solves the multiple problems with missing quoting - a variable which contains a file or directory name should basically always be interpolated in double quotes). Redirection is better performed outside any containing loop;
rm file
while something; do
another thing >>file
done
will open, seek to the end of the file, write, and close the file as many times as the loop runs, whereas
while something; do
another thing
done >file
only performs the open, seek, and close actions once, and avoids having to clear the file before starting the loop. Though your script can be refactored to not have any loops at all;
find ./*/ -type f -name "*.psi_filtered" -execdir cat selectedcombo.txt \;> bigcombo.tsv
Depending on your problem, it might be an error for there to be directories which contain combo.txt but which do not contain any *.psi_filtered files. Perhaps you want to locate and examine these directories.
I create these files by this
for((i=0;i<20000;i++)
do
touch "www.google.com electronic&information this is bash test ${date} .txt"
done
Then, I want do replace "www.google.com" to "HNSD" .
in 20k files, all the filenames include the whitespace.
My first attempt was the following:
rename 's/www.google.com/HNSD/g' *
...but this yielded an "argument list too long" error.
My second attempt is the following:
#!/bin/bash
function _rename ()
{
while read line1; do
#rename 's/www.google.com/HNSD/g' $line1
sed -i "s/www.google.com/HNSD/g" $line1
#echo $line1
done
}
ls -1 | _rename
...but this doesn't rename the files given. How should this be done?
Running a single mv command per file won't cause this problem unless the filenames themselves are so long (or your environment is so full, thus crowding out space shared with argv use) that you can't fit both source and destination names on the command line at once. The following is thus generally quite safe:
for f in *www.google.com*; do
mv "$f" "${f//www.google.com/HNSD}"
done
Here, we're using parameter expansion to perform the rename operation.
The original code was problematic because it tried to put all filenames on the command line at once. If you use xargs or find -exec {} + to split your operation into multiple invocations, you won't have that problem. If you have rename, and want to call it as few times as possible to process all directory entries for the current location, that would look like the following:
printf '%s\0' * | xargs -0 rename 's/www.google.com/HNSD/'
Two options:
Use find, and rename using rename (or similar) within the -exec action:
find . -type f -name '*www.google.com*' -exec rename 's/www.google.com/HNSD/g' {} +
Leverage a for loop to iterate over the files:
for f in *www.google.com*; do rename 's/www.google.com/HNSD/g' "$f"; done
I'm having a really hard time finding an answer for this because most people want to find a list of files, then do the xargs.
I have a file with a list of file names. I would like to go through the list of file names in that file and one by one search a single directory (not recursive) and any time it finds the file, move it to a sub folder.
cat test.txt | xargs find . -type f -iname
At this point, I get this error:
find: paths must precede expression: file1
Why don't you use something like:
for i in `cat test.txt`
do
test -e $i && mv <src> <dst>
done
Hi all I am trying to write a script which recursively searches directories from a parent directory and counts how many text files (.txt) are in the sub directories. I also need to output the files relative path to the parent directory.
Lets say I have a folder named Files
Within this folder there may be:
Files/childFolder1/child1.txt
Files/childFolder1/child2.txt
Files/childFolder1/child3.txt
Files/childFolder2/child4.txt
Files/childFolder2/child5.txt
Files/child6.txt
So the output would be
The files are:
/childFolder1/child1.txt
/childFolder1/child2.txt
/childFolder1/child3.txt
/childFolder2/child4.txt
/childFolder2/child5.txt
/child6.txt
There are 6 files in the 'Files' folder
So far I have a script which is this:
#! /bin/csh
find $argv[1] -iname '*.txt'
set wc=`find $argv[1] -iname '*.txt' | wc -l`
echo "Number of files under" $argv[1] "is" $wc
I have no idea how to make the output so it only shows the file path relative to the directory. Currently my output is something like this:
/home/Alice/Documents/Files/childFolder1/child1.txt
/home/Alice/Documents/Files/childFolder1/child2.txt
/home/Alice/Documents/Files/childFolder1/child3.txt
/home/Alice/Documents/Files/childFolder2/child4.txt
/home/Alice/Documents/Files/childFolder2/child5.txt
/home/Alice/Documents/Files/child6.txt
Number of files under /home/Alice/Documents/Files is 6
Which is not desired. I am also worried about how I am setting the $wc variable. If the directory listing is large then this is going to acquire a massive overhead.
cd $argv[1] first and then use find . -iname '*.txt' to make the results relative to the directory
Try
find $argv[1] -iname '*.txt' -printf "%P \n"
This should give the desired output :)
Hayden
Also, if you don't want to execute find twice, you can either:
store its output in a variable, though I don't know if variables are limited in size in csh ;
store its output in a temporary file, but purists don't like temporary file;
count the number of lines yourself in a while loop which iterates over the results of find.
Note that if you used bash instead, which supports process substitution you could duplicate find's output and pipe it to multiple commands with tee:
find [...] | tee >(cmd1) >(cmd2) >/dev/null