How can I use the relative path or absolute path as a single command line argument in a shell script?
For example, suppose my shell script is on my Desktop and I want to loop through all the text files in a folder that is somewhere in the file system.
I tried sh myshscript.sh /home/user/Desktop, but this doesn't seem feasible. And how would I avoid directory names and file names with whitespace?
myshscript.sh contains:
for i in `ls`
do
cat $i
done
Superficially, you might write:
cd "${1:-.}" || exit 1
for file in *
do
cat "$file"
done
except you don't really need the for loop in this case:
cd "${1:-.}" || exit 1
cat *
would do the job. And you could avoid the cd operation with:
cat "${1:-.}"/*
which lists (cats) all the files in the given directory, even if the directory or the file names contains spaces, newlines or other difficult to manage characters. You can use any appropriate glob pattern in place of * — if you want files ending .txt, then use *.txt as the pattern, for example.
This breaks down if you might have so many files that the argument list is too long. In that case, you probably need to use find:
find "${1:-.}" -type f -maxdepth 1 -exec cat {} +
(Note that -maxdepth is a GNU find extension.)
Avoid using ls to generate lists of file names, especially if the script has to be robust in the face of spaces, newlines etc in the names.
Use a glob instead of ls, and quote the loop variable:
for i in "$1"/*.txt
do
cat "$i"
done
PS: ShellCheck automatically points this out.
Related
I am trying to copy an existing file (that is found in across directories) to a new file with a new name in Bash. For example, 'Preview.json' to 'Performance.json'. I have tried using
find * -type f -name 'Preview.json' -exec cp {} {}"QA" \;
But ended up with 'Preview.jsonQA'. (I am new to Bash.) I have tried moving the "QA" in front of the {} but I got errors because of an invalid path.
In an -exec predicate, the symbol {} represents a path that is being considered, starting at one of the starting-point directories designated in the command. Example: start/dir2/Preview.json. You can form other file names by either prepending or appending characters, but whether that makes sense depends on the details. In your case, appending produces commands such as
cp start/dir2/Preview.json start/dir2/Preview.jsonQA
which is a plausible command in the event that start/dir2/Preview.json exists. But cp does not automatically create directories in the destination path, so the result of prepending characters ...
cp start/dir2/Preview.json QAstart/dir2/Preview.json
... is not as likely to be accepted -- it depends on directory QAstart/dir2 existing.
I think what you're actually looking for may be cp commands of the form ...
cp start/dir2/Preview.json start/dir2/QAPreview.json
... but find cannot do this by itself.
For more flexibility in handling the file names discovered by find, pipe its output into another command. If you want to pass them as command-line arguments to another command, then you can interpose the xargs command to achieve that. The command on the receiving end of the pipe can be a shell function or a compound command if you wish.
For example,
# Using ./* instead of * ensures that file names beginning with - will not
# be misinterpreted as options:
find ./* -type f -name 'Preview.json' |
while IFS= read -r name; do # Read one line and store it in variable $name
# the destination name needs to be computed differently if the name
# contains a / character (the usual case) than if it doesn't:
case "${name}" in
*/*) cp "${name}" "${name%/*}/QA${name##*/}" ;;
*) cp "${name}" "QA${name}" ;;
esac
done
Note that that assumes that none of your directory names contain newline characters (the read command would split up newline-containing filenames). That's a reasonably safe assumption, but not absolutely safe.
Of course, you would generally want to have that in a script, not to try to type it on the fly on the command line.
This question already has answers here:
Rename multiple files based on pattern in Unix
(24 answers)
Closed 5 years ago.
Write a simple script that will automatically rename a number of files. As an example we want the file *001.jpg renamed to user defined string + 001.jpg (ex: MyVacation20110725_001.jpg) The usage for this script is to get the digital camera photos to have file names that make some sense.
I need to write a shell script for this. Can someone suggest how to begin?
An example to help you get off the ground.
for f in *.jpg; do mv "$f" "$(echo "$f" | sed s/IMG/VACATION/)"; done
In this example, I am assuming that all your image files contain the string IMG and you want to replace IMG with VACATION.
The shell automatically evaluates *.jpg to all the matching files.
The second argument of mv (the new name of the file) is the output of the sed command that replaces IMG with VACATION.
If your filenames include whitespace pay careful attention to the "$f" notation. You need the double-quotes to preserve the whitespace.
You can use rename utility to rename multiple files by a pattern. For example following command will prepend string MyVacation2011_ to all the files with jpg extension.
rename 's/^/MyVacation2011_/g' *.jpg
or
rename <pattern> <replacement> <file-list>
this example, I am assuming that all your image files begin with "IMG" and you want to replace "IMG" with "VACATION"
solution : first identified all jpg files and then replace keyword
find . -name '*jpg' -exec bash -c 'echo mv $0 ${0/IMG/VACATION}' {} \;
for file in *.jpg ; do mv $file ${file//IMG/myVacation} ; done
Again assuming that all your image files have the string "IMG" and you want to replace "IMG" with "myVacation".
With bash you can directly convert the string with parameter expansion.
Example: if the file is IMG_327.jpg, the mv command will be executed as if you do mv IMG_327.jpg myVacation_327.jpg. And this will be done for each file found in the directory matching *.jpg.
IMG_001.jpg -> myVacation_001.jpg
IMG_002.jpg -> myVacation_002.jpg
IMG_1023.jpg -> myVacation_1023.jpg
etcetera...
find . -type f |
sed -n "s/\(.*\)factory\.py$/& \1service\.py/p" |
xargs -p -n 2 mv
eg will rename all files in the cwd with names ending in "factory.py" to be replaced with names ending in "service.py"
explanation:
In the sed cmd, the -n flag will suppress normal behavior of echoing input to output after the s/// command is applied, and the p option on s/// will force writing to output if a substitution is made. Since a sub will only be made on match, sed will only have output for files ending in "factory.py"
In the s/// replacement string, we use "& " to interpolate the entire matching string, followed by a space character, into the replacement. Because of this, it's vital that our RE matches the entire filename. after the space char, we use "\1service.py" to interpolate the string we gulped before "factory.py", followed by "service.py", replacing it. So for more complex transformations youll have to change the args to s/// (with an re still matching the entire filename)
Example output:
foo_factory.py foo_service.py
bar_factory.py bar_service.py
We use xargs with -n 2 to consume the output of sed 2 delimited strings at a time, passing these to mv (i also put the -p option in there so you can feel safe when running this). voila.
NOTE: If you are facing more complicated file and folder scenarios, this post explains find (and some alternatives) in greater detail.
Another option is:
for i in *001.jpg
do
echo "mv $i yourstring${i#*001.jpg}"
done
remove echo after you have it right.
Parameter substitution with # will keep only the last part, so you can change its name.
Can't comment on Susam Pal's answer but if you're dealing with spaces, I'd surround with quotes:
for f in *.jpg; do mv "$f" "`echo $f | sed s/\ /\-/g`"; done;
You can try this:
for file in *.jpg;
do
mv $file $somestring_${file:((-7))}
done
You can see "parameter expansion" in man bash to understand the above better.
I have files in directories and sub directories, the filename should be changed such that, the last character should be replaced to a number or a character depending upon arguments provided. I could do it for numbers but not happening for a character.
For eg. if File names are 20170504ABCDXXXYYY6.xml or 20170504CFLFXXXYYY6.cfl.bz2.
If I write the command ./updateLastCharacter 5, file names should be 20170504ABCDXXXYYY5.xml or 20170504CFLFXXXYYY5.cfl.bz2.
If the command is ./updateLastCharacter A, file names should be 20170504ABCDXXXYYYA.xml or 20170504CFLFXXXYYYA.cfl.bz2.
I'm new to shell scripting. I tried a lot for making it happen but what I could do is :
find $directory -exec rename "s/[0-9].xml/$newNumber.xml/;s/[0-9].cfl/$newRevisionNumber.cfl/" {} ";"
This works fine for number but I'm looking for how can I do it for a character with single line command.
A simple solution with rename ( perl function )
find . | rename -n -v 's/[A-Za-z0-9](?=\.)/5/'
-n means no action
-v means verbose to the screen
And you can use renrem program utility directly that is more powerful than rename in Perl.
I wrote that program to myself because I needed a lot to rename or remove files.
I have some files located in one directory /home/john
I want to copy all the files with *.text extension from this directory and save them as *.text.bkup, again in the same directory, i.e. /home/john
Is there a single command with which I can do that?
Also, with extension of the same idea, is it possible to copy all the files with multiple extentions (e.g. *.text & *.doc) as *.text.bkup & *.doc.bkup repectively (again in the same directory)?
This is best accomplished with a Shell loop:
~/tmp$ touch one.text two.text three.doc four.doc
~/tmp$ for FILE in *.text *.doc; do cp ${FILE} ${FILE}.bkup; done
~/tmp$ ls -1
four.doc
four.doc.bkup
one.text
one.text.bkup
three.doc
three.doc.bkup
two.text
two.text.bkup
What happens in the code above is the shell gets all .text and .doc files and then loops through each value one by one, assigning the variable FILE to each value. The code block between the "do" and the "done" is executed for every value of FILE, effectively copying each file to filename.bkup.
You can achieve this easily with find:
find /home/john -iname '*.text' -type f -exec cp \{} \{}.backup \;
No, there is no single/simple command to achieve this with standard tools
But you can write a script like this to do it for you.
for file in *.text
do
cp -i "${file}" "${file}.bkup"
done
with -i option you can confirm each overwriting operation
I sort of use a roundabout way to achieve this. It involves a Perl script and needs additional steps.
Step 1:
Copy the names of all the text files into a text file.
find -maxdepth 1 -type f -name '*.txt' > file_name1.txt
Step 2:
Make a duplicate of the copied file.
cp file_name1.txt file_name2.txt
Now open the file_name2.txt in vi editor and do a simple string substitution.
%s/.text/.text.backup/g
Step 3: Merge the source and destination file names into a single file separated by a comma.
paste -d, file_name1.txt file_name2.txt > file_name.txt
Step 4: Run the below perl script to achieve the desired results
open(FILE1,"<file_name.txt") or die'file doesnt exist'; #opens a file that has source and destination separated beforhand using commas
chomp(#F1_CONTENTS=(<FILE1>)); # copies the content of the file into an array
close FILE1;
while()
{
foreach $f1 (#F1_CONTENTS)
{
#file_name=split(/,/,$f1); # separates the file content based on commas
print "cp $file_name[0] $file_name[1]\n";
system ("cp $file_name[0] $file_name[1]"); # performs the actual copy here
}
last;
}
I am teaching myself more (l)unix skills and wanted to see if I could begin to write a program that will eventually read all .gz files and expand them. However, I want it to be super dynamic.
#!/bin/bash
dir=~/derp/herp/path/goes/here
for file in $(find dir -name '*gz')
do
echo $file
done
So when I excute this file, I simply go
bash derp.sh.
I don't like this. I feel the script is too brittle.
How can I rework my for loop so that I can say
bash derp.sh ~/derp/herp/path/goes/here (1)
I tried re-coding it as follows:
for file in $*
However, I don't want to have to type in bash
derp.sh ~/derp/herp/path/goes/here/*.gz.
How could I rewrite this so I could simply type what is in (1)? I feel I must be missing something simple?
Note
I tried
for file in $*/*.gz and that obviously did not work. I appreciate your assistance, my sources have been a wrox unix text, carpentry v5, and man files. Unfortunately, I haven't found anything that will what I want.
Thanks,
GeekyOmega
for dir in "$#"
do
for file in "$dir"/*.gz
do
echo $file
done
done
Notes:
In the outer loop, dir is assigned successively to each argument given on the command line. The special form "$#" is used so that the directory names that contain spaces will be processed correctly.
The inner loop runs over each .gz file in the given directory. By placing $dir in double-quotes, the loop will work correctly even if the directory name contains spaces. This form will also work correctly if the gz file names have spaces.
#!/bin/bash
for file in $(find "$#" -name '*.gz')
do
echo $file
done
You'll probably prefer "$#" instead of $*; if you were to have spaces in filenames, like with a directory named My Documents and a directory named Music, $* would effectively expand into:
find My Documents Music -name '*.gz'
where "$#" would expand into:
find "My Documents" "Music" -name '*.gz'
Requisite note: Using for file in $(find ...) is generally regarded as a bad practice, because it does tend to break if you have spaces or newlines in your directory structure. Using nested for loops (as in John's answer) is often a better idea, or using find -print0 and read as in this answer.