There are multiple files in /opt/dir/ABC/ named allfile_123-abc allfile_123-def allfile_123-ghi allfile_123-xxx.
I need the files to be named new_name-abc.pgp new_name-def.pgp new_name-ghi.pgp new_name-xxx.pgp and then moved to /usr/tst/output
for file in /opt/dir/ABC/allfile_123* ;
do mv $file /usr/tst/output/"$file.pgp";
rename allfile_123 new_name /usr/tst/output/*.pgp ; done
I know the above doesn't work because $file = /opt/dir/ABC/allfile_123*. Is it possible to make this work, or is it a different command instead of 'for loop'?
This is for the Autosys application in which the jil contains a command to pass to the command line of a linux server running bash.
I could only find versions of each part of my question but not altogether and I was hoping to keep it on the command line of this jil. Unless a script is absolutely necessary.
No need for the loop, you can do this with just rename and mv:
rename -v 's/$/.pgp/' /opt/dir/ABC/allfile_123*
rename -v s/allfile_123/new_name/ /opt/dir/ABC/allfile_123*
mv /opt/dir/ABC/new_name* /usr/tst/output/
But I'm not sure the rename you are using is the same as mine.
However,
since the replacement you want to perform is fairly simple,
it's easy to do in pure Bash:
for file in /opt/dir/ABC/allfile_123*; do
newname=new_name${file##*allfile_123}.gpg
mv "$file" /usr/tst/output/"$newname"
done
If you want to write it on a single line:
for file in /opt/dir/ABC/allfile_123*; do newname=new_name${file##*allfile_123}.gpg; mv "$file" /usr/tst/output/"$newname"; done
Related
When I run mv * with no destination directory on a directory with say 10 files, I get an error as follows
root#tryit-apparent:~/test2# ls
file1.txt file10.txt file2.txt file3.txt file4.txt file5.txt file6.txt file7.txt file8.txt file9.txt
root#tryit-apparent:~/test2# mv *
mv: target 'file9.txt' is not a directory
When I run it on a directory with two files it overwrites the file with one just file.
root#tryit-apparent:~/test# ls
tempfile tempfile2
root#tryit-apparent:~/test# mv *
root#tryit-apparent:~/test# ls
tempfile2
I read the man pages but couldn't understand this behaviour. Would like to know what's causing this behavior and what's going on under the hood?
What other linux commands have such pitfalls and have destructive actions that are executed silently if the user is not aware of such behavior?
In Unix, unlike some other OSes, wildcards like * are expanded by the shell, before being passed to the command being run. So when you run mv * with tempfile and tempfile2 as the only files in the current directory, what the shell actually executes is mv tempfile tempfile2, which as normal will rename the first file over the second one, erasing the previous contents of tempfile2. The shell doesn't know or care that this command treats its last argument specially, and mv has no way of knowing that its two arguments came from a wildcard expansion. Hence the behavior you're seeing.
You can have similar issues even with more than two files. For instance, if you have files named tempfile1 through tempfile9 and a subdirectory named zyzzx, then mv * will move all your temp files into the zyzzx subdirectory.
Mostly, you just have to be aware that this is how wildcards work, and use caution with commands that treat one of their arguments specially (e.g. as a destination). cp is another one to watch out for, for the same reason. For interactive usage, you may want to get used to using the -i option to mv and cp, which asks for confirmation before overwriting files; or use an alias to make this the default.
Move is intented to move or rename a file or a directory, so you need a source and a destination.
If the path of the file is unchange then it becomes a rename operation.
If the path changes and the name remains the same it's a move.
You can do both by chaning the path and the name.
Man pages can be challenging to wrap your head around.
Googling can help: https://www.howtoforge.com/linux-mv-command/
Off the top of my head, you could do a cp operation followed by a rm to achieve similar results, but that's two steps, rather than one.
I am using an application that creates a text file on a Linux server. I then have the ability to execute a shell script (BASH 3.2.57) in which I need to convert the text file from Unix line endings to DOS and also change the extension of the file from .txt to .log.
I currently have a sed based command to do this. This command is rewritten by the application at run time to point to the specific folder and file name, in this example where you see ABC (all capital 3 letters in all my examples are a variable that can be any 3 letters).
pushd /rootfolder/parentfolder/ABC/
sed 's/$/\r/' prABC.txt > prABC.log
popd
The problem with this is that if a user runs the application for 2 different groups, say ABC and DEF at nearly the same time, the script will get overwritten with the DEF variables before ABC had a chance to fire off and do its thing with the file. Additionally the .txt is left in the folder regardless and I would like that to be removed.
A friend of mine came up with the following code that seems to work if its determined to be our best solution, but I would think and hope we have a cleaner more dynamic way to do this. Also this current method requires that when my user decides to add a GHI directory and file I now have to update the code, which i can program my application to do for me but i don't want this script to have to be rewritten every time the application wants to use it.
pushd /rootfolder/parentfolder/ABC
if [[ -f prABC.txt ]]
then
sed 's/$/\r/' prABC.txt > prABC.log
rm prABC.txt
fi
popd
pushd /rootfolder/parentfolder/DEF
if [[ -f prABC.txt ]]
then
sed 's/$/\r/' prABC.txt > prABC.log
rm prABC.txt
fi
popd
I would like to call this script at anytime from my application and it find any file named pr*.txt below the /rootfolder/parentfolder/ directory (if that has to include the parentfolder in its search that won't be a problem) and convert the line endings from LF to CRLF and change the extension of the file from .txt to .log.
I've done a ton of searching and have found near solutions for this but not exactly what I need and I want to be sure it's as safe as possible (issues with using "find with for". I don't know what utilities are installed on this build so i would like to keep it as basic/supportable as possible Thanks in advance :)
You should almost never need pushd and popd in scripts. In fact, you rarely need cd, either.
#!/bin/bash
for d in /rootfolder/parentfolder/ABC /rootfolder/parentfolder/DEF
do
if [[ -f "$d/prABC.txt" ]]
then
sed 's/$/\r/' "$d/prABC.txt" > "$d/prABC.log" &&
rm "$d/prABC.txt"
fi
done
Recall that a && b is shorthand for
if a; then
b
fi
In other words, if sed fails (because the source file can't be read, or the destination can't be written) we don't rm the source file. There should be an error message already so we don't add another one.
Not only is this more succinct, it is also easier to change if you decide that the old file should be renamed instead of removed, or you want to filter out all lines which contain "beef" in the sed script. Generally you should avoid repeated code; see also the DRY principle on Wikipedia.
Something is seriously wrong somewhere if you require DOS line endings in your files on Unix.
I'm trying to zip all the files within a directory which contains .py files individually. But after zipping the files the output that I'm seeing is .py.zip vs just .zip
Here's the one liner command that I'm trying to execute.
cd scripts/python/
for i in *; do zip $i.zip $i; done
This is what you are looking for:
for i in *py; do
zip "${i%.*}".zip "$i";
done
Explanation
${i%.*}: This makes use of Bash's built in parameter expansion. Here it tries to match everything after %. If it does find a match, it uses everything before the match. https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion for more information.
I am looking to write a script to move files from a directory:
/home/mydir/
To another directory based on tokens in the file name. I have a bunch of files named as such:
red_office_mike_2015_montreal_546968.ext
or
$color_$location_$name_$year_$city_$numbers.extension (files will be various movie files: mov, mp4, mkv, etc.)
I would like the script to move the files to the following location:
/dir/work/$color/$name
Then verify the file has successfully copied, and delete the original file once it has.
I would also love it if the script would create the to directory if it does not already exist.
So in summary, I need a script to move files based on underscore separated tokens, create the to directory if it doesn't already exist, verify the successful copy (maybe with a size check), then delete the original file.
I am working on linux, and would prefer a bash script. The variables I have given are generic, and I will incorporate some other things to the script, I'm just looking for help on building the skeleton.
Thanks in advance for any help!
It's not a bash script, but perl is much better at this kind of thing and is installed on all Linux systems
while(<>) {
chomp;
$file = $_;
($colour, $location, $name, $year, $city, $numbers) = split(/_/,$file);
$dest0 = "/dir/work/$colour";
$dest1 = "$dest0/$name";
mkdir ($dest0) unless (-d $dest0);
mkdir ($dest1) unless (-d $dest1);
rename ($file, "$dest1/$file");
}
The script splits your input file on the underscore character, creates all the directories to the destination and then renames the file to the new filename. Rename takes care of all the copying and deleting for you. In fact it just changes the directory entries without any copying at all.
UPDATE
The above version takes its input from a file containing a list of filenames to process. For an alternative version which processes all files in the current directory, replace the while line with
while(glob("*")) {
I was able to fumble around online and come up with a for loop to do this task. I used cut and it made things simple. Here is what worked for me:
#!/bin/sh
cd "${1:-.}"
for f in *.*; do
color=`echo "$f" | cut -d'_' -f1`
name=`echo "$f" | cut -d'_' -f3`
todir="/dir/work/$color/$name"
mkdir -p "$todir"
mv "$f" "$todir"
done
This worked perfectly and I hope it can help others who might need to create directories based on portions of filenames.
The first line under the shebang made it so that it will either look at the current working directory or a directory you pass it as an argument.
Thanks to those who chimed in on the original post. I'm new with scripting so it take me a while to figure this stuff out. I love this site though, it is super helpful!
I'm trying to use something along the lines of
unexpand -t 4 *.php
but am unsure how to write this command to do what I want.
Weirdly,
unexpand -t 4 file.php > file.php
gives me an empty file. (i.e. overwriting file.php with nothing)
I can specify multiple files okay, but don't know how to then overwrite each file.
I could use my IDE, but there are ~67000 instances of to be replaced over 200 files, and this will take a while.
I expect that the answers to my question(s) will be standard unix fare, but I'm still learning...
You can very seldom use output redirection to replace the input. Replacing works with commands that support it internally (since they then do the basic steps themselves). From the shell level, it's far better to work in two steps, like so:
Do the operation on foo, creating foo.tmp
Move (rename) foo.tmp to foo, overwriting the original
This will be fast. It will require a bit more disk space, but if you do both steps before continuing to the next file, you will only need as much extra space as the largest single file, this should not be a problem.
Sketch script:
for a in *.php
do
unexpand -t 4 $a >$a-notab
mv $a-notab $a
done
You could do better (error-checking, and so on), but that is the basic outline.
Here's the command I used:
for p in $(find . -iname "*.js")
do
unexpand -t 4 $(dirname $p)/"$(basename $p)" > $(dirname $p)/"$(basename $p)-tab"
mv $(dirname $p)/"$(basename $p)-tab" $(dirname $p)/"$(basename $p)"
done
This version changes all files within the directory hierarchy rooted at the current working directory.
In my case, I only wanted to make this change to .js files; you can omit the iname clause from find if you wish, or use different args to cast your net differently.
My version wraps filenames in quotes, but it doesn't use quotes around 'interesting' directory names that appear in the paths of matching files.
To get it all on one line, add a semi after lines 1, 3, & 4.
This is potentially dangerous, so make a backup or use git before running the command. If you're using git, you can verify that only whitespace was changed with git diff -w.