I need to copy all files from
/dirA/[NAME].20151231.txt
to
/dirB/20151231.[NAME].txt
and
/dirC/20151231/[NAME].txt
i.e. I need to copy the files, but change the name.
You can assume that I know the "date" string before hand, so we can assume 20151231 is a supplied argument.
if I have a list of names, I can do something like
for n in $names; do; cp /dirA/$n.$date.txt /dirB/$date.$n.txt; done;
But what if I dont have a list of names? I am looking for an elegant solution as extracting them from dirA sounds a bit cumbersome.
Thanks!
A reasonably reliable way of processing this material is:
date=20151231
cd /dirA || exit 1
mkdir -p "/dirC/$date" || exit 1
for file in *."$date.txt"
do
name="${file%.$date.txt}"
cp "$file" "/dirB/$date.$name.txt"
cp "$file" "/dirC/$date/$name.txt"
done
The cd operation is checked; if it fails, there is no point in continuing. Likewise, the mkdir -p operation ensures that the dated directory under /dirC exists or exits. The relevant error messages were already generated by cd and mkdir.
Using the shell globbing to generate the file names is best; it avoids issues with 'what happens if the file name contains spaces (or newlines, or other unexpected characters)'.
The assignment extracts the '[NAME]' portion of the file name. This is then used to copy the file from /dirA to the relevant locations under /dirB and /dirC. It would be feasible to check that /dirB and /dirC also exist if you thought that was necessary.
Maybe I am just awful at asking questions. What I was looking for was a "sed for file names". And I found the answer -- that's rename.
Related
I'm relatively new to bash and I have tried multiples solutions that I could find here but none of them seem to be working in my case. It's pretty simple, I have a folder that looks like this:
- images/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
and I would like to move these jpg files into subfolders based on the prefix number like so:
- images_0/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- images_1/
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
Is there a bash command that could do that in a simple way ?
Thank you
for src in *_*.jpg; do
dest=images_${src%%_*}/
echo mkdir -p "$dest"
echo mv -- "$src" "$dest"
done
Remove both echos if the output looks good.
I would do this with rename a.k.a. Perl rename. It is extremely powerful and performant. Here's a command for your use case:
rename --dry-run -p '$_="images_" . substr($_,0,1) . "/" . $_' ?_*jpg
Let's dissect that. At the right end, we specify we only want to work on files that start with a single character/digit before an underscore so we don't do damage trying to apply the command to files it wasn't meant for. Then --dry-run means it doesn't actually do anything, it just shows you what it would do - this is a very useful feature. Then -p which handily means "create any necessary directories for me as you go". Then the meat of the command. It passes you the current filename in a variable called $_ and we then need to create a new variable called $_ to say what we want the file to be called. In this case we just want the word images_ followed by the first digit of the existing filename and then a slash and the original name. Simples!
Sample Output
'0_image_1.jpg' would be renamed to 'images_0/0_image_1.jpg'
'0_image_2.jpg' would be renamed to 'images_0/0_image_2.jpg'
'1_image_3.jpg' would be renamed to 'images_1/1_image_3.jpg'
Remove the --dry-run and run again for real, if the output looks good.
Using rename has several benefits:
that it will warn and avoid any conflicts if two files rename to the same thing,
that it can rename across directories, creating any necessary intermediate directories on the way,
that you can do a dry run first to test it,
that you can use arbitrarily complex Perl code to specify the new name.
Note: On macOS, you can install rename using homebrew:
brew install rename
Note: On some Ones, rename is referred to as prename for Perl rename.
I've got (what feels like) a fairly simple problem but my complete lack of experience in bash has left me stumped. I've spent all day trying to synthesize a script from many different SO threads explaining how to do specific things with unintuitive commands, but I can't figure out how to make them work together for the life of me.
Here is my situation: I've got a directory full of nested folders each containing a file with extension .7 and another file with extension .pc, plus a whole bunch of unrelated stuff. It looks like this:
Folder A
Folder 1
Folder x
data_01.7
helper_01.pc
...
Folder y
data_02.7
helper_02.pc
...
...
Folder 2
Folder z
data_03.7
helper_03.pc
...
...
Folder B
...
I've got a script that I need to run in each of these folders that takes in the name of the .7 file as an input.
pc_script -f data.7 -flag1 -other_flags
The current working directory needs to be the folder with the .7 file when running the script and the helper.pc file also needs to be present in it. After the script is finished running, there are a ton of new files and directories. However, I need to take just one of those output files, result.h5, and copy it to a new directory maintaining the same folder structure but with a new name:
Result Folder/Folder A/Folder 1/Folder x/new_result1.h5
I then need to run the same script again with a different flag, flag2, and copy the new version of that output file to the same result directory with a different name, new_result2.h5.
The folders all have pretty arbitrary names, though there aren't any spaces or special characters beyond underscores.
Here is an example of what I've tried:
#!/bin/bash
DIR=".../project/data"
for d in */ ; do
for e in */ ; do
for f in */ ; do
for PFILE in *.7 ; do
echo "$d/$e/$f/$PFILE"
cd "$DIR/$d/$e/$f"
echo "Performing operation 1"
pc_script -f "$PFILE" -flag1
mkdir -p ".../results/$d/$e/$f"
mv "results.h5" ".../project/results/$d/$e/$f/new_results1.h5"
echo "Performing operation 2"
pc_script -f "$PFILE" -flag 2
mv "results.h5" ".../project/results/$d/$e/$f/new_results2.h5"
done
done
done
done
Obviously, this didn't work. I've also tried using find with -execdir but then I couldn't figure out how to insert the name of the file into the script flag. I'd appreciate any help or suggestions on how to carry this out.
Another, perhaps more flexible, approach to the problem is to use the find command with the -exec option to run a short "helper-script" for each file found below a directory path that ends in ".7". The -name option allows find to locate all files ending in ".7" below a given directory using simple file-globbing (wildcards). The helper-script then performs the same operation on each file found by find and handles moving the result.h5 to the proper directory.
The form of the command will be:
find /path/to/search -type f -name "*.7" -exec /path/to/helper-script '{}` \;
Where the -f option tells find to only return files (not directories) ending in ".7". Your helper-script needs to be executable (e.g. chmod +x helper-script) and unless it is in your PATH, you must provide the full path to the script in the find command. The '{}' will be replaced by the filename (including relative path) and passed as an argument to your helper-script. The \; simply terminates the command executed by -exec.
(note there is another form for -exec called -execdir and another terminator '+' that can be used to process the command on all files in a given directory -- that is a bit safer, but has additional PATH requirements for the command being run. Since you have only one ".7" file per-directory -- there isn't much benefit here)
The helper-script just does what you need to do in each directory. Based on your description it could be something like the following:
#!/bin/bash
dir="${1%/*}" ## trim file.7 from end of path
cd "$dir" || { ## change to directory or handle error
printf "unable to change to directory %s\n" "$dir" >&2
exit 1
}
destdir="/Result_Folder/$dir" ## set destination dir for result.h5
mkdir -p "$destdir" || { ## create with all parent dirs or exit
printf "unable to create directory %s\n" "$dir" >&2
exit 1
}
ls *.pc 2>/dev/null || exit 1 ## check .pc file exists or exit
file7="${1##*/}" ## trim path from file.7 name
pc_script -f "$file7" -flags1 -other_flags ## first run
## check result.h5 exists and non-empty and copy to destdir
[ -s "result.h5" ] && cp -a "result.h5" "$destdir/new_result1.h5"
pc_script -f "$file7" -flags2 -other_flags ## second run
## check result.h5 exists and non-empty and copy to destdir
[ -s "result.h5" ] && cp -a "result.h5" "$destdir/new_result2.h5"
Which essentially stores the path part of the file.7 argument in dir and changes to that directory. If unable to change to the directory (due to read-permissions, etc..) the error is handled and the script exits. Next the full directory structure is created below your Result_Folder with mkdir -p with the same error handling if the directory cannot be created.
ls is used as a simple check to verify that a file ending in ".pc" exits in that directory. There are other ways to do this by piping the results to wc -l, but that spawns additional subshells that are best avoided.
(also note that Linux and Mac have files ending in ".pc" for use by pkg-config used when building programs from source -- they should not conflict with your files -- but be aware they exists in case you start chasing why weird ".pc" files are found)
After all tests are performed, the path is trimmed from the current ".7" filename storing just the filename in file7. The file7 variabli is then used in your pc_script command (which should also include the full path to the script if not in you PATH). After the pc_script is run [ -s "result.h5" ] is used to verify that result.h5 exists and is non-empty before moving that file to your Result_Folder location.
That should get you started. Using find to locate all .7 files is a simple way to let the tool designed to find the files for you do its job -- rather than trying to hand-roll your own solution. That way you only have to concentrate on what should be done for each file found. (note: I don't have pc_script or the files, so I have not testes this end-to-end, but it should be very close if not right-on-the-money)
There is nothing wrong in writing your own routine, but using find eliminates a lot of area where bugs can hide in your own solution.
Let me know if you have further questions.
How can I pass each one of my repository files and to do something with them?
For instance, I want to make a script:
#!/bin/bash
cd /myself
#for-loop that will select one by one all the files in /myself
#for each X file I will do this:
tar -cvfz X.tar.gz /myself2
So a for loop in bash is similar to python's model (or maybe the other way around?).
The model goes "for instance in list":
for some_instance in "${MY_ARRAY[#]}"; do
echo "doing something with $some_instance"
done
To get a list of files in a directory, the quick and dirty way is to parse the output of ls and slurp it into an array, a-la array=($(ls))
To quick explain what's going on here to the best of my knowledge, assigning a variable to a space-delimited string surrounded with parens splits the string and turns it into a list.
Downside of parsing ls is that it doesn't take into account files with spaces in their names. For that, I'll leave you with a link to turning a directory's contents into an array, the same place I lovingly :) ripped off the original array=($(ls -d */)) command.
you can use while loop, as it will take care of whole lines that include spaces as well:
#!/bin/bash
cd /myself
ls|while read f
do
tar -cvfz "$f.tar.gz" "$f"
done
you can try this way also.
for i in $(ls /myself/*)
do
tar -cvfz $f.tar.gz /myfile2
done
In my configure.ac file, I have this:
AC_CONFIG_FILES([Makefile foo/bar.h],
[mkdir -p ../dir1 && cp foo/bar.h ../dir1]
)
The goal is to:
Generate the Makefile from Makefile.in
Generate foo/bar.h from foo/bar.h.in
Copy foo/bar.h to dir1/bar.h.
While it works, I'm pretty sure I've done that last part wrong. Looking at the generated output, I see:
case $ac_file$ac_mode in
"Makefile":F) mkdir -p ../dir1 && cp foo/bar.h ../dir1
;;
"foo/bar.h":F) mkdir -p ../dir1 && cp foo/bar.h ../dir1
;;
esac
So it looks like it is doing my 'mkdir' command once for each file in the file list which is a bit redundant. The fact that it does this in a 'case' statement suggests there is some way to specify commands to run specific to each file (otherwise why have a 'case'?).
What's the trick?
In case you don't need to keep bar.h in the dir foo, I suppose you can do
AC_CONFIG_FILES([Makefile bar.h:foo/bar.h.in])
source: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Configuration-Files.html#Configuration-Files
In case bar.h needs to be in both locations and a link is acceptable, use
AC_CONFIG_FILES([Makefile foo/bar.h])
AC_CONFIG_LINKS([bar.h:foo.bar.h])
source: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Configuration-Links.html#Configuration-Links
So it turns out I was misinterpreting what I was seeing.
Seeing that 'case' statement made me think there was some way to specify that some of the cmds applied to specific entries from the files parameter (something like [makefile:stuff1 foo/bar.h:stuff2]). But (thankfully) that's not why the 'case' is there.
The trick here is that at the point where the case statement is produced, we aren't just walking the entries from AC_CONFIG_FILES files. So in order to limit the cmds to the just AC_CONFIG_FILES entries, it uses a 'case.'
While my early attempts suggested that using two AC_CONFIG_FILES was not supported, that was wrong too.
My solution was to use 2 AC_CONFIG_FILES.
I want to store output of ls command in my bash script in a variable and use each file name in a loop, but for example one file in the directory has name "Hello world", when I do variable=$(ls) "Hello" and "world" end up as two separate entries, and when I try to do
for i in $variable
do
mv $i ~
done
it shows error that files "Hello" and "world" doesn't exist.
Is there any way I can access all files in current directory and run some command even if the files have space(s) in their names.
If you must, dirfiles=(/path/of/interest/*).
And accept the admonition against parsing the output of ls!
I understand you are new to this and I'd like to help. But it isn't easy for me (us?) to provide you with an answer that would be of much help to you by the way you've stated your question.
Based on what I hear so far, you don't seem to have a basic understanding on how parameter expansions work in the shell. The following two links will be useful to you:
Matching Pathnames, Parameters
Now, if your task at hand is to operate on files meeting certain criteria then find(1) will likely to do the job.
Say it with me: don't parse the output of ls! For more information, see this post on Unix.SE.
A better way of doing this is:
for i in *
do
mv -- "$i" ~
done
or simply
mv -- * ~