The requirement is to make copies of n(n>10000) files in the same linux directory.
The extension of the files has to be intact and can probably add numbers to distinguish among files.
For e.g. if one file is text1.txt the other could be text2.txt
But I have to create multiple copies from multiple files and not from a single file.
Please help.
Bash pattern substitution might help you here. If you e.g. want to copy all .txt files, you can do it like this:
for file in *.txt # add any other name wildcards
do
filename=${file%.*} # removes everything after the last dot
extension=${file##*.} # removes everything before the last dot
cp "$file" "${filename}-copy.${extension}" # adds the -copy suffix to every copy
done
You might want to look into tools like logrotate which can take for example a glob and rotate each of the files on a regular basis.
Related
Here, (on linux)
there is an existing archive named A.zip, which include File1 and File2:
A.zip:
File1
File2
and I run this command: zip A.zip File1 File3, then the archive A.zip becomes like:
A.zip:
File1
File2
File3
however, what I really want to get is a brand new archive A.zip! like:
A.zip:
File1
File3
I know it can be done by run rm A.zip and then run zip A.zip File1 File3, but it is not elegant and if I write these commands into a shell script so A.zip may not exist while the action to remove a non-existent file is not elegant as well.
Is there any options for me to get this done?
Use these option to works:
zip -FSr A.zip File1 File3
OPTIONS
-FS
Synchronize the contents of an archive with the files on the OS. Normally when an archive is updated, new files are added and changed files are updated but files that no longer exist on the
OS are not deleted from the archive. This option enables a new mode that checks entries in the archive against the file system. If the file time and file size of the entry matches that of
the OS file, the entry is copied from the old archive instead of being read from the file system and compressed. If the OS file has changed, the entry is read and compressed as usual. If
the entry in the archive does not match a file on the OS, the entry is deleted. Enabling this option should create archives that are the same as new archives, but since existing entries are
copied instead of compressed, updating an existing archive with -FS can be much faster than creating a new archive. Also consider using -u for updating an archive.
For this option to work, the archive should be updated from the same directory it was created in so the relative paths match. If few files are being copied from the old archive, it may be
faster to create a new archive instead.
Note that the timezone environment variable TZ should be set according to the local timezone in order for this option to work correctly. A change in timezone since the original archive was
created could result in no times matching and recompression of all files.
This option deletes files from the archive. If you need to preserve the original archive, make a copy of the archive first or use the --out option to output the updated archive to a new
file. Even though it may be slower, creating a new archive with a new archive name is safer, avoids mismatches between archive and OS paths, and is preferred.
-r
Travel the directory structure recursively; for example:
zip -r foo.zip foo
or more concisely
zip -r foo foo
In this case, all the files and directories in foo are saved in a zip archive named foo.zip, including files with names starting with ".", since the recursion does not use the shell's file-
name substitution mechanism. If you wish to include only a specific subset of the files in directory foo and its subdirectories, use the -i option to specify the pattern of files to be in‐
cluded. You should not use -r with the name ".*", since that matches ".." which will attempt to zip up the parent directory (probably not what was intended).
Multiple source directories are allowed as in
zip -r foo foo1 foo2
which first zips up foo1 and then foo2, going down each directory.
Note that while wildcards to -r are typically resolved while recursing down directories in the file system, any -R, -x, and -i wildcards are applied to internal archive pathnames once the di‐
rectories are scanned. To have wildcards apply to files in subdirectories when recursing on Unix and similar systems where the shell does wildcard substitution, either escape all wildcards
or put all arguments with wildcards in quotes. This lets zip see the wildcards and match files in subdirectories using them as it recurses.
I'm trying to zip all the files within a directory which contains .py files individually. But after zipping the files the output that I'm seeing is .py.zip vs just .zip
Here's the one liner command that I'm trying to execute.
cd scripts/python/
for i in *; do zip $i.zip $i; done
This is what you are looking for:
for i in *py; do
zip "${i%.*}".zip "$i";
done
Explanation
${i%.*}: This makes use of Bash's built in parameter expansion. Here it tries to match everything after %. If it does find a match, it uses everything before the match. https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion for more information.
I am looking to write a script to move files from a directory:
/home/mydir/
To another directory based on tokens in the file name. I have a bunch of files named as such:
red_office_mike_2015_montreal_546968.ext
or
$color_$location_$name_$year_$city_$numbers.extension (files will be various movie files: mov, mp4, mkv, etc.)
I would like the script to move the files to the following location:
/dir/work/$color/$name
Then verify the file has successfully copied, and delete the original file once it has.
I would also love it if the script would create the to directory if it does not already exist.
So in summary, I need a script to move files based on underscore separated tokens, create the to directory if it doesn't already exist, verify the successful copy (maybe with a size check), then delete the original file.
I am working on linux, and would prefer a bash script. The variables I have given are generic, and I will incorporate some other things to the script, I'm just looking for help on building the skeleton.
Thanks in advance for any help!
It's not a bash script, but perl is much better at this kind of thing and is installed on all Linux systems
while(<>) {
chomp;
$file = $_;
($colour, $location, $name, $year, $city, $numbers) = split(/_/,$file);
$dest0 = "/dir/work/$colour";
$dest1 = "$dest0/$name";
mkdir ($dest0) unless (-d $dest0);
mkdir ($dest1) unless (-d $dest1);
rename ($file, "$dest1/$file");
}
The script splits your input file on the underscore character, creates all the directories to the destination and then renames the file to the new filename. Rename takes care of all the copying and deleting for you. In fact it just changes the directory entries without any copying at all.
UPDATE
The above version takes its input from a file containing a list of filenames to process. For an alternative version which processes all files in the current directory, replace the while line with
while(glob("*")) {
I was able to fumble around online and come up with a for loop to do this task. I used cut and it made things simple. Here is what worked for me:
#!/bin/sh
cd "${1:-.}"
for f in *.*; do
color=`echo "$f" | cut -d'_' -f1`
name=`echo "$f" | cut -d'_' -f3`
todir="/dir/work/$color/$name"
mkdir -p "$todir"
mv "$f" "$todir"
done
This worked perfectly and I hope it can help others who might need to create directories based on portions of filenames.
The first line under the shebang made it so that it will either look at the current working directory or a directory you pass it as an argument.
Thanks to those who chimed in on the original post. I'm new with scripting so it take me a while to figure this stuff out. I love this site though, it is super helpful!
I found this link with an example of how I can actually copy range of files https://serverfault.com/questions/370403/copy-a-range-of-files-in-command-line-zsh-bash, using this
$cp P10802[75-83].JPG ~/Images/.
Is there any way I can also copy range of folders or directory?
Maybe something like this $cp -r folder[001-999] ~/images./
Use the -R flag to recursive copy the directories. According to Can I use shell wildcards to select filenames ranging across double-digit numbers, you can use the syntax {start..end} to match a number range. Putting that together would give you:
cp -R folder{001..999} ~/images./
Yes, using the same logic. Globbing and expansion (which is what bash uses to generate the individual names out of these patterns) work on files as well as on directory names.
I have some different files with the same name and I want to copy all of them to the destination which has a flat structure (no directories, just files), is there any way to append some text onto one of the file names so that both can be copied.
Need to use rsync because there are some files that I need to exclude from the copy.
For example:
dir1/file1.txt
dir1/dir2/file1.txt
both get copied, and in the destination there is:
file1.txt
file1.txt.txt
typically, when I want to do some complex name-mungling, I just write the list of files (with find dir1 >listfiles) and fix it with a text editor.
for example, s/^.*\/([^\/]+)$/cp \0 destination/\1/ converts a file like
dir1/file1.txt
dir1/dir2/file1.txt
to a script like:
cp dir1/file1.txt destination/file1.txt
cp dir1/dir2/file1.txt destination/file1.txt
then you could do something like cut -f 3 <listfiles | sort | uniq -d to find those with the same destination filename. then go back to the editor and fix those lines.
After a few minutes you get a full script for exactly the copy you want, without surprises because you can see each command and apply the best fix for each case.
As far as i know there is no default option in rsync to do that. But i guess that since you are copying files with the same name but from different directories, you are using
multiple rsync commands.
So, this gives you two options:
Create folders..
rsync -av /home/user1/file1 /media/foo/user1/file1
rsync -av /home/user2/file1 /media/foo/user2/file1
etc..
or rename the files with an id
rsync -av /home/user1/file1 /media/foo/parent_dir-file1
rsync -av /home/user2file1 /media/foo/parent_dir-file1
etc..
If you want to use the second solution you can build a simple script. As you are using rsync i suppose that you know the basics on GNU-Linux, so a simple bash script would be enough!
A basic ID is to get the parent folder name and add it as variable to the path of the rsync command. ( it won't always work )
IF you want to be sure of a good id you can for example set a counter and increment like
file1-1
file1-2
file1-3
But you will loose the track of its absolute path.
All the solutions can work, its up to you to choice the one that feed your needs!