How to copy multiple files with varying version numbers from one directory to another using bash? - linux

I have a folder /home/user/Document/filepath where I have three files namely file1-1.1.0.txt, file2-1.1.1.txt, file3-1.1.2.txt
and another folder named /home/user/Document/backuppath where I have to move files from /home/user/Document/folderpath which has file1-1.0.0.txt, file2-1.0.1.txt and file3-1.0.2.txt
task is to copy the specific files from folder path to backup path.
To summarize:
the below is the files.txt where I listed the files which has to be copied:
file1-*.txt
file2-*.txt
The below is the move.sh script that execute the movements
for file in `cat files.txt`; do cp "/home/user/Document/folderpath/$file" "/home/user/Documents/backuppath/" ; done
for the above script I am getting the error like
cp: cannot stat '/home/user/Document/folderpath/file1-*.txt': No such file or directory found
cp: cannot stat '/home/user/Document/folderpath/file2-*.txt': No such file or directory found
what I would like to accomplish is that I would like to use the script to copy specific files using * in the place of version numbers., since the version number may vary in the future.

You have wildcard characters in your files.txt. In your cp command, you are using quotes. These quotes prevent the wildcards to be expanded, as you can clearly see from the error message.
One obvious possibility is to not use quotes:
cp /home/user/Document/folderpath/$file /home/user/Documents/backuppath/
Or not use a loop at all:
cp $(<files.txt) /home/user/Documents/backuppath/
However, this would of course break if one line in your files.txt is a filename pattern which contains white spaces. Therefore, I would recommend a second loop over the expanded pattern:
while read file # Puts the next line into 'file'
do
for f in $file # This expands the pattern in 'file'
do
cp "/home/user/Document/folderpath/$f" /home/user/Documents/backuppath
done
done < files.txt

Related

Remove the last X files of a directory

In colab, I have unzip a file, but now there is too much files in the directory according to the colab's setup. Is there a command line to remove the last x files of a directory?
I know I can remove all the files from this repository with rm -rf *, but I just want to remove for instance the last 100 files of the repository.
Try globing or better REGEX.
The most easy way is with globing you use the star * and some differentiation example: rm *.txt # will delete all files that end with .txt or rm document*.local # will delete all files which start with document and end with .local
The better wey is searching files by attribut and executing command on the result but is a bit complex to explain so check this out.
https://www.cyberciti.biz/faq/linux-unix-how-to-find-and-remove-files/
Using a shell array and parameter expansion:
all_files=(*)
printf '%s\n' "${all_files[#]: -100}" | nl
#rm "${all_files[#]: -100}"
Uncomment the last line if it looks like the correct list of files to delete.
The space between the colon and the minus sign is required to disambiguate from another form of parameter expansion.
Ref: 3.5.3 Shell Parameter Expansion

How to specify the tar final structure

I have this structure:
release/folder1/file1
release/folder2/file2
...
release/folderN/fileN
I want to include all those folders (folder1, folder2 ... folderN) in a tar file.
The key is that I want these folders to be in the final tar within another directory named MYAPP so when you open the tar you can see this:
MYAPP/folder1/file1
MYAPP/folder2/file2
...
MYAPP/folderN/fileN
How can I achieve this without renaming the original "release" directory and/or creating new directories.
Is this possible to achive just in the tar process?
Thanks
Add
--transform=s#^release/#MYAPP/#
to your tar command line.
The argument of the --transform command line is a command that is passed to sed together with the file path before it is stored in the archive (use tar -tf to show the names of the files stored in the archive).
The command s#^release/#MYAPP/# tells sed to search (s) release/ at the beginning of the string (^) and replace it with MYAPP/.
The / at the end of the search and replace strings is needed to be sure the complete name of the component is release (to not replace release.txt). The # character is just a regex delimiter. Usually / is used as a regex delimiter but we prefer to use a different delimiter here to avoid the need to escape / (because it is used in the search and replace strings).
Read more in the documentation of tar and sed.

Linux - Recursively list all the zip files and keep only latest modified 5 files and delete the remaining

In command line, How can we recursively find out all the zip files in a directory and its sub directories and keep only the latest modified 5 files and delete the remaining.
The files paths would be something like below:
basedirectory/2015/12/18/abc.zip
basedirectory/2015/12/18/def.zip
basedirectory/2015/12/18/ghi.zip
basedirectory/2015/12/18/jkl.zip
basedirectory/2015/12/08/mno.zip
basedirectory/2015/12/08/pqr.zip
basedirectory/2015/12/08/stu.zip
basedirectory/2015/12/07/stu.zip
I have a way, but it involves several (easy) steps. There are probably more elegant ways of doing this, but here is how I know how. They come from a couple sources, which I list at the end of my answer. You will use the already installed utilites cd, find, ls, rm and head. it will involve a creating and executing two bash scripts.
Open a terminal and change into your base directory with cd ~/basedirectory
This sets up the following commands. It is important that you stay in this directory for the rest of the commands.
Type findpwd-name *.zip > find_zip
This creates a list of all the zip files with the full path relative to the directory you changed in to. Instead of printing them to the screen, it writes them to a find_zip file in the directory you changed into.
type cp find_zip remove_old_zip
This creates a second, duplicate file that you will later use to delete the old files.
Open the find_zip file in your favorite text editor. If you're not used to using any, you can use gedit. If you don't have it, install it with sudo apt-get udpate && sudo apt-get install gedit
Do a search and replace as follows (in gedit): search for \n , and replace it with " \\n"
This places the list of folders within quotes. the first backslash places a "\" at the end of each line, which means continue reading the next line and execute all the code together. The \n preserves the line endings. The last " puts a quote at the beginning of each line. You need the quotes to escape special characters like ' and ( that may be in your file name.
Create 2 new lines at the top of the file and type:
!/bin/bash
ls -lt \
The first line turns your file into a bash script. The second line will list all the files you found with the find command and order them by date.
Create a new line at the bottom of your file and type: | head -5. Save and exit the file.
| is a "pipe" that will take the output of the ordered file list that ls creates and feed it into the head command. The head command will list just the 5 most recently modified files and display or print them on your screen.
As a result of steps 5-7, your file should go from looking like this:
basedirectory/2015/12/18/abc.zip
basedirectory/2015/12/18/def.zip
basedirectory/2015/12/18/ghi.zip
basedirectory/2015/12/18/jkl.zip
basedirectory/2015/12/08/mno.zip
basedirectory/2015/12/08/pqr.zip
basedirectory/2015/12/08/stu.zip
basedirectory/2015/12/07/stu.zip
to this:
#!/bin/bash
ls -lt \
basedirectory/2015/12/18/abc.zip \
basedirectory/2015/12/18/def.zip \
basedirectory/2015/12/18/ghi.zip \
basedirectory/2015/12/18/jkl.zip \
basedirectory/2015/12/08/mno.zip \
basedirectory/2015/12/08/pqr.zip \
basedirectory/2015/12/08/stu.zip \
basedirectory/2015/12/07/stu.zip \
| head -5
Type bash find_zip into in the terminal. With your newfound list of the 5 most recent files, open up the remove_old_zip file created in step 3.
You will also be turning this file into a bash script, but it will remove all but the five newest files.
Delete the lines in the remove_old_zip file containing the 5 files you want to keep.
Do a search and replace as follows (in gedit): search for \n , and replace it with " \\n"
This is the same as step 5.
Create 2 new lines at the top of the file and type:
!/bin/bash
rm \
This is similar to step 6 except that rm will delete the files still listed.
remove the final \ on the final line of the remove_old_zip file. Save and exit.
Type bash remove_old_zip.
Type rm find_zip remove_old_zip.
This remove the two scripts, which are now useless since the files have been deleted.
sources:
How can I list (ls) the 5 last modified files in a directory?
http://www.geekinterview.com/talk/758-how-to-continue-to-next-line.html
List files recursively in Linux CLI with path relative to the current directory

How to move and number files?

I working with linux, bash.
I have one directory with 100 folders in it, each one named different.
In each of these 100 folders, there is a file called first.bars (so I have 100 files named first.bars). Although all named first.bars, the files are actually slightly different.
I want to get all these files moved to one new folder and rename/number these files so that I know which file comes from which folder. So the first first.bars file must be renamed to 001.bars, the second to 002.bars.. etc.
I have tried the following:
ls -d * >> /home/directorywiththe100folders/list.txt
cat list.txt | while read line;
do cd $line;
mv first.bars /home/newfolder
This does not work because I can't have 100 files, named the same, in one folder. So I only need to know how to rename them. The renaming must be connected to the cat list.txt, because the first line is the folder containing the first file wich is moved and renamed. That file will be called 001.bars.
Try doing this :
$ rename 's/^.*?\./sprintf("%03d.", $c++)/e' *.bar
If you want more information about this command, see this recent response I gave earlier : How do I rename multiple files beginning with a Unix timestamp - imapsync issue
If the rename command is not available,
for d in /home/directorywiththe100folders/*/; do
newfile=$(printf "/home/newfolder/%d.bars" $(( c++ )) )
mv "$d/first.bars" "$newfile"
done

Execute program on Files in subDirectory

I have following architecture of files in a directory.
Directory
/A/abc.xyz
/B/abc.xyz
/C/abc.xyz
/D/abc.xyz
/E/abc.xyz
I want to execute a program on acb.xyz in each SubDirectory. Save Output files in different directory i.e. Directory/processed with the name of SubDirectory appended in the name of output files.
Can it be written in following way? Need corrections.
for i in `ls "Directory/"`
do
program.pl $i/abc.xyz > processed/$i-abc.xyz
done
for dir in Directory/*; do
program.pl "$dir/abc.xyz" > "processed/${dir##*/}-abc.xyz"
done
The ${dir##*/} part strips the leading directory names from $dir, so Directory/A becomes just A. I added quotes to ensure directory names with whitespace don't cause issue (a good habit, even if you know there are no spaces).
As an alternative to the string munging you could simplify this if you first change directory:
cd Directory
for dir in *; do
program.pl "$dir/abc.xyz" > "processed/$dir-abc.xyz"
done

Resources