Bash Script to replicate files - linux

I have 25 files in a directory. I need to amass 25000 files for testing purposes. I thought I could just replicate these files over and over until I get 25000 files. I could manually copy paste 1000 times but that seemed tedious. So I thought I could write a script to do it for me. I tried
cp * .
As a trial but I got an error that said the source and destination file are the same. If I were to automate it how would i do it so that each of the 1000 times the new files are made with unique names?

As discussed in the comments, you can do something like this:
for file in *
do
filename="${file%.*}" # get everything up to last dot
extension="${file##*.}" # get extension (text after last dot)
for i in {00001..10000}
do
cp $file ${filename}${i}${extension}
done
done
The trick for i in {00001..10000} is used to loop from 1 to 10000 having the number with leading zeros.
The ${filename}${i}${extension} is the same as $filename$i$extension but makes more clarity over what is a variable name and what is text. This way, you can also do ${filename}_${i}${extension} to get files like a_23.txt, etc.
In case your current files match a specific pattern, you can always do for file in a* (if they all are on the a + something format).

If you want to keep the extension of the files, you can use this. Assuming, you want to copy all txt-files:
#!/bin/bash
for f in *.txt
do
for i in {1..10000}
do
cp "$f" "${f%.*}_${i}.${f##*.}"
done
done

You could try this:
for file in *; do for i in {1..1000}; do cp $file $file-$i; done; done;
It will append a number to any existing files.

The next script
for file in *.*
do
eval $(sed 's/\(.*\)\.\([^\.]*\)$/base="\1";ext="\2";/' <<< "$file")
for n in {1..1000}
do
echo cp "$file" "$base-$n.$ext"
done
done
will:
take all files with extensions *.*
creates the basename and extension (sed)
in a cycle 1000 times copyes the original file to file-number.extension
it is for DRY-RUN, remove the echo if satisfied

Related

Batch copy and rename multiple files in the same directory

I have 20 files like:
01a_AAA_qwe.sh
01b_AAA_asd.sh
01c_AAA_zxc.sh
01d_AAA_rty.sh
...
Files have a similar format in their names. They begin with 01, and they have 01*AAA*.sh format.
I wish to copy and rename files in the same directory, changing the number 01 to 02, 03, 04, and 05:
02a_AAA_qwe.sh
02b_AAA_asd.sh
02c_AAA_zxc.sh
02d_AAA_rty.sh
...
03a_AAA_qwe.sh
03b_AAA_asd.sh
03c_AAA_zxc.sh
03d_AAA_rty.sh
...
04a_AAA_qwe.sh
04b_AAA_asd.sh
04c_AAA_zxc.sh
04d_AAA_rty.sh
...
05a_AAA_qwe.sh
05b_AAA_asd.sh
05c_AAA_zxc.sh
05d_AAA_rty.sh
...
I wish to copy 20 of 01*.sh files to 02*.sh, 03*.sh, and 04*.sh. This will make the total number of files to 100 in the folder.
I'm really not sure how can I achieve this. I was trying to use for loop in the bash script. But not even sure what should I need to select as a for loop index.
for i in {1..4}; do
cp 0${i}*.sh 0${i+1}*.sh
done
does not work.
There are going to be a lot of ways to slice-n-dice this one ...
One idea using a for loop, printf + brace expansion, and xargs:
for f in 01*.sh
do
printf "%s\n" {02..05} | xargs -r -I PFX cp ${f} PFX${f:2}
done
The same thing but saving the printf in a variable up front:
printf -v prefixes "%s\n" {02..05}
for f in 01*.sh
do
<<< "${prefixes}" xargs -r -I PFX cp ${f} PFX${f:2}
done
Another idea using a pair of for loops:
for f in 01*.sh
do
for i in {02..05}
do
cp "${f}" "${i}${f:2}"
done
done
Starting with:
$ ls -1 0*.sh
01a_AAA_qwe.sh
01b_AAA_asd.sh
01c_AAA_zxc.sh
01d_AAA_rty.sh
All of the proposed code snippets leave us with:
$ ls -1 0*.sh
01a_AAA_qwe.sh
01b_AAA_asd.sh
01c_AAA_zxc.sh
01d_AAA_rty.sh
02a_AAA_qwe.sh
02b_AAA_asd.sh
02c_AAA_zxc.sh
02d_AAA_rty.sh
03a_AAA_qwe.sh
03b_AAA_asd.sh
03c_AAA_zxc.sh
03d_AAA_rty.sh
04a_AAA_qwe.sh
04b_AAA_asd.sh
04c_AAA_zxc.sh
04d_AAA_rty.sh
05a_AAA_qwe.sh
05b_AAA_asd.sh
05c_AAA_zxc.sh
05d_AAA_rty.sh
NOTE: blank lines added for readability
You can't do multiple copies in a single cp command, except when copying a bunch of files to a single target directory. cp will not do the name mapping automatically. Wildcards are expanded by the shell, they're not seen by the commands themselves, so it's not possible for them to do pattern matching like this.
To add 1 to a variable, use $((i+1)).
You can use the shell substring expansion operator to get the part of the filename after the first two characters.
for i in {1..4}; do
for file in 0${i}*.sh; do
fileend=${file:2}
cp "$file" "0$((i+1))$fileend"
done
done

How to rename multiple files in linux and store the old file names with the new file name in a text file?

I am a novice Linux user. I have 892 .pdb files, I want to rename all of them in a sequential order as L1,L2,L3,L4...........,L892. And then I want a text file which contains the old names assigned to new names ( i.e L1,L2,L3). Please help me with this. Thank you for your time.
You could just do:
#!/bin/sh
i=0
for f in *.pdb; do
: $((i += 1))
mv "$f" L"$i" && echo "$f --> L$i"
done > filelist
Note that you probably want to move the files into a different directory, as that will make it easier to recover if an error occurs midway through. Also be wary that this will overwrite any existing files and potentially cause a big mess. It's not idempotent (you can't run it twice). You would probably be better off not doing the move at all and instead do something like:
#!/bin/sh
i=0
mkdir -p newfiles
for f in *.pdb; do
ln "$f" newfiles/L"$((++i))" && printf "%s\0%s\0" "$f" "L$i"
done > filelist
This latter solution creates links to the original files in a subdirectory, so you can run it multiple times without munging the original data. Also, it uses null separators in the file list so you can unambiguously distinguish names that have newlines or tabs or spaces in them. It makes for a list that is not particularly human readable, but you can easily filter it through tr to make it pretty.

Renaming multiple different file extensions with BASH script

I'm trying to create a bash script that takes a directory full of files (about 500 files) that have all different types of extensions (no seriously, like 30 different types of extensions) and I want get rid of all of the extensions, and replace them with .txt
I've been searching around for a while now, and can only find examples of taking a specified extension, and changing it to another specified extension.
Like png --> jpg, or .doc --> .txt
Here's an example I've found:
# Rename all *.txt to *.text
for f in *.txt; do
mv -- "$f" "${f%.txt}.text"
done
This works, but only if you go from .txt to .text, I have multiple different extensions I'm working with.
My current code is:
directory=$1
for item in $directory/*
do
echo mv -- "$item" "$item.txt";
done
This will append the .txt onto them, but unfortunately I am left with the previous ones still attached. E.G. filename.etc.txt, filename.bla.txt
Am I going about this wrong? Any help is greatly appreciated.
It's a trivial change to the first example:
cd "$directory"
# Rename all files to *.txt
for f in *
do
mv -- "$f" "${f%.*}.txt"
done
If a file contains multiple extensions, this will replace only the last one. To remove all extensions, use %% in place of %.

Splitting a large directory into smaller ones in Linux

I have a large directory named as application_pdf which contains 93k files. My use-case is to split the directory into 3 smaller subdirectories (to a different location that the original large directory) containing around 30k files each.
Can this be done directly from the commandline.
Thanks!
Using bash:
x=("path/to/dir1" "path/to/dir2" "path/to/dir3")
c=0
for f in *
do
mv "$f" "${x[c]}"
c=$(( (c+1)%3 ))
done
If you have the rename command from Perl, you could try it like this:
rename --dry-run -pe 'my #d=("dirA","dirB","dirC"); $_=$d[$N%3] . "/$_"' *.pdf
In case you are not that familiar with the syntax:
-p says to create output directories, à la mkdir -p
-e says to execute the following Perl snippet
$d[$N%3] selects one of the directories in array #d as a function of the serially incremented counter $N provided to the snippet by rename
The output value is passed back to rename by setting $_
Remove the --dry-run if it looks good. Please run on a small directory with a copy of 8-10 files first, and make a backup before trying on all your 93k files.
Test
touch {0,1,2,3,4,5,6}.pdf
rename --dry-run -pe 'my #d=("dirA","dirB","dirC"); $_=$d[$N%3] . "/$_"' *.pdf
'0.pdf' would be renamed to 'dirB/0.pdf'
'1.pdf' would be renamed to 'dirC/1.pdf'
'2.pdf' would be renamed to 'dirA/2.pdf'
'3.pdf' would be renamed to 'dirB/3.pdf'
'4.pdf' would be renamed to 'dirC/4.pdf'
'5.pdf' would be renamed to 'dirA/5.pdf'
'6.pdf' would be renamed to 'dirB/6.pdf'
More for my own reference, but if you don't have the Perl rename command, you could do it just in Perl:
perl -e 'use File::Copy qw(move);my #d=("dirA","dirB","dirC"); my $N=0; #files = glob("*.pdf"); foreach $f (#files){my $t=$d[$N++%3] . "/$f"; print "Moving $f to $t\n"; move $f,$t}'
Something like this might work:
for x in $(ls -1 originPath/*.pdf | head -30000); do
mv originPath/$x destinationPath/
done

Copy text from multiple files, same names to different path in bash (linux)

I need help copying content from various files to others (same name and format, different path).
For example, $HOME/initial/baby.desktop has text which I need to write into $HOME/scripts/baby.desktop. This is very simple for a single file, but I have 2500 files in $HOME/initial/ and the same number in $HOME/scripts/ with corresponding names (same names and format). I want append (copy) the content of file in path A to path B (which have the same name and format), to the end of file in path B without erase the content of file in path B.
Example content of $HOME/initial/*.desktop to final $HOME/scripts/*.desktop. I tried the following, but it don't work:
cd $HOME/initial/
for i in $( ls *.desktop ); do egrep "Icon" $i >> $HOME/scripts/$i; done
Firstly, I would backup $HOME/initial and $HOME/scripts, because there is lots of scope for people misunderstanding your question. Like this:
cd $HOME
tar -cvf initial.tar initial
tar -cvf scripts.tar scripts
That will put all the files in $HOME/initial into a single tarfile called initial.tar and all the files in $HOME/scripts into a single tarfile called scripts.tar.
Now for your question... in general, if you want to put the contents of FileB onto the end of FileA, the command is
cat FileB >> FileA
Note the DOUBLE ">>" which means "append" rather than single ">" which means overwrite.
So, I think you want to do this:
cd $HOME/initial/baby.desktop
cat SomeFile >> $HOME/scripts/baby.desktop/SomeFile
where SomeFile is the name of any file you choose to test with. I would test that has worked and then, if you are happy with that, go ahead and run the same command inside a loop:
cd $HOME/initial/baby.desktop
for SOURCE in *
do
DESTINATION="$HOME/scripts/baby.desktop/$SOURCE"
echo Appending "$SOURCE" to "$DESTINATION"
#cat "$SOURCE" >> "$DESTINATION"
done
When the output looks correct, remove the "#" at the start of the penultimate line and run it again.
I solved it, if some people want learn how to resolve is very simple:
using Sed
I need only the match (or pattern) line "Icon=/usr/share/some_picture.png into $HOME/initial/example.desktop to other with same name and format $HOME/scripts/example.desktop, but I had a lot of .desktop files (2500 files)
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do sed -ne '/Icon=/ p' $i >> $HOME/scripts/$i ; done
_________
If you need only copy all to other file with same name and format
using cat
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do cat $i >> $HOME/scripts/$i ; done

Resources