I have many files created from a simulation.
Like this: res_00001.root through res_09999.root.
I would like to create a series of folders that move in batches of 1000 files in sequence to a newly created folder based on the filename we are moving. e.g. folder1 would contain res_00001.root through res_00999.root, folder2 res_01000.root through res_01999.root, ...
I attempted to create a script but it's not working:
#!/bin/bash
N_files=$1
for (( file=0; file<$N_files; ++file )) do #state what file I am looking at
s=file%1000 INPUT=printf data/output_%04lu.root $file` OUTPUT=printf data/folder%02lu/res_%04lu.root $s # move the files
mv INPUT OUTPUT
done`
I've been banging my head against this for sometime, I appreciate any help you can provide.
Updated Answer
You can run this little script if you can't find the rename program - make backup first!
#!/bin/bash
shopt -s nullglob nocaseglob
for f in *.root; do
n=$(tr -dc '[0-9]' <<< $f)
((d=(10#$n/1000)+1))
[ ! -d folder$d ] && mkdir folder$d
echo mv "$f" folder$d/$f
done
Original Answer
Make a backup and see if this helps you on a copy of a small subset of your files:
rename --dry-run 's/[^0-9]//g; my $d=int($_/1000)+1; $_="folder$d/res_$_.root"' *root
Sample Output
'res_00001.root' would be renamed to 'folder1/res_00001.root'
'res_00002.root' would be renamed to 'folder1/res_00002.root'
'res_00003.root' would be renamed to 'folder1/res_00003.root'
'res_00004.root' would be renamed to 'folder1/res_00004.root'
'res_00005.root' would be renamed to 'folder1/res_00005.root'
...
...
'res_00997.root' would be renamed to 'folder1/res_00997.root'
'res_00998.root' would be renamed to 'folder1/res_00998.root'
'res_00999.root' would be renamed to 'folder1/res_00999.root'
'res_01000.root' would be renamed to 'folder2/res_01000.root'
'res_01001.root' would be renamed to 'folder2/res_01001.root'
'res_01002.root' would be renamed to 'folder2/res_01002.root'
'res_01003.root' would be renamed to 'folder2/res_01003.root'
...
...
If it looks good, remove the --dry-run so it actually does stuff rather than just saying what stuff it would do!
s/[^0-9]//g gets rid of anything non-numeric in the filename
my $d=int($_/1000)+1 calculates the directory name
$_="folder$d/res_$_.root" builds the output filename
Related
in my working directory I have several folders which have hundreds of files of different types (.txt, .csv, .png...). These folders are structured as this example:
myDir/
myDir/Folder_1/ ... file_1.txt, file_2.csv, file_3.txt
myDir/Folder_2/ ...
myDir/Folder_3/ ...
...
I need to move all the .csv files to a new directory but keeping the same directory tree like this:
myDir/
myDir/Folder_1/ ...
myDir/Folder_2/ ...
...
myDir/New/
myDir/New/Folder_1/ ... file_2.csv
myDir/New/Folder_2/ ...
...
I found that cp command has --parents flag that does what I want to do, but I don't want to copy these files, I only want to move them.
I searched for this question before and found this one: Bash script for moving files and their parent directory
It is quite similar to what I need to do, but I'm not able to understand it.
Check the commands the following snippet prints out. If they do what you want, then just remove the echo (before mkdir and mv).
for path in myDir/*; do
[[ -d "$path" ]] && echo mkdir -p "myDir/New/${path#myDir/}"
done
for csv in myDir/*/*.csv; do
echo mv "$csv" "myDir/New/${csv#myDir/}"
done
I have a large directory named as application_pdf which contains 93k files. My use-case is to split the directory into 3 smaller subdirectories (to a different location that the original large directory) containing around 30k files each.
Can this be done directly from the commandline.
Thanks!
Using bash:
x=("path/to/dir1" "path/to/dir2" "path/to/dir3")
c=0
for f in *
do
mv "$f" "${x[c]}"
c=$(( (c+1)%3 ))
done
If you have the rename command from Perl, you could try it like this:
rename --dry-run -pe 'my #d=("dirA","dirB","dirC"); $_=$d[$N%3] . "/$_"' *.pdf
In case you are not that familiar with the syntax:
-p says to create output directories, à la mkdir -p
-e says to execute the following Perl snippet
$d[$N%3] selects one of the directories in array #d as a function of the serially incremented counter $N provided to the snippet by rename
The output value is passed back to rename by setting $_
Remove the --dry-run if it looks good. Please run on a small directory with a copy of 8-10 files first, and make a backup before trying on all your 93k files.
Test
touch {0,1,2,3,4,5,6}.pdf
rename --dry-run -pe 'my #d=("dirA","dirB","dirC"); $_=$d[$N%3] . "/$_"' *.pdf
'0.pdf' would be renamed to 'dirB/0.pdf'
'1.pdf' would be renamed to 'dirC/1.pdf'
'2.pdf' would be renamed to 'dirA/2.pdf'
'3.pdf' would be renamed to 'dirB/3.pdf'
'4.pdf' would be renamed to 'dirC/4.pdf'
'5.pdf' would be renamed to 'dirA/5.pdf'
'6.pdf' would be renamed to 'dirB/6.pdf'
More for my own reference, but if you don't have the Perl rename command, you could do it just in Perl:
perl -e 'use File::Copy qw(move);my #d=("dirA","dirB","dirC"); my $N=0; #files = glob("*.pdf"); foreach $f (#files){my $t=$d[$N++%3] . "/$f"; print "Moving $f to $t\n"; move $f,$t}'
Something like this might work:
for x in $(ls -1 originPath/*.pdf | head -30000); do
mv originPath/$x destinationPath/
done
This should be a relatively straightforward problem but I haven't found any answers within stackoverflow. In a given directory, I have ~1000 files that are numbered (e.g. chem-0320.inp). I would like to cp the numbered file to a correspondingly numbered directory; all copied files will be renamed with the same name. I would like to do this for a specified numbered of files (#'s 300-500 for example).
For example, I would like to copy chem-0320.inp to a directory named 320 and rename it mech.dat.
Another example: copy chem-0430.inp to a directory named 430 and rename it mech.dat.
Thanks in advance for your help!
The following script would do the work for you
for file in *.inp
do
dir=$(echo $file | sed -r 's/[^0-9]+0([0-9]+).*/\1/g')
mv $file $dir/mech.dat
done
"cd" first to right dir. Subdirs will be created there.
#!/bin/bash
lo_limit=300
hi_limit=500
for file in ./*.inp
do
dir="${file//[^0-9]/}"
dir_cut="${dir:1:3}" # leading zero cut off
if [ $dir_cut -ge $lo_limit ] && [ $dir_cut -le $hi_limit ]; then
echo "$file $dir_cut"
mkdir -p "$dir_cut"
cp "$file" "$dir_cut"/mech.dat
fi
done
I need help copying content from various files to others (same name and format, different path).
For example, $HOME/initial/baby.desktop has text which I need to write into $HOME/scripts/baby.desktop. This is very simple for a single file, but I have 2500 files in $HOME/initial/ and the same number in $HOME/scripts/ with corresponding names (same names and format). I want append (copy) the content of file in path A to path B (which have the same name and format), to the end of file in path B without erase the content of file in path B.
Example content of $HOME/initial/*.desktop to final $HOME/scripts/*.desktop. I tried the following, but it don't work:
cd $HOME/initial/
for i in $( ls *.desktop ); do egrep "Icon" $i >> $HOME/scripts/$i; done
Firstly, I would backup $HOME/initial and $HOME/scripts, because there is lots of scope for people misunderstanding your question. Like this:
cd $HOME
tar -cvf initial.tar initial
tar -cvf scripts.tar scripts
That will put all the files in $HOME/initial into a single tarfile called initial.tar and all the files in $HOME/scripts into a single tarfile called scripts.tar.
Now for your question... in general, if you want to put the contents of FileB onto the end of FileA, the command is
cat FileB >> FileA
Note the DOUBLE ">>" which means "append" rather than single ">" which means overwrite.
So, I think you want to do this:
cd $HOME/initial/baby.desktop
cat SomeFile >> $HOME/scripts/baby.desktop/SomeFile
where SomeFile is the name of any file you choose to test with. I would test that has worked and then, if you are happy with that, go ahead and run the same command inside a loop:
cd $HOME/initial/baby.desktop
for SOURCE in *
do
DESTINATION="$HOME/scripts/baby.desktop/$SOURCE"
echo Appending "$SOURCE" to "$DESTINATION"
#cat "$SOURCE" >> "$DESTINATION"
done
When the output looks correct, remove the "#" at the start of the penultimate line and run it again.
I solved it, if some people want learn how to resolve is very simple:
using Sed
I need only the match (or pattern) line "Icon=/usr/share/some_picture.png into $HOME/initial/example.desktop to other with same name and format $HOME/scripts/example.desktop, but I had a lot of .desktop files (2500 files)
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do sed -ne '/Icon=/ p' $i >> $HOME/scripts/$i ; done
_________
If you need only copy all to other file with same name and format
using cat
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do cat $i >> $HOME/scripts/$i ; done
I want to write a script that do specific thing:
I have a txt file e.g.
from1/from2/from3/apple.file;/to1/to2/to3;some not important stuff
from1/from2/banana.file;/to1/to5;some not important stuff
from1/from10/plum.file;/to1//to5/to100;some not important stuff
Now i want to copy file from each line (e.g. apple.file), from original directory tree to new, non existing directories, after first semicolon (;).
I try few code examples from similar questions, but nothing works fine and I'm too weak in bash scripting, to find errors.
Please help :)
need to add some conditions:
file not only need to be copy, but also rename. Example line in file.txt:
from1/from2/from3/apple.file;to1/to2/to3/juice.file;some1
from1/from2/banana.file;to1/to5/fresh.file;something different from above
so apple.file need to be copy and rename to juice.file and put in to1/to2/to3/juice.file
I think thaht cp will also rename file but
mkdir -p "$to"
from answer below will create full folder path with juice.file as folder
In addidtion after second semicolon in each line will be something different, so how to cut it off?
Thanks for all help
EDIT: There will be no spaces in input txt file.
Try this code..
cat file | while IFS=';' read from to some_not_important_stuff
do
to=${to:1} # strip off leading space
mkdir -p "$to" # create parent for 'to' if not existing yet
cp -i "$from" "$to" # option -i to get a warning when it would overwrite something
done
Using awk
(run the awk command first and confirm the output is fine, then add |sh to do the copy)
awk -F";" '{printf "cp %s %s\n",$1,$2}' file |sh
Using shell (get updated that need manually create folder, base on alfe's
while IFS=';' read from to X
do
mkdir -p $to
cp $from $to
done < file
I had this same problem and used tar to solve it! Posted here:
tmpfile=/tmp/myfile.tar
files="/some/folder/file1.txt /some/other/folder/file2.txt"
targetfolder=/home/you/somefolder
tar --file="$tmpfile" "$files"
tar --extract --file="$tmpfile" --directory="$targetfolder"
In this case, tar will automatically create all (sub)folders for you! Best,
Nabi