Script to copy files, rename it and share it to azure blob - linux

There is a folder in which the application generates different files, such as those below.
Delhi_20221234556.csv
Melbourne_20123456789.csv
Goa_1234567890.csv
Sydney_2022345689.csv
The script should removes numbers from files as below, then copies them to Azure blob storage, and after copying them to Azure blob, it should move the file to the archive folder with the date.
Delhi.csv
Melbourne.csv
Goa.csv
Sydney.csv
Initially I was doing it for Goa_1234567890.csv as below.
#!/bin/bash
find /abc/def -type f -name 'Goa*' -execdir bash -c 'for f in "$#";
do mv "$f" "${f%%_*}.csv";
done' bash {} +
./azcopy cp "/abc/def/Goa.csv" "https://<Blobaccount>.blob.core.windows.net/abc\home\xyz?<SAS- Token>"
mv /abc/def/Goa.csv /abc/def/Archive/Goa$(date +%F).csv`
Would anyone be able to help me improve the above script to achieve this?

You may want to generalize your script to handle multiple filenames
other than Goa*. Assuming the csv files are located just under the
/abc/def directory (not in the subdirectories), would you please try:
#!/bin/bash
dir="/abc/def"
mkdir -p "$dir"/Archive/
for f in "$dir/"*.csv; do
if [[ -f $f ]]; then
new="${f%%_*}.csv"
mv -- "$f" "$new"
./azcopy cp "$new" "https://<Blobaccount>.blob.core.windows.net/abc\home\xyz?<SAS- Token>"
base="${new##*/}"
mv -- "$new" "$dir/Archive/${base%.csv}$(date +%F).csv"
fi
done
Please note it doesn't consider the case there are multiple
filenames which share the same city name with different numbers.

Related

Renaming files in subdirectories deletes the files

I am writing a script that renames *.MP4 files on an inserted SD card and then rsyncs them.
The directory with the *.MP4 files does not always have the same name:
eg: it could be /DCIM/123_PANA/ or /DCIM/141_PANA/ etc
So I'm trying to write a script that will see what folders are in the /DCIM path, and rename all the *.MP4 files, (there is also a MISC folder in this path which I suspect is causing the issue)
I am using a couple of variables to rename the files also
What I have is:
for f in /media/pi/LUMIX/DCIM/*; do
if [ -d "$f" ]; then
echo $f
for file in $(find $f -name 'P*.MP4')
do
echo $file ">" $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
mv $file $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
done
fi
done
But what seems to happen is I end up with a single file with the prefix only (say 08_nb1_) in the _PANA folder, all the others have been deleted. Obviously this is not my desired result!
UPDATE:
$cardname is of the format _nb2_
When I do as asked replace mv with echo here is the output:
/media/pi/LUMIX/DCIM/141_PANA
mv /media/pi/LUMIX/DCIM/141_PANA/P1410192.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410192.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410193.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410193.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410194.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410194.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410195.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410195.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410196.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410196.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410197.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410197.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410198.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410198.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410199.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410199.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410200.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410200.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410201.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410201.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410202.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410202.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410203.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410203.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410204.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410204.MP4
/media/pi/LUMIX/DCIM/MISC
OK I fixed it by filtering the additional directory names and limiting to only the ones with *_PANA - which solved the issue. I also added a the rsync part and demounted the SD card (if required using Zenity)
A text file placed on the SD card identifies it as a unique card giving each file a unique name when rsyncing it to the backup folder. Renaming on the SD card means
that it can still be used an written to if not full, but we then know which files have been backed up.
Very useful in the field when filming with multiple cards, crews. All running on a Rpi4
for f in /media/pi/LUMIX/DCIM/*_PANA/; do
if [ -d "$f" ]; then
echo "$f"
for file in $(find $f -wholename '*_PANA/P*.MP4')
do
mv "$file" $(dirname "${file}")/$(date +"%d")"$cardname"$(basename "${file}")
done
rsync --stats -u --progress "$f"/*.MP4 /media/pi/VDRIVE/ | tee /home/pi/Documents/ytu/rsync.txt | zenity --icon-name="dialog-warning" \
--width=300 --progress --pulsate --auto-close --auto-kill \
--title="Copying $sdn"
zenity --question --text="Unmount Card?"
if [ $? = 0 ]; then
umount /media/pi/LUMIX
else
exit
fi
fi
done

Bash script to transfer all files from current directory to specific directory based on name

I have these files:
100-1.jpg
100-2.jpg
200-1.jpg
200-2.jpg
I want these to be transferred to specific folder based on filename
100/100-1.jpg
100/100-2.jpg
200/200-1.jpg
200/200-2.jpg
How do I do this?
What I have tried so far
cd ~/images
for f in *.jpg
do
mv -v "$f" ~/images/${f}/${f%}.jpg
done
how do I know I cut the string before the dash e.g 200-1 to 200 and store in a variable?
so I can do it like this
cd ~/images
for f in *.jpg
name="$f without the .jpg"
do
mv -v "$f" ~/images/${f}/${f%}.jpg
done
#!/bin/bash
cd ~/images
for f in *.jpg
do
mkdir -p ${f%-*}
echo ${f%-*}
mv "$f" ~/images/${f%-*}/${f%}
done

Rename all files in multiple folders with some condition in single linux command os script.

I have multiple folders with multiple files. I need to rename those files with the same name like the folder where the file stored with "_partN" prefix.
As example,
I have a folder named as "new_folder_for_upload" which have 2 files. I need to convert the name of these 2 files like,
new_folder_for_upload_part1
new_folder_for_upload_part2
I have so many folders like above which have multiple files. I need to convert all the file names as I describe above.
Can anybody help me to find out for a single linux command or script to do this work automatically?
Assuming bash shell, and assuming you want the file numbering to restart for each subdirectory, and doing the moving of all files to the top directory (leaving empty subdirectories). Formatted as script for easier reading:
find . -type f -print0 | while IFS= read -r -d '' file
do
myfile=$(echo $file | sed "s#./##")
mydir=$(dirname "$myfile")
if [[ $mydir != $lastdir ]]
then
NR=1
fi
lastdir=${mydir}
mv "$myfile" "$(dirname "$myfile")_part${NR}"
((NR++))
done
Or as one-line command:
find . -type f -print0 | while IFS= read -r -d '' file; do myfile=$(echo $file | sed "s#./##"); mydir=$(dirname "$myfile"); if [[ $mydir != $lastdir ]]; then NR=1; fi; lastdir=${mydir}; mv "$myfile" "$(dirname "$myfile")_part${NR}"; ((NR++)); done
Beware. This is armed, and will do a bulk renaming / moving of every file in or below your current work directory. Use at your own risk.
To delete the empty subdirs:
find . -depth -empty -type d -delete

using IF to see a directory exists if not do something

I am trying to move the directories from $DIR1 to $DIR2 if $DIR2 does not have the same directory name
if [[ ! $(ls -d /$DIR2/* | grep test) ]] is what I currently have.
then
mv $DIR1/test* /$DIR2
fi
first it gives
ls: cannot access //data/lims/PROCESSING/*: No such file or directory
when $DIR2 is empty
however, it still works.
secondly
when i run the shell script twice.
it doesn't let me move the directories with the similar name.
for example
in $DIR1 i have test-1 test-2 test-3
when it runs for the first time all three directories moves to $DIR2
after that i do mkdir test-4 at $DIR1 and run the script again..
it does not let me move the test-4 because my loop thinks that test-4 is already there since I am grabbing all test
how can I go around and move test-4 ?
Firstly, you can check whether or not a directory exists using bash's built in 'True if directory exists' expression:
test="/some/path/maybe"
if [ -d "$test" ]; then
echo "$test is a directory"
fi
However, you want to test if something is not a directory. You've shown in your code that you already know how to negate the expression:
test="/some/path/maybe"
if [ ! -d "$test" ]; then
echo "$test is NOT a directory"
fi
You also seem to be using ls to get a list of files. Perhaps you want to loop over them and do something if the files are not a directory?
dir="/some/path/maybe"
for test in $(ls $dir);
do
if [ ! -d $test ]; then
echo "$test is NOT a directory."
fi
done
A good place to look for bash stuff like this is Machtelt Garrels' guide. His page on the various expressions you can use in if statements helped me a lot.
Moving directories from a source to a destination if they don't already exist in the destination:
For the sake of readability I'm going to refer to your DIR1 and DIR2 as src and dest. First, let's declare them:
src="/place/dir1/"
dest="/place/dir2/"
Note the trailing slashes. We'll append the names of folders to these paths so the trailing slashes make that simpler. You also seem to be limiting the directories you want to move by whether or not they have the word test in their name:
filter="test"
So, let's first loop through the directories in source that pass the filter; if they don't exist in dest let's move them there:
for dir in $(ls -d $src | grep $filter); do
if [ ! -d "$dest$dir" ]; then
mv "$src$dir" "$dest"
fi
done
I hope that solves your issue. But be warned, #gniourf_gniourf posted a link in the comments that should be heeded!
If you need to mv some directories to another according to some pattern, than you can use find:
find . -type d -name "test*" -exec mv -t /tmp/target {} +
Details:
-type d - will search only for directories
-name "" - set search pattern
-exec - do something with find results
-t, --target-directory=DIRECTORY move all SOURCE arguments into DIRECTORY
There are many examples of exec or xargs usage.
And if you do not want to overwrite files, than add -n option to mv command:
find . -type d -name "test*" -exec mv -n -t /tmp/target {} +
-n, --no-clobber do not overwrite an existing file

moving files to different directories

I'm trying to move media and other files which are in a specified directory to another directory and create another one if it does not exits (where the files will go), and create a directory the remaining files with different extensions will go. My first problem is that my script is not making a new directory and it is not moving the files to other directories and what code can I use to move files with different extensions to one directory?
This is what i have had so far, correct me where I'm wrong and help modify my script:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo #directory where the media files will go
WA=/home/katy/do # directory where the other files will go
if [ ! -d "$To" ]; then
mkdir -p "$To"
fi
cd $From
find path -type f -name"*.mp4" -exec mv {} $To \;
I'd solve it somewhat like this:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo # directory where the media files will go
WA=/home/katy/do # directory where the other files will go
cd "$From"
find . -type f \
| while read file; do
dir="$(dirname "$file")"
base="$(basename "$file")"
if [[ "$file" =~ \.mp4$ ]]; then
target="$To"
else
target="$WA"
fi
mkdir -p "$target/$dir"
mv -i "$file" "$target/$dir/$base"
done
Notes:
mkdir -p will not complain if the directory already exists, so there's no need to check for that.
Put double quotes around all filenames in case they contain spaces.
By piping the output of find into a while loop, you also avoid getting bitten by spaces, because read will read until a newline.
You can modify the regex according to taste, e.g. \.(mp3|mp4|wma|ogg)$.
In case you didn't know, $(...) will run the given command and stick its output back in the place of the $(...) (called command substitution). It is almost the same as `...` but slightly better (details).
In order to test it, put echo in front of mv. (Note that quotes will disappear in the output.)
cd $From
find . -type f -name "*.mp4" -exec mv {} $To \;
^^^
or
find $From -type f -name "*.mp4" -exec mv {} $To \;
^^^^^
cd $From
mv *.mp4 $To;
mv * $WA;

Resources