how to get the filename from the mentioned list - linux

I am new to linux and/or scripting, so bear with me. I want a script which can get the files for a Linux directory. here what I tried for getting the filename.
for NAME in $(ls -1 *.wav /some/path | cut -d "/" -f3 | cut -d "-" -f1-5)
if the filename contains -IN or -OUT then they will be sox -m and after that mv to another directory but if the some other files then it will be just mv
for the reference, filenames be like
1030-04-06-2015-1433414216.wav
1030-04-06-2015-1433414318.wav
1030-04-06-2015-1433414440.wav
1043-21-05-2015-1432207256.wav
1043-21-05-2015-1432207457.wav
1046-20-05-2015-1432137944.wav
1046-20-05-2015-1432138015.wav
1046-20-05-2015-1432138704.wav
1431709157.93900.0-in.wav
1431709157.93900.0-out.wav
1431709157.93900.1-in.wav
1431709157.93900.1-out.wav
1431710008.94059.0-in.wav
1431710008.94059.0-out.wav
1431710008.94059.1-in.wav
1431710008.94059.1-out.wav
1431710008.94059.1.wav
1431710008.94059.2.wav
1431713190.94698.2-in.wav
1431713190.94698.2-out.wav
1431713190.94698.2.wav
1431721107.96010.0-in.wav
1431721107.96010.0-out.wav
1431721107.96010.1.wav

This should work:
#!/bin/bash
regex='(.*)(-out|-in)(.txt)'
for file in *.txt;do
# $file is the full path to the file
if [[ $file =~ $regex ]];then
#filename in this block contains -in or -out
filename="${file##*\/}"
inorout="${BASH_REMATCH[2]}"
extension="${BASH_REMATCH[3]}"
filenamewithoutextension="${filename%.*}"
filenamewithoutioroutorextension="${filenamewithoutextension/%$inorout/}"
filenamewithoutiorout="${filenamewithoutioroutorextension}$extension"
echo "$filename" "$inorout" "$extension" "$filenamewithoutextension" "$filenamewithoutiorout" "$filenamewithoutioroutorextension"
#do something here sox-m mv or whatever
else
#filename in this block doesn't contain -in or -out
echo "do something else"
fi
done
Explanation:
"${file##*\/}" is the string left by cutting */ from $file i.e the basename (filename).
"${BASH_REMATCH[2]}" is the second captured group from the pattern matching done by [[ $file =~ $regex ]] i.e -in or -out
"${BASH_REMATCH[3]}" is the third captured group i.e .wav.
"${filename%.*}" is the string left by cutting .* from $file i.e filename without .wav
Resources you should check on:
Bash Parameter Expansion
Pattern Matching
Bash Variables

Here is one way to do it.
>cat test.sh
#!/bin/bash
destination="./DEST"
# Loop over every file
for file in "${#}" ; do
# If it is "-in", then sox
if [[ "${file}" =~ "-in" ]] || [[ "${file}" =~ "-out" ]] ; then
printf "sox -m ${file}; "
fi
echo "mv ${file} ${destination}"
done
When run, I get the following output.
>../test.sh *
mv 1030-04-06-2015-1433414216.wav ./DEST
mv 1030-04-06-2015-1433414318.wav ./DEST
mv 1030-04-06-2015-1433414440.wav ./DEST
mv 1043-21-05-2015-1432207256.wav ./DEST
mv 1043-21-05-2015-1432207457.wav ./DEST
mv 1046-20-05-2015-1432137944.wav ./DEST
mv 1046-20-05-2015-1432138015.wav ./DEST
mv 1046-20-05-2015-1432138704.wav ./DEST
sox -m 1431709157.93900.0-in.wav; mv 1431709157.93900.0-in.wav ./DEST
sox -m 1431709157.93900.0-out.wav; mv 1431709157.93900.0-out.wav ./DEST
sox -m 1431709157.93900.1-in.wav; mv 1431709157.93900.1-in.wav ./DEST
sox -m 1431709157.93900.1-out.wav; mv 1431709157.93900.1-out.wav ./DEST
sox -m 1431710008.94059.0-in.wav; mv 1431710008.94059.0-in.wav ./DEST
sox -m 1431710008.94059.0-out.wav; mv 1431710008.94059.0-out.wav ./DEST
sox -m 1431710008.94059.1-in.wav; mv 1431710008.94059.1-in.wav ./DEST
sox -m 1431710008.94059.1-out.wav; mv 1431710008.94059.1-out.wav ./DEST
mv 1431710008.94059.1.wav ./DEST
mv 1431710008.94059.2.wav ./DEST
sox -m 1431713190.94698.2-in.wav; mv 1431713190.94698.2-in.wav ./DEST
sox -m 1431713190.94698.2-out.wav; mv 1431713190.94698.2-out.wav ./DEST
mv 1431713190.94698.2.wav ./DEST
sox -m 1431721107.96010.0-in.wav; mv 1431721107.96010.0-in.wav ./DEST
sox -m 1431721107.96010.0-out.wav; mv 1431721107.96010.0-out.wav ./DEST
mv 1431721107.96010.1.wav ./DEST
mv DEST ./DEST
If you want to execute the commands, simply cut and paste these lines into a shell, or modify the bash script to execute these commands rather than print them.

Related

Renaming files in subdirectories deletes the files

I am writing a script that renames *.MP4 files on an inserted SD card and then rsyncs them.
The directory with the *.MP4 files does not always have the same name:
eg: it could be /DCIM/123_PANA/ or /DCIM/141_PANA/ etc
So I'm trying to write a script that will see what folders are in the /DCIM path, and rename all the *.MP4 files, (there is also a MISC folder in this path which I suspect is causing the issue)
I am using a couple of variables to rename the files also
What I have is:
for f in /media/pi/LUMIX/DCIM/*; do
if [ -d "$f" ]; then
echo $f
for file in $(find $f -name 'P*.MP4')
do
echo $file ">" $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
mv $file $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
done
fi
done
But what seems to happen is I end up with a single file with the prefix only (say 08_nb1_) in the _PANA folder, all the others have been deleted. Obviously this is not my desired result!
UPDATE:
$cardname is of the format _nb2_
When I do as asked replace mv with echo here is the output:
/media/pi/LUMIX/DCIM/141_PANA
mv /media/pi/LUMIX/DCIM/141_PANA/P1410192.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410192.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410193.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410193.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410194.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410194.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410195.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410195.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410196.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410196.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410197.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410197.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410198.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410198.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410199.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410199.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410200.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410200.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410201.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410201.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410202.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410202.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410203.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410203.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410204.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410204.MP4
/media/pi/LUMIX/DCIM/MISC
OK I fixed it by filtering the additional directory names and limiting to only the ones with *_PANA - which solved the issue. I also added a the rsync part and demounted the SD card (if required using Zenity)
A text file placed on the SD card identifies it as a unique card giving each file a unique name when rsyncing it to the backup folder. Renaming on the SD card means
that it can still be used an written to if not full, but we then know which files have been backed up.
Very useful in the field when filming with multiple cards, crews. All running on a Rpi4
for f in /media/pi/LUMIX/DCIM/*_PANA/; do
if [ -d "$f" ]; then
echo "$f"
for file in $(find $f -wholename '*_PANA/P*.MP4')
do
mv "$file" $(dirname "${file}")/$(date +"%d")"$cardname"$(basename "${file}")
done
rsync --stats -u --progress "$f"/*.MP4 /media/pi/VDRIVE/ | tee /home/pi/Documents/ytu/rsync.txt | zenity --icon-name="dialog-warning" \
--width=300 --progress --pulsate --auto-close --auto-kill \
--title="Copying $sdn"
zenity --question --text="Unmount Card?"
if [ $? = 0 ]; then
umount /media/pi/LUMIX
else
exit
fi
fi
done

Avoid collision, if copying files

I was trying to copy all files of a certain filetype from all subfolders to one place. Unfortunately, this might cause collisions, if two files have the same name from two different subfolders.
I was using
find ./ -name '*.jpg' -exec mv -u '{}' . \;
How can I adjust this to automatically rename files (e.g. append "_1") to avoid collisions.
Or better: check if the files are the same (e.g. same size) beforehand. If yes, ignore (overwrite would be fine, too). If No, rename to avoid collision.
Suggestion would be appreciated. Thanks!
You could check before moving each individual file. Here I've used cksum to compare, which returns both the filesize and a simple checksum.
find ./ -name '*.jpg' -print0 |
while read -d '' -r path; do
file=$(basename "$path")
if [[ -e $file ]]; then
if [[ $(cksum "$file" | awk '{print $1 $2}') = $(cksum "$path" | awk '{print $1 $2}') ]]; then
continue
fi
read -n 1 -p "File '$file' would be overwritten by '$path', continue? (y/N) " -r prompt </dev/tty
if [[ $prompt != [Yy] ]]; then
continue
fi
fi
mv -f -v "$path" "$file"
done

batch rename dropbox conflict files

I have a large number of conflict files generated (incorrectly) by the dropbox service. These files are on my local linux file system.
Example file name = compile (master's conflicted copy 2013-12-21).sh
I would like to rename the file with its correct original name, in this case compile.sh and remove any existing file with that name. Ideally this could be scripted or in such a way to be recursive.
EDIT
After looking over the solution provided and playing around and further research I cobbled together something that works well for me:
#!/bin/bash
folder=/path/to/dropbox
clear
echo "This script will climb through the $folder tree and repair conflict files"
echo "Press a key to continue..."
read -n 1
echo "------------------------------"
find $folder -type f -print0 | while read -d $'\0' file; do
newname=$(echo "$file" | sed 's/ (.*conflicted copy.*)//')
if [ "$file" != "$newname" ]; then
echo "Found conflict file - $file"
if test -f $newname
then
backupname=$newname.backup
echo " "
echo "File with original name already exists, backup as $backupname"
mv "$newname" "$backupname"
fi
echo "moving $file to $newname"
mv "$file" "$newname"
echo
fi
done
all files from current directory:
for file in *
do
newname=$(echo "$file" | sed 's/ (.*)//')
if [ "$file" != "$newname" ]; then
echo moving "$file" to "$newname"
# mv "$file" "$newname" #<--- remove the comment once you are sure your script does the right thing
fi
done
or to recurse, put the following into a script that i'll call /tmp/myrename:
file="$1"
newname=$(echo "$file" | sed 's/ (.*)//')
if [ "$file" != "$newname" ]; then
echo moving "$file" to "$newname"
# mv "$file" "$newname" #<--- remove the comment once you are sure your script does the right thing
fi
then find . -type f -print0 | xargs -0 -n 1 /tmp/myrename (This is a bit hard to do on the command line without using an extra script because the file names contain blanks).
a small contribution:
I've had a problem this this script. The files with spaces in their name do not made a copy. So I modified line 17 :
-------cut-------------cut---------
if test -f "$newname"
-------cut-------------cut---------
This script displayed above is now outdated; the following works fine with the latest version of Dropbox running on Linux Mint at the time of writing:
#!/bin/bash
#modify this as needed
folder="./"
clear
echo "This script will climb through the $folder tree and repair conflict files"
echo "Press a key to continue..."
read -n 1
echo "------------------------------"
find "$folder" -type f -print0 | while read -d $'\0' file; do
newname=$(echo "$file" | sed 's/ (.*Case Conflict.*)//')
if [ "$file" != "$newname" ]; then
echo "Found conflict file - $file"
if test -f "$newname"
then
backupname=$newname.backup
echo " "
echo "File with original name already exists, backup as $backupname"
mv "$newname" "$backupname"
fi
echo "moving $file to $newname"
mv "$file" "$newname"
echo
fi
done
You can use the tool Dropbox Conflict Fix. It resolved all my conflicted copy files.

How to grep for a pattern in the files in tar archive without filling up disk space

I have a tar archive which is very big ~ 5GB.
I want to grep for a pattern on all files (and also print the name of the file that has the pattern ) in the archive but do not want to fill up my disk space by extracting the archive.
Anyway I can do that?
I tried these, but this does not give me the file names that contain the pattern, just the matching lines:
tar -O -xf test.tar.gz | grep 'this'
tar -xf test.tar.gz --to-command='grep awesome'
Also where is this feature of tar documented? tar xf test.tar $FILE
Seems like nobody posted this simple solution that processes the archive only once:
tar xzf archive.tgz --to-command \
'grep --label="$TAR_FILENAME" -H PATTERN ; true'
Here tar passes the name of each file in a variable (see the docs) and it is used by grep to print it with each match. Also true is added so that tar doesn't complain about failing to extract files that don't match.
Here's my take on this:
while read filename; do tar -xOf file.tar "$filename" | grep 'pattern' | sed "s|^|$filename:|"; done < <(tar -tf file.tar | grep -v '/$')
Broken out for explanation:
while read filename; do -- it's a loop...
tar -xOf file.tar "$filename" -- this extracts each file...
| grep 'pattern' -- here's where you put your pattern...
| sed "s|^|$filename:|"; - prepend the filename, so this looks like grep. Salt to taste.
done < <(tar -tf file.tar | grep -v '/$') -- end the loop, get the list of files as to fead to your while read.
One proviso: this breaks if you have OR bars (|) in your filenames.
Hmm. In fact, this makes a nice little bash function, which you can append to your .bashrc file:
targrep() {
local taropt=""
if [[ ! -f "$2" ]]; then
echo "Usage: targrep pattern file ..."
fi
while [[ -n "$2" ]]; do
if [[ ! -f "$2" ]]; then
echo "targrep: $2: No such file" >&2
fi
case "$2" in
*.tar.gz) taropt="-z" ;;
*) taropt="" ;;
esac
while read filename; do
tar $taropt -xOf "$2" \
| grep "$1" \
| sed "s|^|$filename:|";
done < <(tar $taropt -tf $2 | grep -v '/$')
shift
done
}
Here's a bash function that may work for you. Add the following to your ~/.bashrc
targrep () {
for i in $(tar -tzf "$1"); do
results=$(tar -Oxzf "$1" "$i" | grep --label="$i" -H "$2")
echo "$results"
done
}
Usage:
targrep archive.tar.gz "pattern"
It's incredibly hacky, but you could abuse tar's -v option to process and delete each file as it is extracted.
grep_and_delete() {
if [ -n "$1" -a -f "$1" ]; then
grep -H 'this' -- "$1" </dev/null
rm -f -- "$1" </dev/null
fi
}
mkdir tmp; cd tmp
tar -xvzf test.tar.gz | (
prev=''
while read pathname; do
grep_and_delete "$prev"
prev="$pathname"
done
grep_and_delete "$prev"
)
tar -tf test.tar.gz | grep -v '/$'| \
xargs -n 1 -I _ \
sh -c 'tar -xOf test.tar.gz _|grep -q <YOUR SEARCH PATTERN> && echo _'
Try:
tar tvf name_of_file |grep --regex="pattern"
The t option will test the tar file without extracting the files. The v is verbose and the f prints he filenames. This should save you considerable hard disk space.
may help
zcat log.tar.gz | grep -a -i "string"
zgrep -i "string" log.tar.gz
http://www.commandlinefu.com/commands/view/9261/grep-compressed-log-files-without-extracting

copy a directory structure with file names without content

I have a huge directory structure of movie files. For analysis of that structure I want to copy the entire directory structure, i.e. folders and files however I don't want to copy all the movie files while I want to keep there file names. Ideally I get zero-byte files with the original movie file name.
I tried to and then rsync to my remote machine which didn't fetch the link files.
Any ideas how to do that w/o writing scripts?
You can use find:
find src/ -type d -exec mkdir -p dest/{} \; \
-o -type f -exec touch dest/{} \;
Find directory (-d) under (src/) and create (mkdir -p) them under dest/ or (-o) find files (-f) and touch them under dest/.
This will result in:
dest/src/<file-structre>
You can user mv creatively to resolve this issue.
Other (partial) solution can be achieved with rsync:
rsync -a --filter="-! */" sorce_dir/ target_dir/
The trick here is the --filter=RULE option that excludes (-) everything that is not (!) a directory (*/)
On ubuntu you can try:
cp -r --attributes-only <source_dir> <target_dir>
It doesn't copy file data.
From manpage of cp
--attributes-only
don't copy the file data, just the attributes
Note: I'm not sure this option available for other distributions, if anybody can confirm please update the answer.
I needed an alternative to this to sync only the file structure:
rsync --recursive --times --delete --omit-dir-times --itemize-changes "$src_path/" "$dst_path"
This is how I realized it:
# sync source to destination
while IFS= read -r -d '' src_file; do
dst_file="$dst_path${src_file/$src_path/}"
# new files
if [[ ! -e "$dst_file" ]]; then
if [[ -d "$src_file" ]]; then
mkdir -p "$dst_file"
elif [[ -f $src_file ]]; then
touch -r "$src_file" "$dst_file"
else
echo "Error: $src_file is not a dir or file"
fi
echo -n "+ "
ls -ld "$src_file"
# modification time changed (files only)
elif [[ -f $dst_file ]] && [[ $(date -r "$src_file") != $(date -r "$dst_file") ]]; then
touch -r "$src_file" "$dst_file"
echo -n "+ "
ls -ld "$src_file"
fi
done < <(find "$src_path" -print0)
# delete files in destination if they disappeared in source
while IFS= read -r -d '' dst_file; do
src_file="$src_path${dst_file/$dst_path/}"
# file disappeard on source
if [[ ! -e "$src_file" ]]; then
delinfo=$(ls -ld "$dst_file")
if [[ -d "$dst_file" ]] && rmdir "$dst_file" 2>/dev/null; then
echo -n "- $delinfo"
elif [[ -f $dst_file ]] && rm "$dst_file"; then
echo -n "- $delinfo"
fi
fi
done < <(find "$dst_path" -print0)
As you can see I use echo and ls to display changes.
ls > listOfMovie.txt; You will have the list of your films in a .txt file
.For multiple directories see the man page.

Resources