I have a code to output data to a txt file but i would like to remove the ./ from the output on each file name
code is as follows
#!/bin/bash
# fill with more extensions or have it as a cmd line arg
TYPES=( mov mp4 avi mp3 wma)
DIR=$1
# Create a regex of the extensions for the find command
echo "Available Media Files in Directory"
TYPES_RE="\\("${TYPES[1]}
for t in "${TYPES[#]:1:${#TYPES[*]}}"; do
TYPES_RE="${TYPES_RE}\\|${t}"
done
TYPES_RE="${TYPES_RE}\\)"
# Set the field seperator to newline instead of space
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
# Generate output from path and size using: `stat -c "%s" filepath`
OUTPUT=""
for f in `find ${DIR} -type f -regex ".*\.${TYPES_RE}"`; do
OUTPUT=`echo ${f}`";"$OUTPUT
done
# Reset IFS
IFS=$SAVEIFS
# Reverse numeric sort the output and replace ; with \n for printing
echo $OUTPUT | tr ';' '\n' | sed 's/.*/"&"/' | sort -nr >playlist.txt
Result is:
"./You Da One.mp3"
"./Wiz Khalifa Roll Up.mp4"
"./Vybz Kartel neva get a gal.mp3"
"./Tyga Rack City.mp4"
"./Tyga Lap Dance.mp4"
"./Travis Porter Make It Rain.mp4"
"./Travis Porter ft. Tyga Ayy Ladies.mp4"
"./Snoop Dogg feat. Wiz Khalifa Bruno Mars Young Wild & Free.mp4"
"./Shot Caller.mp3"
"./Chris Brown - Your Body.mp4"
"./Chris Brown Turn Up The Music.mp4"
Need to remove the ./ from each file
Thanks
You can use basename command to strip the directory filename, in your script probably a good way to use it is in the find command, i.e.
find ${DIR} -type f -regex ".*\.${TYPES_RE}" -exec basename '{}' \;
change
echo $OUTPUT | tr ';' '\n' | sed 's/.*/"&"/' | sort -nr >playlist.txt
to
echo $OUTPUT | tr ';' '\n' | sed 's/^\.\///gi' | sed 's/.*/"&"/' | sort -nr >playlist.txt
Related
I need to calculate folder size in bytes.
if folder name contains space /folder/with spaces/ then following command not work properly
wc -c `find /folder -type f` | grep total | awk '{print $1}'
with error
wc: /folder/with: No such file or directory
wc: spaces/file2: No such file or directory
How can it done?
Try this line instead:
find /folder -type f | xargs -I{} wc -c "{}" | awk '{print $1}'
You need the names individually quoted.
$: while read n; # assign whole row read to $n
do a+=("$n"); # add quoted "$n" to array
done < <( find /folder -type f ) # reads find as a stream
$: wc -c "${a[#]}" | # pass wc the quoted names
sed -n '${ s/ .*//; p; }' # ignore all but total, scrub and print
Compressed to short couple lines -
$: while read n; do a+=( "$n"); done < <( find /folder -type f )
$: wc -c "${a[#]}" | sed -n '${ s/ .*//; p; }'
This is because bash (different to zsh) word-splits the result of the command substitution. You could use an array to collect the file names:
files=()
for entry in *
do
[[ -f $entry ]] && files+=("$entry")
done
wc -c "${files[#]}" | grep .....
I am very new to bash commands. Could someone please help me on this issue?
I have to find all .txt files in the current directory and add a text in the beginning of those files. I have written below command-
find . -name *.txt | xargs sed -i '1iadd text here'
This command works fine for all the non-empty files. But it's not working for those files which are empty. I figured out that it's because the sed command fails to find 1st line in empty files and hence the command is not executed.
Is there any other way to prepend the text for empty files as well?
The ed Unix text editor can do that.
for f in *.txt; do
printf '%s\n' '0a' 'insert some text here' . w | ed -s "$f"
done
find . -type f -name '*.txt' -exec sh -c 'for f; do printf "%s\n" 0a "insert some text here" . w | ed -s "$f"; done' {} +
find . -type f -name '*.txt' -print0 | while IFS= read -rd '' file; do ed -s "$file" <<< $'0a\ninsert some text here\n.\nw\nq'; done
Can be done using an ed script.
cat script.ed
Out put
0a
insert some text here
.
w
q
Now a for loop
for f in *.txt; do ed -s "$f" < ./script.ed; done
Using find.
find . -type f -name '*.txt' -exec sh -c 'ed -s "$1" < ./script.ed' sh {} \;
A combination of both.
find . -type f -name '*.txt' -exec sh -c 'for f; do ed -s "$f" < ./script.ed; done' _ {} +
The first line is 1 and the action is i (which means insert) in your example which is also true with ed, meaning it will not work also with ed because the file is empty and contains no lines, but here I'm using the address as 0 and the action is a which means append, works.
The script.ed is an arbitrary name and the ./ in ./script.ed means the current working directory if your ed script is somewhere else add/change that to the absolute path of your ed script.
A word of caution, ed edit's the files in-place so make sure make a backup of what you're editing just in case...
It is hard to improve short oneliners. Sometimes it is good to prepare a typical, self-explaining piece of code, less compact, with some assumptions (here: temporary file), but working in 100% of the cases, e.g:
for file in `ls *.txt`; do awk 'BEGIN {print "add text here"}{print$0}' $file > tmp.tmp | mv tmp.tmp $file; done
or rather (edited):
for file in ./*.txt; do awk 'BEGIN {print "add text here"}{print$0}' "$file" > tmp.tmp | mv tmp.tmp "$file"; done
and then to try to mix the solutions.
EDIT:
If you have to use find, xargs and sed, and sed does not work properly with empty files, you can append an empty line to the file, insert the text, and then delete the appended line:
find . -type f -name '*.txt' | xargs -I "%" sh -c 'echo "" >> "%"; sed -i -e "1iadd text here" -e "$ d" "%"'
Prepend in-place with ex:
ex -s '+0s/^/add text here/' '+:wq' my_file
From stdin to stdout:
ex -s '+0s/^/add text here/' '+%print' '+:wq' /dev/stdin
Note that this only works with a single file, unlike sed.
So for your situation:
$ ls
empty not_empty
$ stat --format '%n: %s' *
empty: 0
not_empty: 6
$ cat empty
$ cat not_empty
a
b
c
$ find . -type f | xargs -I '{}' ex -s '+0s/^/add text here/' '+:wq' '{}'
$ cat empty
add text here
$ cat not_empty
add text herea
b
c
Note that -I is used to force xargs to execute ex once per file rather than try to aggregate arguments.
And for completeness, a filter from stdin to stdout example:
$ printf "%s\n" hello world | ex -s '+0s/^/add text here/' '+%print' '+:wq' /dev/stdin
add text herehello
world
$ cat /dev/null | ex -s '+0s/^/add text here/' '+%print' '+:wq' /dev/stdin
add text here
How can I find files that have long chains of consecutive 0s (zero bytes - 0x00) as a result of disk failure? For example, how can I find files that have more than 10000 zero bytes in sequence?
Sure, I can write a program using Java or other programming language, but is there a way to do it using more or less standard Linux command line tools?
Update
You can generate test file with dd if=/dev/zero of=zeros bs=1 count=100000.
This may be a start:
find /some/starting/point -type f -size +10000 -exec \
perl -nE 'if (/\x0{10000}/) {say $ARGV; close ARGV}' '{}' +
To test for a single file, named filename:
if tr -sc '\0' '\n' < filename | tr '\0' Z | grep -qE 'Z{1000}'; then
# ...
fi
You can now use a suitable find command to filter relevant files for test.
For example, all *.txt files in PWD:
while read -rd '' filename;do
if tr -sc '\0' '\n' < "$filename" | tr '\0' Z | grep -qE 'Z{1000}'; then
# For example, simply print "$filename"
printf '%s\n' "$filename"
fi
done < <(find . -type f -name '*.txt' -print0)
Find and grep should work just fine:
grep -E "(\0)\1{1000}" <file name>
if it's a single file or a group of files in the same dir
If you want to search throughout the system there's:
find /dir/ -exec grep -E "(\0)\1{1000}" {} \; 2> /dev/null
this is very slow though, if you're looking for something faster and can do without the thousand(or large number) of zeros
I'll suggest replacing the grep with 'grep 000000000*' instead
Need to search through all sub folders of current folder recursively and list all files of certain type and number of duplicates
e.g. if current folder is home and there are 2 sub folders dir1 and dir2
Then i need it to search dir1 and dir2 and list file names and number of duplicates
this is what i have so far:
I am using
find -name "*.h" .
to get a list of all the files of certain type.
I need to now count duplicates and create a new list like
file1.h 2
file2.h 1
where file1 is file name and 2 is number of duplicates overall.
Use uniq --count
You can use a set of core utilities to do this quickly. For example, given the following setup:
mkdir -p foo/{bar,baz}
touch foo/bar/file{1,2}.h
touch foo/baz/file{2,3}.h
you can then find (and count) the files with a pipeline like this:
find foo -name \*.h -print0 | xargs -0n1 basename | sort | uniq -c
This results in the following output:
1 file1.h
2 file2.h
1 file3.h
If you want other output formats, or to order the list in some other way than alphabetically by file, you can extend the pipeline with another sort (e.g. sort -nr) or reformat your columns with sed, awk, perl, ruby, or your text-munging language of choice.
find -name "*.h"|awk -F"/" '{a[$NF]++}END{for(i in a)if(a[i]>1)print i,a[i]}'
Note: This will print files with similar names and only if there are more than one.
Using a shell script, the following code will print a filename of there are duplicates, then below that list all duplicates.
The script is used as in the following exanmple:
./find_duplicate.sh ./ Project
and will search the current directory tree for file names with 'project' in it.
#! /bin/sh
find "${1}" -iname *"${2}"* -printf "%f\n" \
| tr '[A-Z]' '[a-z]' \
| sort -n \
| uniq -c \
| sort -n -r \
| while read LINE
do
COUNT=$( echo ${LINE} | awk '{print $1}' )
[ ${COUNT} -eq 1 ] && break
FILE=$( echo ${LINE} | cut -d ' ' -f 2-10000 2> /dev/null )
echo "count: ${COUNT} | file: ${FILE}"
FILE=$( echo ${FILE} | sed -e s/'\['/'\\\['/g -e s/'\]'/'\\\]'/g )
find ${1} -iname "${FILE}" -exec echo " {}" ';'
echo
done
if you wish to search for all files (and not search for a pattern in the name, replace the line:
find "${1}" -iname *"${2}"* -printf "%f\n" \
with
find "${1}" -type f -printf "%f\n" \
I need some help combining elements of scripts to form a read output.
Basically I need to get the file name of a user for the folder structure listed below and using count the number of lines in the folder for that user with the file type *.ano
This is shown in the extract below, to note that the location on the filename is not always the same counting from the front.
/home/user/Drive-backup/2010 Backup/2010 Account/Jan/usernameneedtogrep/user.dir/4.txt
/home/user/Drive-backup/2011 Backup/2010 Account/Jan/usernameneedtogrep/user.dir/3.ano
/home/user/Drive-backup/2010 Backup/2010 Account/Jan/usernameneedtogrep/user.dir/4.ano
awk -F/ '{print $(NF-2)}'
This will give me the username I need but I also need to know how many non blank lines they are in that users folder for file type *.ano. I have the grep below that works but I dont know how to put it all together so it can output a file that makes sense.
grep -cv '^[[:space:]]*$' *.ano | awk -F: '{ s+=$2 } END { print s }'
Example output needed
UserA 500
UserB 2
UserC 20
find /home -name '*.ano' | awk -F/ '{print $(NF-2)}' | sort | uniq -c
That ought to give you the number of "*.ano" files per user given your awk is correct. I often use sort/uniq -c to count the number of instances of a string, in this case username, as opposed to 'wc -l' only counting input lines.
Enjoy.
Have a look at wc (word count).
To count the number of *.ano files in a directory you can use
find "$dir" -iname '*.ano' | wc -l
If you want to do that for all directories in some directory, you can just use a for loop:
for dir in * ; do
echo "user $dir"
find "$dir" -iname '*.ano' | wc -l
done
Execute the bash-script below from folder
/home/user/Drive-backup/2010 Backup/2010 Account/Jan
and it will report the number of non-blank lines per user.
#!/bin/bash
#save where we start
base=$(pwd)
# get all top-level dirs, skip '.'
D=$(find . \( -type d ! -name . -prune \))
for d in $D; do
cd $base
cd $d
# search for all files named *.ano and count blank lines
sum=$(find . -type f -name *.ano -exec grep -cv '^[[:space:]]*$' {} \; | awk '{sum+=$0}END{print sum}')
echo $d $sum
done
This might be what you want (untested): requires bash version 4 for associative arrays
declare -A count
cd /home/user/Drive-backup
for userdir in */*/*/*; do
username=${userdir##*/}
lines=$(grep -cv '^[[:space:]]$' $userdir/user.dir/*.ano | awk '{sum += $2} END {print sum}')
(( count[$username] += lines ))
done
for user in "${!count[#]}"; do
echo $user ${count[$user]}
done
Here's yet another way of doing it (on Mac OS X 10.6):
find -x "$PWD" -type f -iname "*.ano" -exec bash -c '
ar=( "${#%/*}" ) # perform a "dirname" command on every array item
printf "%s\000" "${ar[#]%/*}" # do a second "dirname" and add a null byte to every array item
' arg0 '{}' + | sort -uz |
while IFS="" read -r -d '' userDir; do
# to-do: customize output to get example output needed
echo "$userDir"
basename "$userDir"
find -x "${userDir}" -type f -iname "*.ano" -print0 |
xargs -0 -n 500 grep -hcv '^[[:space:]]*$' | awk '{ s+=$0 } END { print s }'
#xargs -0 -n 500 grep -cv '^[[:space:]]*$' | awk -F: '{ s+=$NF } END { print s }'
printf '%s\n' '----------'
done