How to recursively list all files and directories - linux

Using the tcsh shell on Free BSD, is there a way to recursively list all files and directories including the owner, group and relative path to the file?
ls -alR comes close, but it does not show the relative path in front of every file, it shows the path at the top of a grouping i.e.
owner% ls -alR
total 0
drwxr-xr-x 3 owner group 102 Feb 1 10:50 .
drwx------+ 27 owner group 918 Feb 1 10:49 ..
drwxr-xr-x 5 owner group 170 Feb 1 10:50 subfolder
./subfolder:
total 16
drwxr-xr-x 5 owner group 170 Feb 1 10:50 .
drwxr-xr-x 3 owner group 102 Feb 1 10:50 ..
-rw-r--r-- 1 owner group 0 Feb 1 10:50 file1
-rw-r--r-- 1 owner group 0 Feb 1 10:50 file2
What I would like is output like:
owner group ./relative/path/to/file
The accepted answer to this question shows the relative path to a file, but does not show the owner and group.

How about this:
find . -exec ls -dl \{\} \; | awk '{print $3, $4, $9}'

Use tree. Few linux distributions install it by default (in these dark days of only GUIs :-), but it's always available in the standard repositories. It should be available for *BSD also, see http://mama.indstate.edu/users/ice/tree/
Use:
tree -p -u -g -f -i
or
tree -p -u -g -f
or check the man page for many other useful arguments.

Works in Linux Debian:
find $PWD -type f

find comes close:
find . -printf "%u %g %p\n"
There is also "%P", which removes the prefix from the filename, if you want the paths to be relative to the specified directory.
Note that this is GNU find, I don't know if the BSD find also supports -printf.

You've already got an answer that works, but for reference you should be able to do this on the BSDs (I've tested it on a mac) :
find . -ls

If you fancy using Perl don't use it as a wrapper around shell commands. Doing it in native Perl is faster, more portable, and more resilient. Plus it avoids ad-hoc regexes.
use File::Find;
use File::stat;
find (\&myList, ".");
sub myList {
my $st = lstat($_) or die "No $file: $!";
print getgrnam($st->gid), " ",
getpwuid($st->uid), " ",
$File::Find::name, "\n";
}

Simple way I found was this:
ls -lthr /path_to_directory/*
" * " - represents levels.
Ajiths-MBP:test ajith$ ls -lthr *
test2:
total 0
-rw-r--r-- 1 ajith staff 0B Oct 17 18:22 test2.txt
test3:
total 0
-rw-r--r-- 1 ajith staff 0B Oct 17 18:22 test3.txt
test1:
total 0
-rw-r--r-- 1 ajith staff 0B Oct 17 18:21 test1.txt
drwxr-xr-x 3 ajith staff 96B Oct 17 18:22 test1_sub_dir
Ajiths-MBP:test ajith$ ls -lthr */*
-rw-r--r-- 1 ajith staff 0B Oct 17 18:21 test1/test1.txt
-rw-r--r-- 1 ajith staff 0B Oct 17 18:22 test2/test2.txt
-rw-r--r-- 1 ajith staff 0B Oct 17 18:22 test3/test3.txt
test1/test1_sub_dir:
total 0
-rw-r--r-- 1 ajith staff 0B Oct 17 18:22 test1_sub_file.txt

Use a shell script. Or a Perl script. Example Perl script (because it's easier for me to do):
#!/usr/bin/perl
use strict;
use warnings;
foreach(`find . -name \*`) {
chomp;
my $ls = `ls -l $_`;
# an incomprehensible string of characters because it's Perl
my($owner, $group) = /\S+\s+\S+\s+(\S+)\s+(\S)+/;
printf("%-10s %-10s %s\n", $owner, $group, $_);
}
Perhaps a bit more verbose than the other answers, but should do the trick, and should save you having to remember what to type. (Code untested.)

Related

Bash shell complains of invalid options I didn't use ... only on *.mp4 file extensions

I have a bunch of MP4 files that look like this:
-rw-rw-r-- 1 116M Apr 19 06:08 lULIqx9Akn4.mp4
These are youtube videos. When I try to do anything with all of them, I get a weird error. Every command I try says that I'm using invalid options (that I am not using). Here are some examples.
$ ls *.mp4
/bin/ls: invalid option -- '7'
Try '/bin/ls --help' for more information.
$ mv *.mp4 videos/
mv: invalid option -- 'L'
Try 'mv --help' for more information.
$ cp *.mp4 videos/.
cp: invalid option -- '7'
Try 'cp --help' for more information.
It doesn't do the same thing with a different extension (*.mp3, *.txt, *.sh).
What's going on? How do I fix this?
I used this as a cheap workaround,
find . -name "*.mp4" -exec mv {} videos/. \;
but I want to understand what's happening, not just get the job done.
One of your filenames starts with a hyphen, e.g,. -7 or -L. Try ls -- *.mp4 or cp -- *.mp4 videos. Also, allow me to suggest UNIX and Linux Stack Exchange for shell questions :) .
Solution:
Either move the files,
mv -- *.mp4 ./videos
or rename the files in situ...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Explanation:
My sense is you have a file with a leading - in the directly... most commands stop you creating such files but if you copy them from another operating system it can occur. Thus, you need to rename any files with a leading - in their filename...
Let me explain with an example...
Let's try to create a file with a leading -:
touch "-7ULIqx9Akn4.mp4"
touch: illegal option -- 7
we can get around this as follows:
>touch -- "-7ULIqx9Akn4.mp4"
> ls -al -- -*.mp4
total 0
-rw-r--r--# 1 n staff 0 Apr 29 13:02 -7ULIqx9Akn4.mp4
ok, now lets set up an example and demonstrate a solution...
> ls -la
total 0
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -75438752.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -85438750.mp4
drwxr-xr-x# 7 n staff 238 Apr 29 12:49 .
drwxr-xr-x# 6 n staff 204 Apr 29 11:18 ..
-rw-r--r--# 1 n staff 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-rw-r--# 1 n staff 0 Apr 29 11:19 lULIqx9Akn4.mp4
next:
ls -- -*.mp4
-75438752.mp4 -85438750.mp4
ok, lets now rename these files...
A little explanation here, the following command uses mv to remove the leading character. i.e. Find files with a leading - and remove the leading character {$file:1} from the filename...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Result:
> for file in -*.mp4; do mv -- "$file" "${file:1}"; done
> ll
total 0
drwxr-xr-x# 7 n 238 Apr 29 12:52 ./
drwxr-xr-x# 6 n 204 Apr 29 11:18 ../
-rw-r--r--# 1 n 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 75438752.mp4
-rw-r--r--# 1 n 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 85438750.mp4
-rw-rw-r--# 1 n 0 Apr 29 11:19 lULIqx9Akn4.mp4
Note
The above does not account for duplicate file names...

Bash - Filter out directories from ls -la output, but leave directory "."

I am trying to count size of all files and subdirectories starting from ./ using oneliner:
ls -laR | grep -v "\.\." | awk '{total += $5} END {print total}'
but this counts size of subdirectories twice because output of ls -laR | grep -v "\.\." is:
.:
total 32
drwxr-xr-x 3 root root 4096 Nov 29 22:59 .
-rw-r--r-- 1 root root 55 Nov 29 02:19 131
-rw-r--r-- 1 root root 50 Nov 29 01:28 abc
-rw-r--r-- 1 root root 1000 Nov 29 01:27 access.log
drwxr-xr-x 2 root root 4096 Nov 29 22:24 asd
-rwx------ 1 root root 458 Nov 29 02:54 oneliners.sh
-rwx------ 1 root root 2136 Nov 29 17:56 regexp.sh.skript
./asd:
total 32
drwxr-xr-x 2 root root 4096 Nov 29 22:24 .
-rw-r--r-- 1 root root 21298 Nov 29 22:26 asd
so it counts directory asd twice. once in listing of directory .: as:
drwxr-xr-x 2 root root 4096 Nov 29 22:24 asd
and 2nd time in listing of directory ./asd: as:
drwxr-xr-x 2 root root 4096 Nov 29 22:24 .
I expect, this will happen for every subdirectory. Is there a way to remove them once from ls output? Usint grep -v '^d' removes all directories, so they wont be counted at all. I know I can do it simply bu using du -sb, but I need it to be done with fancy oneliner.
ls -FlaR |grep -v '\s\.\{1,\}/$' |awk '{total += $5} END {print total}'
includes the size of folders inside '.', but not the size of '.' itself. Comparing with du, the answer is quite different -as du is about the space on disk (relates to blocks).
The answer I get using your awk script is closer to what the OS reports -if you subtract the directory sizes you get a match, which suggests that MacOS X uses a method similar to
ls -FlaR |grep -v '^d.*/$' |awk '{total += $5} END {print total}'
for calculating the size of the content of a folder.

Split texts into smaller texts of n number of words

I have a large number of texts (several thousand) in a txt format and would like to split them into 500-word long chunks and to save these chunks into separate folders.
< *.txt tr -c A-Za-z0-9 \\n | grep -v '^$' | split -l 500
can do the job but it splits texts to one word per line, whereas I would like to retain the original format.
I was wondering if there is a bash command or Python script to do this.
You should also be able to do that with csplit, but I had better luck with the perl solution found here; https://unix.stackexchange.com/questions/66513/how-can-i-split-a-large-text-file-into-chunks-of-500-words-or-so
Thanks to Joseph R.
$ cat generatewordchunks.pl
perl -e '
undef $/;
$file=<>;
while($file=~ /\G((\S+\s+){500})/gc)
{
$i++;
open A,">","chunk-$i.txt";
print A $1;
close A;
}
$i++;
if($file=~ /\G(.+)\Z/sg)
{
open A,">","chunk-$i.txt";
print A $1;
}
' $1
$ ./generatewordchunks.pl woord.list
$ ls -ltr
total 13
-rwxrwx--- 1 root vboxsf 5934 Jul 31 16:03 woord.list
-rwxrwx--- 1 root vboxsf 362 Jul 31 16:08 generatewordchunks.pl
-rwxrwx--- 1 root vboxsf 4203 Jul 31 16:11 chunk-1.txt
-rwxrwx--- 1 root vboxsf 1731 Jul 31 16:11 chunk-2.txt

backup script in shell

I am new in shell script.Will you please suggest how to write backup shell script. I am having following formated data in target directory.
StoreID_date_time.zip
Like:
-rw------- 1 rupesh ldapusers 8267310 Mar 22 12:00 44_22032014_115629.zip
-rw------- 1 rupesh ldapusers 8269938 Mar 22 12:07 44_22032014_120013.zip
-rw------- 1 rupesh ldapusers 8267110 Mar 22 12:14 44_22032014_120704.zip
-rw------- 1 rupesh ldapusers 8254223 Mar 22 14:25 45_22032014_142155.zip
-rw------- 1 rupesh ldapusers 7871060 Mar 22 12:11 48_22032014_120813.zip
-rw------- 1 rupesh ldapusers 8314418 Mar 22 12:22 48_22032014_121038.zip
-rw------- 1 rupesh ldapusers 8254699 Mar 24 12:13 49_22032014_145338.zip
Now I want to backup files with following way:
Backup directory : /backup/date/storeid/zip files of that store
like:
/backup/22032014/44/44_22032014_115629.zip,44_22032014_120013.zip...so on
/backup/22032014/45/45_22032014_142155.zip
/backup/22032014/48/48_22032014_120813.zip,48_22032014_121038.zip
/backup/22032014/49/49_22032014_145338.zip
for next day /backup/23032014/respective_storeIDfolder&files
Please give some hint or code example so I can move foreword.
I have coded in bare minimum steps without doing a real check but verified it. It works fine with some dummy files I created on my box :)
#!/bin/bash
for i in $(find * -type f -iname '*.zip' )
do
echo "Zip file : "$i
store_id=$(echo $i | cut -d "_" -f 1 );
timestamp=$(echo $i | cut -d "_" -f 2 );
echo Store id = ${store_id}
# I am assuming all these directories here will be of teh same pattern name. Else put a numeric check down.
mkdir -p /backup/${timestamp}/${store_id}
cp -f $i /backup/${timestamp}/${store_id}/
done;

Tail latest file that matches a selected rule

Have a directory that multiple processes log to and I want to tail the latest file of a selected process.
in ~/bashrc I have added the following
function __taillog {
tail -f $(find $1 -maxdepth 1 -type f -printf "%T# %p\n" | sort -n | tail -n 1 | cut -d' ' -f 2-)
}
alias taillog='__taillog'
Taken from: https://superuser.com/questions/117596/how-to-tail-the-latest-file-in-a-directory
An example of the log file directory
-rw-r--r-- 1 genesys genesys 2284 Mar 19 16:34 gdalog.20130319_163436_906.log
-rw-r--r-- 1 genesys genesys 131072 Mar 19 16:34 gdalog.20130319_163436_906.snapshot.log
-rw-r--r-- 1 genesys genesys 10517 Mar 19 16:54 lcalog.20130319_163332_719.log
-rw-r--r-- 1 genesys genesys 131072 Mar 19 16:54 lcalog.20130319_163332_719.snapshot.log
-rw-r--r-- 1 genesys genesys 3792 Mar 19 16:37 StatServer_TLSTest.20130319_163700_703.log
-rw-r--r-- 1 genesys genesys 160562 Mar 19 16:52 StatServer_TLSTest.20130319_163712_045.log
-rw-r--r-- 1 genesys genesys 49730 Mar 19 16:54 StatServer_TLSTest.20130319_165217_402.log
-rw-r--r-- 1 genesys genesys 53960 Mar 20 09:55 StatServer_TLSTest.20130319_165423_702.log
-rw-r--r-- 1 genesys genesys 131072 Mar 20 09:56 StatServer_TLSTest.20130319_165423_702.snapshot.log
So to tail the all StatServer the command would be
taillog /home/user/logs/StatServer*
and it would tail the latest file for that application in the given path
The issue is the tail displays some of the file output but does not show any updates when the log file is appended. If the following command is run the log is tailed correctly
tail -f $(find /home/user/logs/StatServer* -maxdepth 1 -type f -printf "%T# %p\n" | sort -n | tail -n 1 | cut -d' ' -f 2-)
Some how adding this command as a bash function then calling it from an alias causes it to not operate as desired.
Any suggestion on a better way are welcome.
I believe you should be running this command:
taillog /home/user/logs
When you say /home/user/logs/this_app* you're passing all the files that match the pattern as argument to taillog and only using the first argument i.e. $1, and the command eventually translates to tail -f $1.
Instead $1 should be the directory where find should look for the files at that directory level (i.e. /home/user/logs in your case), then pipe the results to sort, tail and cut.
I didn't have any problems running your taillog function on linux/bash. Perhaps the log output is being buffered, so changes aren't being written right away? You might try turning off the [log]buffering option for this StatServer.

Resources