I am pretty stuck with that script.
#!/bin/bash
STARTDIR=$1
MNTDIR=/tmp/test/mnt
find $STARTDIR -type l |
while read file;
do
echo Found symlink file: $file
DIR=`sed 's|/\w*$||'`
MKDIR=${MNTDIR}${DIR}
mkdir -p $MKDIR
cp -L $file $MKDIR
done
I passing some directory to $1 parameter, this directory have three symbolic links. In while statement echoed only first match, after using sed I lost all other matches.
Look for output below:
[artyom#LBOX tmp]$ ls -lh /tmp/imp/
total 16K
lrwxrwxrwx 1 artyom adm 19 Aug 8 10:33 ok1 -> /tmp/imp/sym3/file1
lrwxrwxrwx 1 artyom adm 19 Aug 8 09:19 ok2 -> /tmp/imp/sym2/file2
lrwxrwxrwx 1 artyom adm 19 Aug 8 10:32 ok3 -> /tmp/imp/sym3/file3
[artyom#LBOX tmp]$ ./copy.sh /tmp/imp/
Found symlink file: /tmp/imp/ok1
[artyom#LBOX tmp]$
Can somebody help with that issue?
Thanks
You forgot to feed something to sed. Without explicit input, it reads nothing in this construction. I wouldn't use this approach anyway, but just use something like:
DIR=`dirname "$file"`
Related
How do I calculate the sum total size of multiple files located in different directories?
I have a text file containing the full path and name of the files.
I figure a simple script using while read line and du -h might do the trick...
Example of text file (new2.txt) containing list of files to sum:
/mount/st4000/media/A/amediafile.ext
/mount/st4000/media/B/amediafile.ext
/mount/st4000/media/C/amediafile.ext
/mount/st4000/media/D/amediafile.ext
/mount/st4000/media/E/amediafile.ext
/mount/st4000/media/F/amediafile.ext
/mount/st4000/media/G/amediafile.ext
/mount/st4000/media/H/amediafile.ext
/mount/st4000/media/I/amediafile.ext
/mount/st4000/media/J/amediafile.ext
/mount/st4000/media/K/amediafile.ext
Note: the folder structure is not necessarily consecutive as in A..K
Based on the suggestion from AndreaT, adapting it slightly, I tried
while read mediafile;do du -b "$mediafile"|cut -f -1>>subtotals.txt;done<new2.txt
subtotals.txt looks like
733402685
944869798
730564608
213768
13332480
366983168
6122559750
539944960
735039488
1755005744
733478912
To add all the subtotals
sum=0; while read num; do ((sum += num)); done < subtotals.txt; echo $sum
Assuming that file input is like this
/home/administrator/filesum/cliprdr.c
/home/administrator/filesum/cliprdr.h
/home/administrator/filesum/event.c
/home/administrator/filesum/event.h
/home/administrator/filesum/main.c
/home/administrator/filesum/main.h
/home/administrator/filesum/utils.c
/home/administrator/filesum/utils.h
and the result of command ls -l is
-rw-r--r-- 1 administrator administrator 13452 Oct 4 17:56 cliprdr.c
-rw-r--r-- 1 administrator administrator 1240 Oct 4 17:56 cliprdr.h
-rw-r--r-- 1 administrator administrator 8141 Oct 4 17:56 event.c
-rw-r--r-- 1 administrator administrator 2164 Oct 4 17:56 event.h
-rw-r--r-- 1 administrator administrator 32403 Oct 4 17:56 main.c
-rw-r--r-- 1 administrator administrator 1074 Oct 4 17:56 main.h
-rw-r--r-- 1 administrator administrator 5452 Oct 4 17:56 utils.c
-rw-r--r-- 1 administrator administrator 1017 Oct 4 17:56 utils.h
the simplest command to run is:
cat filelist.txt | du -cb | tail -1 | cut -f -1
with following output (in bytes)
69370
Keep in mind that du prints actual disk usage rounded up to a multiple of (usually) 4kb instead of logical file size.
For small files this approximation may not be acceptable.
To sum one directory, you will have to do a while, and export the result to the parent shell.
I used an echo an the subsequent eval :
eval ' let sum=0$(
ls -l | tail -n +2 |\
while read perms link user uid size date day hour name ; do
echo -n "+$size" ;
done
)'
It produces a line, directly evaluated, which looks like
let sum=0+205+1201+1201+1530+128+99
You just have to reproduce twice this command on both folders.
The du command doesn't have a -b option on the unix systems I have available. And there are other ways to get file size.
Assuming you like the idea of a while loop in bash, the following might work:
#!/bin/bash
case "$(uname -s)" in
Linux) stat_opt=(-c '%s') ;;
*BSD|Darwin) stat_opt=(-f '%z') ;;
*) printf 'ERROR: I don'\''t know how to run on %s\n' "$(uname -s)" ;;
esac
declare -i total=0
declare -i count=0
declare filename
while read filename; do
[[ -f "$filename" ]] || continue
(( total+=$(stat "${stat_opt[#]}" "$filename") ))
(( count++ ))
done
printf 'Total: %d bytes in %d files.\n' "$total" "$count"
This would take your list of files as stdin. You can run it in BSD unix or in Linux -- the options to the stat command (which is not internal to bash) are the bit that are platform specific.
I want to display all the column headers when I type ls -l command in bash shell in unix/linux
When we type ls -ltr on command prompt we get something like the following.
-r--r--r-- 2 makerpm root 1898 Jan 28 14:52 sample3
-r--r--r-- 2 makerpm root 1898 Jan 28 14:52 sample1
What I want is to know whether ls has any options to display with column headers:
File_Permissions Owner Group Size Modified_Time Name
-r--r--r-- 2 makerpm root 1898 Jan 28 14:52 sample3
-r--r--r-- 2 makerpm root 1898 Jan 28 14:52 sample1
exa is a replacement/enhancement for ls. If you pass on the arguments -lh with exa, it will include a header row printing the column names like so:
exa -lh
Example output:
Permissions Size User Date Modified Name
.rwx------ 19 username 29 Sep 11:25 dont_cra.sh
drw-r----- - username 29 Sep 11:26 f1
.rw-r--r--# 811k username 29 Sep 11:25 row_count.dat
.rw-r--r-- 54 username 29 Sep 11:25 some_text.txt
You can set up an alias in .bashrc that replaces ls with exa.
The last answer using sed was slick, but unfortunately, if you have color added to your output (which most people do) it removes all color. I would like to suggest a better way, and ironically, simpler too.
First off, my .bashrc USED to have the following:
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
fi
alias ls='ls -AFhls --color --group-directories-first'
To be honest, you don't need the dircolors part, that is just a little extra I use, you could have something as simple as:
alias ls='ls --color=auto'
I wanted column headers too, Googled it, and wound up here. However after I tried what the previous user suggested, with sed and realizing everything was white, and my colors have all gone away.
That's when I tried something different in my .bashrc file, and it worked.
Simply alias ls, echo first, then place a semi-colon, then your ls command.
My .bashrc file now has the following line.
alias ls='echo "Dir Size|Perms|Link Count|Owner|Group|Size|Mod. Time|Name"; ls -AFhls --color --group-directories-first'
When doing it this way, utilizing echo instead of sed, all colors continue to work.
Not sure if this is specific to my terminal's output but, this worked for me.
And this is the output it yields. As you can see, it preserves color using this method. Just be sure to keep the ${1} variable in double quotes so files and directories with spaces in the name won't cause an error.
Here is the code so you can copy and paste for testing.
long_ls() {
local VAR="Permissions|Owner|Group|Size|Modified|Name"
if [ ! "${1}" ]; then
echo -e "$VAR" | column -t -s"|" && ls -l
else
echo -e "$VAR" | column -t -s"|" && ls -l "${1}"
fi
}
alias lls=$"long_ls ${1}"
I am handed a path of a directory ( sometimes path of a file ).
Which utility / shell script will reliably give me the UUID of the filesystem on which is this directory ( or file ) located / stored ?
( by UUID of filesystems I mean the "UUID=..." entry as shown by e.g. blkid )
( this is happnening on a redhat linux )
give this line a try:
sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
in above line, $file is the variable to save the file/dir. You may want to check if the file/dir exists, before call the line.
And this line needs root permission (sudo)
\df is just for avoiding to use alias if you had one, for example with -T option, it conflicts with --output
Some test :
kent$ file="/home/kent/.vimrc"
kent$ sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
9da1040a-4a24-4a00-9c62-bad8cc3c028d
kent$ file="/etc"
kent$ sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
2860a386-af71-4a28-86d7-00ccf5d12b4d
Find the device of the mount point of the path,
DEVICE=$(df /path/to/some_file_or_directory | grep "$MOUNTPOINT\$"| cut -f1 -d" ")
and get the UUID of the device:
sudo blkid $DEVICE
Simply, you can type like this,
pchero#mywork:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Jan 23 09:03 0267689b-b929-4f30-b8a4-08c742f0746f -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 23 09:03 2d682ea1-dab0-49ba-a77a-9335ccd47e58 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 23 09:03 64e733e9-2e6a-4d3e-aabe-d0d26fbfc991 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 23 09:03 a99fb356-4e01-4a1c-af41-001b0fd8a844 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jan 23 09:03 f2f7618e-76c5-4e9a-9657-e002d9a66ccf -> ../../sda4
I have a directory in which I will have some folders with date format (YYYYMMDD) as shown below -
david#machineX:/database/batch/snapshot$ ls -lt
drwxr-xr-x 2 app kyte 86016 Oct 25 05:19 20141023
drwxr-xr-x 2 app kyte 73728 Oct 18 00:21 20141016
drwxr-xr-x 2 app kyte 73728 Oct 9 22:23 20141009
drwxr-xr-x 2 app kyte 81920 Oct 4 03:11 20141002
Now I need to extract latest date folder from the /database/batch/snapshot directory and then construct the command in my shell script like this -
./file_checker --directory /database/batch/snapshot/20141023/ --regex ".*.data" > shardfile_20141023.log
Below is my shell script -
#!/bin/bash
./file_checker --directory /database/batch/snapshot/20141023/ --regex ".*.data" > shardfile_20141023.log
# now I need to grep shardfile_20141023.log after above command is executed
How do I find the latest date folder and construct above command in a shell script?
Look, this is one of approaches, just grep only folders that have 8 digits:
ls -t1 | grep -P -e "\d{8}" | head -1
Or
ls -t1 | grep -E -e "[0-9]{8}" | head -1
You could try the following in your script:
pushd /database/batch/snapshot
LATESTDATE=`ls -d * | sort -n | tail -1`
popd
./file_checker --directory /database/batch/snapshot/${LATESTDATE}/ --regex ".*.data" > shardfile_${LATESTDATE}.log
See BashFAQ#099 aka "How can I get the newest (or oldest) file from a directory?".
That being said, if you don't care for actual modification time and just want to find the most recent directory based on name you can use an array and globbing (note: the sort order with globbing is subject to LC_COLLATE):
$ find
.
./20141002
./20141009
./20141016
./20141023
$ foo=( * )
$ echo "${foo[${#foo[#]}-1]}"
20141023
I back up my files using rsync. Right after a sync, I ran it expecting to see nothing, but instead it looked like it was skipping directories. I've (obviously) changed names, but I believe I've still captured all the information I could. What's happening here?
$ ls -l /source/backup/myfiles
drwxr-xr-x 2 me me 4096 2010-10-03 14:00 foo
drwxr-xr-x 2 me me 4096 2011-08-03 23:49 bar
drwxr-xr-x 2 me me 4096 2011-08-18 18:58 baz
$ ls -l /destination/backup/myfiles
drwxr-xr-x 2 me me 4096 2010-10-03 14:00 foo
drwxr-xr-x 2 me me 4096 2011-08-03 23:49 bar
drwxr-xr-x 2 me me 4096 2011-08-18 18:58 baz
$ file /source/backup/myfiles/foo
/source/backup/myfiles/foo/: directory
Then I sync (expecting no changes):
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
And here's the weird part:
$ echo 'hi' > /source/backup/myfiles/foo/test
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
backup/myfiles/foo
backup/myfiles/foo/test
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
So it worked:
$ ls -l /source/backup/myfiles/foo
-rw-r--r-- 1 me me 3126091 2010-06-15 22:22 IMGP1856.JPG
-rw-r--r-- 1 me me 3473038 2010-06-15 22:30 P1010615.JPG
-rw-r--r-- 1 me me 3 2011-08-24 13:53 test
$ ls -l /destination/backup/myfiles/foo
-rw-r--r-- 1 me me 3126091 2010-06-15 22:22 IMGP1856.JPG
-rw-r--r-- 1 me me 3473038 2010-06-15 22:30 P1010615.JPG
-rw-r--r-- 1 me me 3 2011-08-24 13:53 test
but still:
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
Other notes:
My actual directories "foo" and "bar" do have spaces, but no other strange characters. Other directories have spaces and have no problem. I 'stat'-ed and saw no differences between the directories that don't rsync and the ones that do.
If you need more information, just ask.
Are you absolutely sure those individual files are not symbolic links?
Rsync has a few useful flags such as -l which will "copy symlinks as symlinks". Adding -l to your command:
rsync -rtvpl /source/backup /destination
I believe symlinks are skipped by default because they can be a security risk. Check the man page or --help for more info on this:
rsync --help | grep link
To verify these are symbolic links or pro-actively to find symbolic links you can use file or find:
$ file /path/to/file
/path/to/file: symbolic link to `/path/file`
$ find /path -type l
/path/to/file
Are you absolutely sure that it's not a symbolic link directory?
try a:
file /source/backup/myfiles/foo
to make sure it's a directory
Also, it could very well be a loopback mount
try
mount
and make sure that /source/backup/myfiles/foo is not listed.
You should try the below command, most probably it will work for you:
rsync -ravz /source/backup /destination
You can try the following, it will work
rsync -rtvp /source/backup /destination
I personally always use this syntax in my script and works a treat to backup the entire system (skipping sys/* & proc/* nfs4/*)
sudo rsync --delete --stats --exclude-from $EXCLUDE -rlptgoDv / $TARGET/ | tee -a $LOG
Here is my script run by root's cron daily:
#!/bin/bash
#
NFS="/nfs4"
HOSTNAME=`hostname`
TIMESTAMP=`date "+%Y%m%d_%H%M%S"`
EXCLUDE="/home/gcclinux/Backups/root-rsync.excludes"
TARGET="${NFS}/${HOSTNAME}/SYS"
LOGDIR="${NFS}/${HOSTNAME}/SYS-LOG"
CMD=`/usr/bin/stat -f -L -c %T ${NFS}`
## CHECK IF NFS IS MOUNTED...
if [[ ! $CMD == "nfs" ]];then
echo "NFS NOT MOUNTED"
exit 1
fi
## CHECK IF LOG DIRECTORY EXIST
if [ ! -d "$LOGDIR" ]; then
/bin/mkdir -p $LOGDIR
fi
## CREATE LOG HEADER
LOG=$LOGDIR/"rsync_result."$TIMESTAMP".txt"
echo "-------------------------------------------------------" | tee -a $LOG
echo `date` | tee -a $LOG
echo "" | tee -a $LOG
## START RUNNING BACKUP
/usr/bin/rsync --delete --stats --exclude-from $EXCLUDE -rlptgoDv / $TARGET/ | tee -a $LOG
In some cases just copy file to another location (like home) then try again