Formatting text and adding text-part between specific pattern - linux

On GNU/Linux, I've following output by running ls command:
$ ls -l /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 Mar 4 18:21 Documents+Edu -> ../../sda4
lrwxrwxrwx 1 root root 10 Mar 4 18:17 Ext4 -> ../../sda3
lrwxrwxrwx 1 root root 10 Mar 4 18:21 Game+Movie -> ../../sda6
lrwxrwxrwx 1 root root 10 Mar 4 18:42 OS -> ../../sda7
lrwxrwxrwx 1 root root 10 Mar 4 18:20 Software+Video -> ../../sda5
And I want the following output:
sudo mount /dev/sda4 /media/$USER/Documents+Edu
sudo mount /dev/sda3 /media/$USER/Ext4
sudo mount /dev/sda6 /media/$USER/Game+Movie
sudo mount /dev/sda7 /media/$USER/OS
sudo mount /dev/sda5 /media/$USER/Software+Video
In other words:-
Adding text sudo mount /dev/ before every /sdax and /media/$USER/ before labels like: Documents+Edu etc.
Formatting as needed.
How can I achieve desired output on Linux by using commands like: grep, cut, awk, sed, etc.?

Another option could be to avoid ls altogether iterating through the files with readlink:
$ cat ../nols.sh
cd /dev/disk/by-label
for f in *
do
v=$(basename $(readlink $f))
echo "sudo mount /dev/$v /media/\$USER/$f"
done
$ sh ../nols.sh
sudo mount /dev/sda4 /media/$USER/Documents+Edu
sudo mount /dev/sda3 /media/$USER/Ext4
sudo mount /dev/sda6 /media/$USER/Game+Movie
sudo mount /dev/sda7 /media/$USER/OS
sudo mount /dev/sda5 /media/$USER/Software+Video

Since you don't need anything apart from the names and its symbolic links, it might me best to just do ls -1 (one instead of L) ot just get the last column. Then, pipe to this awk:
awk -F"->|/" '{printf "sudo mount /dev/%s /media/$USER/%s\n", $NF, $1}'
It slices the text in either -> or / separators. Based on that, it gets the last one and the first one and populates the string accordingly.
Test
$ ls -1 /dev/disk/by-label/ | awk -F"->|/" '{printf "sudo mount /dev/%s /media/$USER/%s\n", $NF, $1}'
sudo mount /dev/sda4 /media/$USER/Documents+Edu
sudo mount /dev/sda3 /media/$USER/Ext4
sudo mount /dev/sda6 /media/$USER/Game+Movie
sudo mount /dev/sda7 /media/$USER/OS
sudo mount /dev/sda5 /media/$USER/Software+Video

# -- Prepare ---
Folder='/dev/disk/by-label/'
HeaderLen=$(ls -ld . | wc -c )
# -- execute ---
ls -l "${Folder}" | sed "1d;s#.\{${HeaderLen}\}\(.*\) -> .*\(/[^/]*\)#sudo mount /dev\2 /media/\$USER/\1#;t;d"
estimate the first partof ls that is not used (and will be removed)
list the folder pipe to a sed
remove 1st line (total info)
remove first part and take 2 group of text separate by ->, rewrtie it in new format
if substitution occur, go to end of script (print)
if no, remove the line (and cycle without print). This remove file not linked in your format

Related

Why cannot root writes on a file that it owns and has write access to?

I need to write to a.txt. The file is owned by root with a read-write access. But still I cannot write over it with a sudo. Why?
% ls -l
total 8
-rw-r--r-- 1 root staff 6 Mar 24 00:30 a.txt
% sudo echo "hi" >> a.txt
zsh: permission denied: a.txt
The redirection happens before the commands are run, i.e. using the original user.
Work-around:
sudo sh -c 'echo "hi" >> a.txt'

how to get uuid of filesystem given a path?

I am handed a path of a directory ( sometimes path of a file ).
Which utility / shell script will reliably give me the UUID of the filesystem on which is this directory ( or file ) located / stored ?
( by UUID of filesystems I mean the "UUID=..." entry as shown by e.g. blkid )
( this is happnening on a redhat linux )
give this line a try:
sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
in above line, $file is the variable to save the file/dir. You may want to check if the file/dir exists, before call the line.
And this line needs root permission (sudo)
\df is just for avoiding to use alias if you had one, for example with -T option, it conflicts with --output
Some test :
kent$ file="/home/kent/.vimrc"
kent$ sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
9da1040a-4a24-4a00-9c62-bad8cc3c028d
kent$ file="/etc"
kent$ sudo blkid -o value $(\df --output=source "$file"|tail -1)|head -1
2860a386-af71-4a28-86d7-00ccf5d12b4d
Find the device of the mount point of the path,
DEVICE=$(df /path/to/some_file_or_directory | grep "$MOUNTPOINT\$"| cut -f1 -d" ")
and get the UUID of the device:
sudo blkid $DEVICE
Simply, you can type like this,
pchero#mywork:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Jan 23 09:03 0267689b-b929-4f30-b8a4-08c742f0746f -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 23 09:03 2d682ea1-dab0-49ba-a77a-9335ccd47e58 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 23 09:03 64e733e9-2e6a-4d3e-aabe-d0d26fbfc991 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 23 09:03 a99fb356-4e01-4a1c-af41-001b0fd8a844 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jan 23 09:03 f2f7618e-76c5-4e9a-9657-e002d9a66ccf -> ../../sda4

linux command to get size of files and directories present in a particular folder? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
How can I see the size of files and directories in Linux? If use df -m, then it shows the size of all the directory at the top level, but, for the directories and files inside the directory, how do I check the size?
Use ls command for files and du command for directories.
Checking File Sizes
ls -l filename #Displays Size of the specified file
ls -l * #Displays Size of All the files in the current directory
ls -al * #Displays Size of All the files including hidden files in the current directory
ls -al dir/ #Displays Size of All the files including hidden files in the 'dir' directory
ls command will not list the actual size of directories(why?). Therefore, we use du for this purpose.
Checking Directory sizes
du -sh directory_name #Gives you the summarized(-s) size of the directory in human readable(-h) format
du -bsh * #Gives you the apparent(-b) summarized(-s) size of all the files and directories in the current directory in human readable(-h) format
Including -h option in any of the above commands (for Ex: ls -lh * or du -sh) will give you size in human readable format (kb, mb,gb, ...)
For more information see man ls and man du
There is du command.
Size of a directory and/or file, in a human-friendly way:
$ du -sh .bashrc /tmp
I memorised it as a non-existent English word dush.
--apparent-size command line switch makes it measure apparent sizes (what ls shows) rather than actual disk usage.
Use ls -s to list file size, or if you prefer ls -sh for human readable sizes.
For directories use du, and again, du -h for human readable sizes.
You can use:
ls -lh
Using this command you'll see the apparent space of the directory and true space of the files and in details the names of the files displayed, besides the size and creation date of each.
There is also a great ncdu utility - it can show directory size with detailed info about subfolders and files.
Installation
Ubuntu:
$ sudo apt-get install ncdu
Usage
Just type ncdu [path] in the command line. After a few seconds for analyzing the path, you will see something like this:
$ ncdu 1.11 ~ Use the arrow keys to navigate, press ? for help
--- / ---------------------------------------------------------
. 96,1 GiB [##########] /home
. 17,7 GiB [# ] /usr
. 4,5 GiB [ ] /var
1,1 GiB [ ] /lib
732,1 MiB [ ] /opt
. 275,6 MiB [ ] /boot
198,0 MiB [ ] /storage
. 153,5 MiB [ ] /run
. 16,6 MiB [ ] /etc
13,5 MiB [ ] /bin
11,3 MiB [ ] /sbin
. 8,8 MiB [ ] /tmp
. 2,2 MiB [ ] /dev
! 16,0 KiB [ ] /lost+found
8,0 KiB [ ] /media
8,0 KiB [ ] /snap
4,0 KiB [ ] /lib64
e 4,0 KiB [ ] /srv
! 4,0 KiB [ ] /root
e 4,0 KiB [ ] /mnt
e 4,0 KiB [ ] /cdrom
. 0,0 B [ ] /proc
. 0,0 B [ ] /sys
# 0,0 B [ ] initrd.img.old
# 0,0 B [ ] initrd.img
# 0,0 B [ ] vmlinuz.old
# 0,0 B [ ] vmlinuz
Delete the currently highlighted element with d, exit with CTRL + c
File Size in MB
ls -l --b=M filename | cut -d " " -f5
File Size in GB
ls -l --b=G filename | cut -d " " -f5
Go to the chosen directory and execute:
$ du -d 1 -h
where:
-d 1 is the depth of the directories
-h is the human-readable option
You'll see like that:
0 ./proc
8.5M ./run
0 ./sys
56M ./etc
12G ./root
33G ./var
23M ./tmp
3.2G ./usr
154M ./boot
26G ./home
0 ./media
0 ./mnt
421M ./opt
0 ./srv
2.6G ./backups
80G .
ls -l --block-size=M will give you a long format listing (needed to actually see the file size) and round file sizes up to the nearest MiB.
If you want MB (10^6 bytes) rather than MiB (2^20 bytes) units, use --block-size=MB instead.
If you don't want the M suffix attached to the file size, you can use something like --block-size=1M. Thanks Stéphane Chazelas for suggesting this.
This is described in the man page for ls; man ls and search for SIZE. It allows for units other than MB/MiB as well, and from the looks of it (I didn't try that) arbitrary block sizes as well (so you could see the file size as number of 412-byte blocks, if you want to).
Note that the --block-size parameter is a GNU extension on top of the Open Group's ls, so this may not work if you don't have a GNU userland (which most Linux installations do). The ls from GNU coreutils 8.5 does support --block-size as described above.
All you need is -l and --block-size flags
Size of all files and directories under working directory (in MBs)
ls -l --block-size=M
Size of all files and directories under working directory (in GBs)
ls -l --block-size=G
Size of a specific file or directory
ls -l --block-size=M my_file.txt
ls -l --block-size=M my_dir/
ls --help
-l use a long listing format
--block-size=SIZE: scale sizes by SIZE before printing them; e.g.,
'--block-size=M' prints sizes in units of
1,048,576 bytes; see SIZE format below
SIZE is an integer and optional unit (example: 10M is 10*1024*1024).
Units are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ...
(powers of 1000).
If you are using it in a script, use stat.
$ date | tee /tmp/foo
Wed Mar 13 05:36:31 UTC 2019
$ stat -c %s /tmp/foo
29
$ ls -l /tmp/foo
-rw-r--r-- 1 bruno wheel 29 Mar 13 05:36 /tmp/foo
That will give you size in bytes. See man stat for more output format options.
The OSX/BSD equivalent is:
$ date | tee /tmp/foo
Wed Mar 13 00:54:16 EDT 2019
$ stat -f %z /tmp/foo
29
$ ls -l /tmp/foo
-rw-r--r-- 1 bruno wheel 29 Mar 13 00:54 /tmp/foo
You can use below command to get list of files in easily human readable format.
ls -lrtsh
You can use ncdu disk usage analyzer here. It displays the size of the files and directories in an ncurses interface. You can navigate to each directory and see the files sizes from the same interface.
To install
$ sudo apt-get install ncdu
To analyze
$ ncdu <directory>
I do the following all the time:
$ du -sh backup-lr-May-02-2017-1493723588.tar.gz
NB:
-s, --summarize
display only a total for each argument
-h, --human-readable
print sizes in human readable format (e.g., 1K 234M 2G)
To get the total size of directory or the total size of file use,
du -csh <directory or filename*> | grep total
Use ls command with -h argument: [root#hots19 etc]# ls -lh
h : for human readable.
Exemple:
[root#CIEYY1Z3 etc]# ls -lh
total 1.4M
-rw-r--r--. 1 root root 44M Sep 15 2015 adjtime
-rw-r--r--. 1 root root 1.5K Jun 7 2013 aliases
-rw-r--r-- 1 root root 12K Nov 25 2015 aliases.db
drwxr-xr-x. 2 root root 4.0K Jan 11 2018 alternatives
-rw-------. 1 root root 541 Jul 8 2014 anacrontab
-rw-r--r--. 1 root root 55M Sep 16 2014 asound.conf
-rw-r--r--. 1 root root 1G Oct 6 2014 at.deny
du -sh [file_name]
works perfectly to get size of a particular file.
I prefer this command ll -sha.
I'm a Ubuntu 16.04 user myself and I find that the ll command is by far the easiest way to see a directory's contents. I've noticed that not all Linux distributions support this command, but there's probably a workaround/install for each distro out there.
Example:
user#user-XPS-15-9560:/$ ll
total 188
drwxr-xr-x 27 root root 4096 Jan 26 09:13 ./
drwxr-xr-x 27 root root 4096 Jan 26 09:13 ../
drwxr-xr-x 2 root root 4096 Jan 22 15:13 bin/
drwxr-xr-x 4 root root 12288 Jan 29 11:35 boot/
drwxr-xr-x 2 root root 4096 Sep 3 18:14 cdrom/
drwxr-xr-x 20 root root 4440 Feb 5 08:43 dev/
drwxr-xr-x 153 root root 12288 Feb 2 15:17 etc/
drwxr-xr-x 4 root root 4096 Sep 3 18:15 home/
...
The biggest advantage for me is that it's quick and really intuitive to use.
UPDATE: what I didn't know was that on Ubuntu it's a pre-configured alias. You can easily set it yourself by executing alias ll="ls -la" on the command line, or by adding this entry in your .bashrc config file:
sudo nano ~/.bashrc
...add line described above and save file by pressing Ctrl+X and Y...
source ~/.bashrc
ls -sh video.mp4 | sed s/video.mp4//g
output,
5.6M
You have to differenciate between file size and disk usage. The main difference between the two comes from the fact that files are "cut into pieces" and stored in blocks.
Modern block size is 4KiB, so files will use disk space multiple of 4KiB, regardless of how small they are.
If you use the command stat you can see both figures side by side.
stat file.c
If you want a more compact view for a directory, you can use ls -ls, which will give you usage in 1KiB units.
ls -ls dir
Also du will give you real disk usage, in 1KiB units, or dutree with the -u flag.
Example: usage of a 1 byte file
$ echo "" > file.c
$ ls -l file.c
-rw-r--r-- 1 nacho nacho 1 Apr 30 20:42 file.c
$ ls -ls file.c
4 -rw-r--r-- 1 nacho nacho 1 Apr 30 20:42 file.c
$ du file.c
4 file.c
$ dutree file.c
[ file.c 1 B ]
$ dutree -u file.c
[ file.c 4.00 KiB ]
$ stat file.c
File: file.c
Size: 1 Blocks: 8 IO Block: 4096 regular file
Device: 2fh/47d Inode: 2185244 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ nacho) Gid: ( 1000/ nacho)
Access: 2018-04-30 20:41:58.002124411 +0200
Modify: 2018-04-30 20:42:24.835458383 +0200
Change: 2018-04-30 20:42:24.835458383 +0200
Birth: -
In addition, in modern filesystems we can have snapshots, sparse files (files with holes in them) that further complicate the situation.
You can see more details in this article: understanding file size in Linux
you can use ls -sh in linux you can do sort also
you need to go to dir where you want to check the size of files
go to specific directory then run below command
# du -sh *
4.0K 1
4.0K anadb.sh --> Shell file
4.0K db.sh/ --> shell file
24K backup4/ --> Directory
8.0K backup6/ --> Directory
1.9G backup.sql.gz --> sql file

rsync prints "skipping non-regular file" for what appears to be a regular directory

I back up my files using rsync. Right after a sync, I ran it expecting to see nothing, but instead it looked like it was skipping directories. I've (obviously) changed names, but I believe I've still captured all the information I could. What's happening here?
$ ls -l /source/backup/myfiles
drwxr-xr-x 2 me me 4096 2010-10-03 14:00 foo
drwxr-xr-x 2 me me 4096 2011-08-03 23:49 bar
drwxr-xr-x 2 me me 4096 2011-08-18 18:58 baz
$ ls -l /destination/backup/myfiles
drwxr-xr-x 2 me me 4096 2010-10-03 14:00 foo
drwxr-xr-x 2 me me 4096 2011-08-03 23:49 bar
drwxr-xr-x 2 me me 4096 2011-08-18 18:58 baz
$ file /source/backup/myfiles/foo
/source/backup/myfiles/foo/: directory
Then I sync (expecting no changes):
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
And here's the weird part:
$ echo 'hi' > /source/backup/myfiles/foo/test
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
backup/myfiles/foo
backup/myfiles/foo/test
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
So it worked:
$ ls -l /source/backup/myfiles/foo
-rw-r--r-- 1 me me 3126091 2010-06-15 22:22 IMGP1856.JPG
-rw-r--r-- 1 me me 3473038 2010-06-15 22:30 P1010615.JPG
-rw-r--r-- 1 me me 3 2011-08-24 13:53 test
$ ls -l /destination/backup/myfiles/foo
-rw-r--r-- 1 me me 3126091 2010-06-15 22:22 IMGP1856.JPG
-rw-r--r-- 1 me me 3473038 2010-06-15 22:30 P1010615.JPG
-rw-r--r-- 1 me me 3 2011-08-24 13:53 test
but still:
$ rsync -rtvp /source/backup /destination
sending incremental file list
backup/myfiles
skipping non-regular file "backup/myfiles/foo"
skipping non-regular file "backup/myfiles/bar"
Other notes:
My actual directories "foo" and "bar" do have spaces, but no other strange characters. Other directories have spaces and have no problem. I 'stat'-ed and saw no differences between the directories that don't rsync and the ones that do.
If you need more information, just ask.
Are you absolutely sure those individual files are not symbolic links?
Rsync has a few useful flags such as -l which will "copy symlinks as symlinks". Adding -l to your command:
rsync -rtvpl /source/backup /destination
I believe symlinks are skipped by default because they can be a security risk. Check the man page or --help for more info on this:
rsync --help | grep link
To verify these are symbolic links or pro-actively to find symbolic links you can use file or find:
$ file /path/to/file
/path/to/file: symbolic link to `/path/file`
$ find /path -type l
/path/to/file
Are you absolutely sure that it's not a symbolic link directory?
try a:
file /source/backup/myfiles/foo
to make sure it's a directory
Also, it could very well be a loopback mount
try
mount
and make sure that /source/backup/myfiles/foo is not listed.
You should try the below command, most probably it will work for you:
rsync -ravz /source/backup /destination
You can try the following, it will work
rsync -rtvp /source/backup /destination
I personally always use this syntax in my script and works a treat to backup the entire system (skipping sys/* & proc/* nfs4/*)
sudo rsync --delete --stats --exclude-from $EXCLUDE -rlptgoDv / $TARGET/ | tee -a $LOG
Here is my script run by root's cron daily:
#!/bin/bash
#
NFS="/nfs4"
HOSTNAME=`hostname`
TIMESTAMP=`date "+%Y%m%d_%H%M%S"`
EXCLUDE="/home/gcclinux/Backups/root-rsync.excludes"
TARGET="${NFS}/${HOSTNAME}/SYS"
LOGDIR="${NFS}/${HOSTNAME}/SYS-LOG"
CMD=`/usr/bin/stat -f -L -c %T ${NFS}`
## CHECK IF NFS IS MOUNTED...
if [[ ! $CMD == "nfs" ]];then
echo "NFS NOT MOUNTED"
exit 1
fi
## CHECK IF LOG DIRECTORY EXIST
if [ ! -d "$LOGDIR" ]; then
/bin/mkdir -p $LOGDIR
fi
## CREATE LOG HEADER
LOG=$LOGDIR/"rsync_result."$TIMESTAMP".txt"
echo "-------------------------------------------------------" | tee -a $LOG
echo `date` | tee -a $LOG
echo "" | tee -a $LOG
## START RUNNING BACKUP
/usr/bin/rsync --delete --stats --exclude-from $EXCLUDE -rlptgoDv / $TARGET/ | tee -a $LOG
In some cases just copy file to another location (like home) then try again

How to check syslog in Bash on Linux?

In C we log this way:
syslog( LOG_INFO, "proxying %s", url );
In Linux how can we check the log?
How about less /var/log/syslog?
On Fedora 19, it looks like the answer is /var/log/messages. Although check /etc/rsyslog.conf if it has been changed.
By default it's logged into system log at /var/log/syslog, so it can be read by:
tail -f /var/log/syslog
If the file doesn't exist, check /etc/syslog.conf to see configuration file for syslogd.
Note that the configuration file could be different, so check the running process if it's using different file:
# ps wuax | grep syslog
root /sbin/syslogd -f /etc/syslog-knoppix.conf
Note: In some distributions (such as Knoppix) all logged messages could be sent into different terminal (e.g. /dev/tty12), so to access e.g. tty12 try pressing Control+Alt+F12.
You can also use lsof tool to find out which log file the syslogd process is using, e.g.
sudo lsof -p $(pgrep syslog) | grep log$
To send the test message to syslogd in shell, you may try:
echo test | logger
For troubleshooting use a trace tool (strace on Linux, dtruss on Unix), e.g.:
sudo strace -fp $(cat /var/run/syslogd.pid)
A very cool util is journalctl.
For example, to show syslog to console: journalctl -t <syslog-ident>, where <syslog-ident> is identity you gave to function openlog to initialize syslog.
tail -f /var/log/syslog | grep process_name
where process_name is the name of the process we are interested in
If you like Vim, it has built-in syntax highlighting for the syslog file, e.g. it will highlight error messages in red.
vi +'syntax on' /var/log/syslog
On some Linux systems (e.g. Debian and Ubuntu) syslog is rotated daily and you have multiple log files where two newest files are uncompressed while older ones are compressed:
$ ls -l /var/log/syslog*
-rw-r----- 1 root adm 888238 Aug 25 12:02 /var/log/syslog
-rw-r----- 1 root adm 1438588 Aug 25 00:05 /var/log/syslog.1
-rw-r----- 1 root adm 95161 Aug 24 00:07 /var/log/syslog.2.gz
-rw-r----- 1 root adm 103829 Aug 23 00:08 /var/log/syslog.3.gz
-rw-r----- 1 root adm 82679 Aug 22 00:06 /var/log/syslog.4.gz
-rw-r----- 1 root adm 270313 Aug 21 00:10 /var/log/syslog.5.gz
-rw-r----- 1 root adm 110724 Aug 20 00:09 /var/log/syslog.6.gz
-rw-r----- 1 root adm 178880 Aug 19 00:08 /var/log/syslog.7.gz
To search all the syslog files you can use the following commands:
$ sudo zcat -f `ls -tr /var/log/syslog*` | grep -i error | less
where zcat first decompresses and prints all syslog files (oldest first), grep makes a search and less is paging the results of the search.
To do the same but with the lines prefixed with the name of the syslog file you can use zgrep:
$ sudo zgrep -i error `ls -tr /var/log/syslog*` | less
$ zgrep -V | grep zgrep
zgrep (gzip) 1.6
In both cases sudo is required if syslog files are not readable by ordinary users.

Resources