Related
I am trying to find a specific file/folder based on directory name . There are one thing i could not understand which is tabcmd -> ../bin.2012....
When i did a ls -lrt on a specific path. for e.g
cmd : ls -lrt /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251
it will list as
lrwxrwxrwx. 1 root root 33 Dec 23 07:56 atrdiag -> ../bin.20212.21.1217.2251/atrdiag
lrwxrwxrwx. 1 root root 32 Dec 23 07:56 tabcmd -> ../bin.20212.21.1217.2251/tabcmd
lrwxrwxrwx. 1 root root 39 Dec 23 07:56 serveractutil -> ../bin.20212.21.1217.2251/serveractutil
lrwxrwxrwx. 1 root root 34 Dec 23 07:56 odbcinst -> ../bin.20212.21.1217.2251/odbcinst
lrwxrwxrwx. 1 root root 31 Dec 23 07:56 iusql -> ../bin.20212.21.1217.2251/iusql
lrwxrwxrwx. 1 root root 30 Dec 23 07:56 isql -> ../bin.20212.21.1217.2251/isql
lrwxrwxrwx. 1 root root 37 Dec 23 07:56 custactutil -> ../bin.20212.21.1217.2251/custactutil
lrwxrwxrwx. 1 root root 29 Dec 23 07:56 tsm -> ../bin.20212.21.1217.2251/tsm
The objective is to find tabcmd path dynamically as this path will change from time to time. "I could not use whereis command as it is not registered as a system path/variable"
Alternatively, what i did was
TEST=`find /opt/tableau -type d -name "tabcmd"`
echo "Tabcmd path is : $TEST"
However with the cmd i tried ,, it is returning empty
it returned "Tabcmd path is : "
Update :
when i tried using command below
TABCMD_PATH=`find /opt/tableau/tableau_server -name "tabcmd"`
echo $TABCMD_PATH
/opt/tableau/tableau_server/packages/bin.20212.21.1217.2251/tabcmd /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251/tabcmd
this it will print 2 times instead of 1 and if this is the approach i should take ?
It is empty because you are finding for a directory called tabcmd, you have to find by file -type f
TEST=`find /opt/tableau -type d -name "tabcmd"`
echo "Tabcmd path is : $TEST"
And the following print two path because the file is in both paths, the file link and the symbolic link:
TABCMD_PATH=`find /opt/tableau/tableau_server -name "tabcmd"`
echo $TABCMD_PATH
/opt/tableau/tableau_server/packages/bin.20212.21.1217.2251/tabcmd /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251/tabcmd
You have to find the file excluding the symbolic link:
TABCMD_PATH=`find /opt/tableau/tableau_server -type f -name "tabcmd"`
echo $TABCMD_PATH
I interpret inode as a pointer to the actual place where the file is stored.
But I have problem understanding:
If I use cp file1 file2 in a place where file2 already exists, the inode doesn't change. And If there is originally a hard-link to file2, they now both point to the new file just copied here.
The only reason I can think of is that Linux interprets this as modifying
the file instead of deleting and creating a new file. I don't understand why it's designed this way?
But when I use mv file1 file2, the inode changes to the inode of file1.
You are correct in stating that cp will modify the file instead of deleting and recreating.
Here is a view of the underlying system calls as seen by strace (part of the output of strace cp file1 file2):
open("file2", O_WRONLY|O_TRUNC) = 4
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
stat("file1", {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
open("file1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
open("file2", O_WRONLY|O_TRUNC) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "hi\n", 65536) = 3
write(4, "hi\n", 3) = 3
read(3, "", 65536) = 0
close(4) = 0
close(3) = 0
As you can see, it detects that file2 is present (stat returns 0), but then opens it for writing (O_WRONLY|O_TRUNC) without first doing an unlink.
See for example POSIX.1-2017, which specifies that the destination file shall only be unlink-ed where it could not be opened for writing and -f is used:
A file descriptor for dest_file shall be obtained by performing
actions equivalent to the open() function defined in the System
Interfaces volume of POSIX.1-2017 called using dest_file as the path
argument, and the bitwise-inclusive OR of O_WRONLY and O_TRUNC as the
oflag argument.
If the attempt to obtain a file descriptor fails and the -f option is
in effect, cp shall attempt to remove the file by performing actions
equivalent to the unlink() function defined in the System Interfaces
volume of POSIX.1-2017 called using dest_file as the path argument. If
this attempt succeeds, cp shall continue with step 3b.
This implies that if the destination file exists, the copy will succeed (without resorting to -f behaviour) if the cp process has write permission on it (not necessarily run as the user that owns the file), even if it does not have write permission on the containing directory. By contrast, unlinking and recreating would require write permission on the directory. I would speculate that this is behind the reason why the standard is as it is.
The --remove-destination option on GNU cp will make it do instead what you thought ought to be the default.
Here is the relevant part of the output of strace cp --remove-destination file1 file2. Note the unlink this time.
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
stat("file1", {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
lstat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
unlink("file2") = 0
open("file1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
open("file2", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "hi\n", 65536) = 3
write(4, "hi\n", 3) = 3
read(3, "", 65536) = 0
close(4) = 0
close(3) = 0
When you use mv and the source and destination paths are on the same file filesystem, it will do an rename, and this will have the effect of unlinking any existing file at the target path. Here is the relevant part of the output of strace mv file1 file2.
access("file2", W_OK) = 0
rename("file1", "file2") = 0
In either case where an destination path is unlinked (whether explicitly by unlink() as called from cp --remove-destination, or as part of the effect of rename() as called from mv), the link count of the inode to which it was pointing will be decremented, but it will remain on the filesystem if either the link count is still >0 or if any processes have open filehandles on it. Any other (hard) links to this inode (i.e. other directory entries for it) will remain.
Investigating using ls -i
ls -i will show the inode numbers (as the first column when combined with -l), which helps demonstrate what is happening.
Example with default cp action
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:43 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:43 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:43 file3
$ cp file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:43 file1
50 -rw-rw-r-- 2 myuser mygroup 3 Jun 13 10:43 file2 <=== exsting inode
50 -rw-rw-r-- 2 myuser mygroup 3 Jun 13 10:43 file3 <=== exsting inode
(Note existing inode 50 now has size 3).
Example with --remove-destination
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:46 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:46 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:46 file3
$ cp --remove-destination file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:46 file1
55 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:47 file2 <=== new inode
50 -rw-rw-r-- 1 myuser mygroup 6 Jun 13 10:46 file3 <=== existing inode
(Note new inode 55 has size 3. Unmodified inode 50 still has size 6.)
Example with mv
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 11:05 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 11:05 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 11:05 file3
$ mv file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 11:05 file2 <== existing inode
50 -rw-rw-r-- 1 myuser mygroup 6 Jun 13 11:05 file3 <== existing inode
#alaniwi's answer covers what is is happening, but there's an implicit why here as well.
The reason cp works the way it does is to provide a way of replacing a file with mulitple names, having all of those names refer to the new file. When the destination of cp is a file that already exists, possibly with multiple names via hard or soft links, cp will make all of those names refer to the new file. There will be no 'orphan' references to the old file left over.
Given this command, it is pretty easy to get the 'just change the file for one name' behavior -- unlink the file first. Given just that as a primitive it would be very hard to implement the 'change all references to point to the new contents' behavior.
Of course, doing rm+cp has some race condition issues (it is two commands), which is why the install command got added in BSD unix -- it basically just does rm + cp, along with some checks to make it atomic in the rare case two people try to install to the same path simultaneously, as well as the more serious problems of someone reading from the file you're trying to install to (a problem with plain cp). Then the GNU version added options to backup the old version and various other useful bookkeeping.
An inode is a collection of metadata for a file, i.e. information about a file, in a Unix/ Unix-like filesystem. It includes permission data, last access/ modify time, file size, etc.
Notably, a file's name/ path is not part of the inode. A filename is just a human-readable identifier for an inode. A file can have one or more names, the number of which is represented in the inode by its number of "links" (hard links). The number associated with the inode, the inode number, which I believe you're interpreting as its physical location on disk, is rather simply a unique identifier for the inode. An inode does contain the location of the file on disk, but that is not the inode number.
So knowing this, the difference you're seeing is in how cp and mv function. When you cp a file you're creating a new inode with a new name and copying the contents of the old file to a new location on disk. When you mv a file all you're doing is changing one of its names. If the new name is already the name of another file, the name is disassociated with the old file (and the old file's link count is reduced by 1) and associated with the new file.
You can read more about inodes here.
Specifiacally, I need to give files rw----r--
and dirs rwx--xr-x
Use umask 062.
This works because umask only unsets bits, and files aren't normally created with executable bits set in the first place:
$ umask 062
$ touch myfile; mkdir mydir
$ ls -ld myfile mydir
drwx--xr-x 1 user user 0 Dec 5 15:21 mydir
-rw----r-- 1 user user 0 Dec 5 15:21 myfile
I am running a cron-ed bash script to extract cache hits and bytes served per IP address. The script (ProxyUsage.bash) has two parts:
(uniqueIP.awk) find unique IPs and create a bash script do add up the hits and bytes
run the hits and bytes per IP
ProxyUsage.bash
#!/usr/bin/env bash
sudo gawk -f /home/maxg/scripts/uniqueIP.awk /var/log/squid3/access.log.1 > /home/maxg/scripts/pxyUsage.bash
source /home/maxg/scripts/pxyUsage.bash
uniqueIP.awk
{
arrIPs[$3]++;
}
END {
for (n in arrIPs) {
m++; # count arrIPs elements
#print "Array elements: " m;
arrAddr[i++] = n; # fill arrAddr with IPs
#print i " " n;
}
asort(arrAddr); # sort the array values
for (i = 1; i <= m; i++) { # write one command line per IP address
#printf("#!/usr/bin/env bash\n");
printf("sudo gawk -f /home/maxg/scripts/proxyUsage.awk -v v_Var=%s /var/log/squid3/access.log.1 >> /home/maxg/scripts/pxyUsage.txt\n", arrAddr[i])
}
}
pxyUsage.bash
sudo gawk -f /home/maxg/scripts/proxyUsage.awk -v v_Var=192.168.1.13 /var/log/squid3/access.log.1 >> /home/maxg/scripts/pxyUsage.txt
sudo gawk -f /home/maxg/scripts/proxyUsage.awk -v v_Var=192.168.1.14 /var/log/squid3/access.log.1 >> /home/maxg/scripts/pxyUsage.txt
sudo gawk -f /home/maxg/scripts/proxyUsage.awk -v v_Var=192.168.1.22 /var/log/squid3/access.log.1 >> /home/maxg/scripts/pxyUsage.txt
TheProxyUsage.bash script runs as scheduled and creates the pxyUsage.bash script.
However the pxyUsage.text file is not amended with the latest values when the script runs.
So far I run pxyUsage.bash every day myself, as I cannot figure out, why the result is not written to file.
Both bash scripts are set to execute. Actually the file permissions are below:
-rwxr-xr-x 1 maxg maxg 169 Mar 14 08:40 ProxySummary.bash
-rw-r--r-- 1 maxg maxg 910 Mar 15 17:15 proxyUsage.awk
-rwxrwxrwx 1 maxg maxg 399 Mar 17 06:10 pxyUsage.bash
-rw-rw-rw- 1 maxg maxg 2922 Mar 17 07:32 pxyUsage.txt
-rw-r--r-- 1 maxg maxg 781 Mar 16 07:35 uniqueIP.awk
Any hints appreciated. Thanks.
The sudo(8) command requires a pseudo-tty and you do not have one allocated under cron(8); you do have one allocated when logged in the usual way.
Instead of mucking about with sudo(8), just run the script as the correct user.
If you cannot do that, then in the root crontab, do something like this:
su - username /path/to/mycommand arg1 arg2...
This will work because root can use su(1) without neding a password.
I'm using Bash on Debian GNU/Linux 6.0. Is it possible to get the file creation date/time? Not the modification date/time.
ls -lh a.txt and stat -c %y a.txt both only give the modification time.
Unfortunately your quest won't be possible in general, as there are only 3 distinct time values stored for each of your files as defined by the POSIX standard (see Base Definitions section 4.8 File Times Update)
Each file has three distinct associated timestamps: the time of last
data access, the time of last data modification, and the time the file
status last changed. These values are returned in the file
characteristics structure struct stat, as described in <sys/stat.h>.
EDIT: As mentioned in the comments below, depending on the filesystem used metadata may contain file creation date. Note however storage of information like that is non standard. Depending on it may lead to portability problems moving to another filesystem, in case the one actually used somehow stores it anyways.
ls -i file #output is for me 68551981
debugfs -R 'stat <68551981>' /dev/sda3 # /dev/sda3 is the disk on which the file exists
#results - crtime value
[root#loft9156 ~]# debugfs -R 'stat <68551981>' /dev/sda3
debugfs 1.41.12 (17-May-2010)
Inode: 68551981 Type: regular Mode: 0644 Flags: 0x80000
Generation: 769802755 Version: 0x00000000:00000001
User: 0 Group: 0 Size: 38973440
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 76128
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x526931d7:1697cce0 -- Thu Oct 24 16:42:31 2013
atime: 0x52691f4d:7694eda4 -- Thu Oct 24 15:23:25 2013
mtime: 0x526931d7:1697cce0 -- Thu Oct 24 16:42:31 2013
**crtime: 0x52691f4d:7694eda4 -- Thu Oct 24 15:23:25 2013**
Size of extra inode fields: 28
EXTENTS:
(0-511): 352633728-352634239, (512-1023): 352634368-352634879, (1024-2047): 288392192-288393215, (2048-4095): 355803136-355805183, (4096-6143): 357941248-357943295, (6144
-9514): 357961728-357965098
mikyra's answer is good. The fact just like what he said.
[jason#rh5 test]$ stat test.txt
File: `test.txt'
Size: 0 Blocks: 8 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 588720 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 500/ jason) Gid: ( 500/ jason)
Access: 2013-03-14 01:58:12.000000000 -0700
Modify: 2013-03-14 01:58:12.000000000 -0700
Change: 2013-03-14 01:58:12.000000000 -0700
if you want to verify, which file was created first, you can structure your file name by appending system date when you create a series of files.
Note that if you've got your filesystem mounted with noatime for performance reasons, then the atime will likely show the creation time. Given that noatime results in a massive performance boost (by removing a disk write for every time a file is read), it may be a sensible configuration option that also gives you the results you want.
Creation date/time is normally not stored. So no, you can't.
You can find creation time - aka birth time - using stat and also match using find.
We have these files showing last modified time:
$ ls -l --time-style=long-iso | sort -k6
total 692
-rwxrwx---+ 1 XXXX XXXX 249159 2013-05-31 14:47 Getting Started.pdf
-rwxrwx---+ 1 XXXX XXXX 275799 2013-12-30 21:12 TheScienceofGettingRich.pdf
-rwxrwx---+ 1 XXXX XXXX 25600 2015-05-07 18:52 Thumbs.db
-rwxrwx---+ 1 XXXX XXXX 148051 2015-05-07 18:55 AsAManThinketh.pdf
To find files created within a certain time frame using find as below.
Clearly, the filesystem knows about the birth time of a file:
$ find -newerBt '2014-06-13' ! -newerBt '2014-06-13 12:16:10' -ls
20547673299906851 148 -rwxrwx--- 1 XXXX XXXX 148051 May 7 18:55 ./AsAManThinketh.pdf
1407374883582246 244 -rwxrwx--- 1 XXXX XXXX 249159 May 31 2013 ./Getting\ Started.pdf
We can confirm this using stat:
$ stat -c "%w %n" * | sort
2014-06-13 12:16:03.873778400 +0100 AsAManThinketh.pdf
2014-06-13 12:16:04.006872500 +0100 Getting Started.pdf
2014-06-13 12:16:29.607075500 +0100 TheScienceofGettingRich.pdf
2015-05-07 18:32:26.938446200 +0100 Thumbs.db
stat man pages explains %w:
%w time of file birth, human-readable; - if unknown
ls -i menus.xml
94490 menus.xml
Here the number 94490 represents inode
Then do a:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-root 4.0G 3.4G 408M 90% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 124M 27M 92M 23% /boot
/dev/mapper/vg-var 7.9G 1.1G 6.5G 15% /var
To find the mounting point of the root "/" filesystem, because the file menus.xml is on '/' that is '/dev/mapper/vg-root'
debugfs -R 'stat <94490>' /dev/mapper/vg-root
The output may be like the one below:
debugfs -R 'stat <94490>' /dev/mapper/vg-root
debugfs 1.41.12 (17-May-2010)
Inode: 94490 Type: regular Mode: 0644 Flags: 0x0
Generation: 2826123170 Version: 0x00000000
User: 0 Group: 0 Size: 4441
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 16
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
atime: 0x5266e47b -- Wed Oct 23 09:47:55 2013
mtime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
Size of extra inode fields: 4
Extended attributes stored in inode body:
selinux = "unconfined_u:object_r:usr_t:s0\000" (31)
BLOCKS:
(0-1):375818-375819
TOTAL: 2
Where you can see the creation time:
ctime: 0x5266e438 -- Wed Oct 23 09:46:48 2013
stat -c %w a.txt
%w returns the file creation(birth) date if it is available, which is rare.
Here's the link
As #mikyra explained, creation date time is not stored anywhere.
All the methods above are nice, but if you want to quickly get only last modify date, you can type:
ls -lit /path
with -t option you list all file in /path odered by last modify date.
If you really want to achieve that you can use a file watcher like inotifywait.
You watch a directory and you save information about file creations in separate file outside that directory.
while true; do
change=$(inotifywait -e close_write,moved_to,create .)
change=${change#./ * }
if [ "$change" = ".*" ]; then ./scriptToStoreInfoAboutFile; fi
done
As no creation time is stored, you can build your own system based on inotify.
Cited from https://unix.stackexchange.com/questions/50177/birth-is-empty-on-ext4/131347#131347 , the following shellscript would work to get creation time:
get_crtime() {
for target in "${#}"; do
inode=$(stat -c %i "${target}")
fs=$(df "${target}" | tail -1 | awk '{print $1}')
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null | grep -oP 'crtime.*--\s*\K.*')
printf "%s\t%s\n" "${target}" "${crtime}"
done
}
even better:
lsct ()
{
debugfs -R 'stat <'`ls -i "$1" | (read a b;echo -n $a)`'>' `df "$1" | (read a; read a b; echo "$a")` 2> /dev/null | grep --color=auto crtime | ( read a b c d;
echo $d )
}
lsct /etc
Wed Jul 20 19:25:48 2016
Another trick to add to your arsenal is the following:
$ grep -r "Copyright" /<path-to-source-files>/src
Generally speaking, if one changes a file they should claim credit in the “Copyright”. Examine the results for dates, file names, contributors and contact email.
example grep result:
/<path>/src/someobject.h: * Copyright 2007-2012 <creator's name> <creator's email>(at)<some URL>>