Is there any way to store file creation time in linux? May be filesystem that supports that. I mean CREATION TIME not mtime or ctime.
Most (all?) Linux filesystems do not store the time when the file was created. Instead, they just store when the file was last modified, when it was last accessed, and when it last had an administrative action applied to it (e.g., changing its permissions). Of those, the best approximation to what you're looking for is usually the modification time.
I do believe that Ext4 has creation time support.
Try this debugfs:
debugfs -R 'stat <inode>' /dev/block_dev.
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x51872d50:df6167a0 -- Mon May 6 07:10:56 2013
atime: 0x5183b4e1:d2ad63cc -- Fri May 3 16:00:17 2013
mtime: 0x51872d50:df6167a0 -- Mon May 6 07:10:56 2013
crtime: 0x5183b4e1:d2ad63cc -- Fri May 3 16:00:17 2013
Understanding EXT4 (Part 2): Timestamps
On NTFS using ntfs-3g you can access the creation time via an extended attribute (see attr(5)).
If you want to store file creation times, you can use filesystem event notification tool like inotify.
It has system calls that make you aware of IN_CREATE (a file creation event) or some other event, When a file creation event happens you can simply log that with current time in a file of your preference.
Related
Is it possible to monitor all write access to the filesystem of all process under linux?
I've some different mounted filesystems. A lot of them are tempfs.
I'm interested in all writes to the root filesystem except the tempfs,devtmpfs etc.
I'm looking for something that will output: <PID xy> write n Bytes to /targe/filepath.
What monitoring tool can list all this write syscalls? Can they be filtered by mount points?
iotop (kernel version 2.6.20 or higher) or dstat could help you. E.g. iotop -o -b -d 10 like discussed in this similar thread.
/proc/diskstats has data for all the block devices.
https://www.kernel.org/doc/Documentation/iostats.txt
The /proc/diskstats file displays the I/O statistics of block devices. Each line contains the following 14 fields:
1 - major number
2 - minor mumber
3 - device name
4 - reads completed successfully
5 - reads merged
6 - sectors read
7 - time spent reading (ms)
8 - writes completed
9 - writes merged
10 - sectors written
11 - time spent writing (ms)
12 - I/Os currently in progress
13 - time spent doing I/Os (ms)
14 - weighted time spent doing I/Os (ms)
For more details refer to Documentation/iostats.txt
You can write a SystemTap script to monitor filesystem operations. Maybe you can visit the Brendan D. Gregg's blog, where there are many monitor tools.
fatrace (File Activity Trace)
fatrace reports file access events (Open, Read, Write, Close) from all running processes. Its main purpose is to find processes which keep waking up the disk unnecessarily and thus prevent some power saving.
When running it outputs one line per event in this format:
<timestamp> <processName(id)>: <accessType> </path/to/file>
For example:
23:10:21.375341 Plex Media Serv(2290): W /srv/dev-disk-by-uuid-UID/Plex/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log
From which you easily get the all necessary infos
Timestamp from the --timestamp option
Process name (who is accessing)
File operation (O-pen R-read W-rite C-lose)
Filepath (where is it writing to).
You can limit the search scope with --current-mount to only record events on partition/mount of current directory.
So simply cd into the volume which corresponds to your spinning HDD first, and there run ftrace with the --current-mount option.
Without this option, all (real) partitions/mount points are being watched.
Very practical
With it I found out easily that the reason why my NAS disk was spinning 24/7 also when nobody accessed the NAS and also no maintenance tasks where about to run was unnecessary logging of the Plex Media Server.
fsStat class instance returns mtime, atime and ctime date objects, but there seems to be API only for changing mtime and atime (last modification and access i guess). How can i change creation time to create exact copy of file as it'd be also created the same time as original one?
It's not possible at present with Node itself, but you can use https://github.com/baileyherbert/utimes (a native add-on for Node) to change the creation time (aka btime) of a file on Windows and Mac.
tl;tr: That's not possible atm (Node.js <= v6).
Even though, fs.stat() returns the birthtime of files:
birthtime "Birth Time" - Time of file creation. Set once when the file is created. On filesystems where birthtime is not available, this field may instead hold either the ctime or 1970-01-01T00:00Z (ie, unix epoch timestamp 0)…
Prior to Node v0.12, the ctime held the birthtime on Windows systems. Note that as of v0.12, ctime is not "creation time", and on Unix systems, it never was.
Updating is not possible. From https://github.com/joyent/node/issues/8252:
fs.utimes uses utime(2) (http://linux.die.net/man/2/utime), which doesn't allow you to change the ctime.
(Same holds for birthtime)
The fs.utimes and fs.utimesSync methods are built into Node.js core. See https://nodejs.org/api/fs.html#fs_fs_utimes_path_atime_mtime_callback
Note:
The value should be a Unix timestamp in seconds. For example, Date.now() returns milliseconds, so it should be divided by 1000 before passing it in.
To convert a JS Date object to seconds:
new Date().getTime()/1000|0
On a Linux system (the one in front of me is an Ubuntu 10.04, but that shouldn't matter), how can I tell which of two files created within the same second was created first? The process I control creates neither itself; in all other respects the ctime would, I think, do the trick, but the 1 second resolution is a problem.
For background, I'm trying to reliably determine whether a potentially stale pidfile refers to the current process with that pid. If there's a better way to do that, I'm all ears.
Actually, on modern Unices with modern filesystems, the file modification time is stored in a timespec. Details:
The standard says stat looks like this WRT times:
struct timespec st_atim Last data access timestamp.
struct timespec st_mtim Last data modification timestamp.
struct timespec st_ctim Last file status change timestamp.
And a timespec
time_t tv_sec seconds
long tv_nsec nanoseconds
So, doing a stat on my Linux 2.6.39:
Access: 2011-07-14 15:38:20.016666721 +0300
Modify: 2011-06-10 03:06:12.000000000 +0300
Change: 2011-06-17 11:01:35.416667110 +0300
In conclusion, I think you've got enough precision there if the hardware is supplying it.
You can try ls -rt to sort the files by time on the hope that the file header has more precision than the default list time format displays. But if the file system doesn't have the information, there is no way to do this.
Other options? You could add an ID to the file and always increment it but as soon as you try to load this ID from the file system (when you create a new process), you'll run into problems with locking.
So how can you make sure the PID file is not stale? Answer: Use the daemon script. It runs a process in the background and makes sure the PID file gets deleted as soon as the process exits.
We are having a problem on Linux with directory inodes getting large and slow to navigate over time, as many files are created and removed. For example:
% ls -ld foo
drwxr-xr-x 2 webuser webuser 1562624 Oct 26 18:25 foo
% time find foo -type f | wc -l
518
real 0m1.777s
user 0m0.000s
sys 0m0.010s
% cp -R foo foo.tmp
% ls -ld foo.tmp
drwxr-xr-x 2 webuser webuser 45056 Oct 26 18:25 foo.tmp
% time find foo.tmp -type f | wc -l
518
real 0m0.198s
user 0m0.000s
sys 0m0.010s
The original directory has 518 files, takes 1.5 MB to represent, and takes 1.7 seconds to traverse.
The rebuilt directory has the same number of files, takes 45K to represent and .2 seconds to traverse.
I'm wondering what would cause this. My guess is fragmentation - this is not supposed to be a problem with Unix file systems in general, but in this case we are using the directory for short-term cache files and are thus constantly creating, renaming and removing a large number of small files.
I'm also wondering if there's a way to dump the literal binary contents of the directory - that is, read the directory as if it were a file - which would perhaps give me insight into why it is so big. Neither read() nor sysread() from Perl will allow me to:
swartz> perl -Mautodie -MPOSIX -e 'sysopen(my $fh, "foo", O_RDONLY); my $len = sysread($fh, $buf, 1024);'
Can't sysread($fh, '', '1024'): Is a directory at -e line 1
System info:
Linux 2.6.18-128.el5PAE #1 SMP Wed Dec 17 12:02:33 EST 2008 i686 i686 i386 GNU/Linux
Thanks!
Jon
For question 1, external fragmentation normally causes an overhead of about 2x or so,1 plus you have internal fragmentation from allocation granularity. Neither of these comes close to explaining your observation.
So, I don't think it is normal steady-state fragmentation.
The most obvious speculation is that 1.5MB is the high-water mark; at one time it really did have either 1.5MB bytes of entries or 1.5MB/2 bytes of entries with expected fragmentation.
Another speculation is that the 50% rule is being defeated by a non-Markovian allocation. Imagine that I name files with "tmp%d", so, tmp1, tmp2, ... tmp1000, tmp1001, ...
The problem here is that rm tmp1 doesn't make room for tmp1001. This is obviously a wild guess.
Q2: There isn't a good way to read the raw directory. AFAIK, you would need to either hack the kernel or use debugfs to change the inode type, read it, then change it back, or use debugfs to read the inode, get the block numbers, then read the blocks. A functional debugging approach is probably more reasonable.
You can address the performance issue by making sure that indexing is enabled. See tune2fs.
1Knuth's fifty percent rule: in the steady state, 50% of ops are allocations, 50% are frees, 50% of free blocks merge, then holes are 50% of allocations, and 50% of the space is wasted. (Aka, 100% overhead.) This is considered "normal". Malloc has the same problem.
It happens because of fragmentation due to reiterated file creation and deletion. As the inode size increases, it never shrinks again, so it stays big even if mostly empty.
I think you have mainly two measures to confront the problem:
Build a subdirectories structure in order to prevent too many children under one single directory parent. For example, if you are creating files whose path has a format like dir/file-%06d, then you are causing it to have one million children with their expected huge directory inode. You would rather design some subtree structure decomposing the filenames into their variable prefixes, e.g., if your file is file-123456.ext, allocate them under something like dir/files/1/2/3/4/123456.ext. This strategy will limit the maximum amount of children to 1000 under the final directory leaf. The level of decompositon would depend on the size of the variable part of the filename.
As a countermeasure, once you already have huge directory inodes, there's little more to do other than to create a new (small-inode) sibling directory, to move all the original (.)files to the new directory, to delete the original directory and to rename the new directory to the original name. Beware of concurrently running services under the original path.
Some shell-fu involving find and stat --printf='%b' or %s on directories may help you to detect other troublesome spots into your filesystem, and put them under closer observation.
For specific filesystem details, look at this post in ServerFault.com
How can I get a history of uptimes for my debian box? After a reboot, I dont see an option for the uptime command to print a history of uptimes. If it matters, I would like to use these uptimes for graphing a page in php to show my webservers uptime lengths between boots.
Update:
Not sure if it is based on a length of time or if last gets reset on reboot but I only get the most recent boot timestamp with the last command. last -x also does not return any further info. Sounds like a script is my best bet.
Update:
Uptimed is the information I am looking for, not sure how to grep that info in code. Managing my own script for a db sounds like the best fit for an application.
Install uptimed. It does exactly what you want.
Edit:
You can apparantly include it in a PHP page as easily as this:
<? system("/usr/local/bin/uprecords -a -B"); ?>
Examples
the last command will give you the reboot times of the system. You could take the difference between each successive reboot and that should give the uptime of the machine.
update
1800 INFORMATION answer is a better solution.
You could create a simple script which runs uptime and dumps it to a file.
uptime >> uptime.log
Then set up a cron job for it.
Try this out:
last | grep reboot
according to last manual page:
The pseudo user reboot logs in each time the system is rebooted.
Thus last reboot will show a log of all reboots since the log file
was created.
so last column of #last reboot command gives you uptime history:
#last reboot
reboot system boot **************** Sat Sep 21 03:31 - 08:27 (1+04:56)
reboot system boot **************** Wed Aug 7 07:08 - 08:27 (46+01:19)
This isn't stored between boots, but The Uptimes Project is a third-party option to track it, with software for a range of platforms.
Another tool available on Debian is uptimed which tracks uptimes between boots.
I would create a cron job to run at the required resolution (say 10 minutes) by entering the following [on one single line - I've just separated it for formatting purposes] in your crontab (cron -l to list, cron -e to edit).
0,10,20,30,40,50 * * * *
/bin/echo $(/bin/date +\%Y-\%m-\%d) $(/usr/bin/uptime)
>>/tmp/uptime.hist 2>&1
This appends the date, time and uptime to the uptime.hist file every ten minutes while the machine is running. You can then examine this file manually to figure out the information or write a script to process it as you see fit.
Whenever the uptime reduces, there's been a reboot since the previous record. When there are large gaps between lines (i.e., more than the expected ten minutes), the machine's been down during that time.
This information is not normally saved. However, you can sign up for an online service that will do this for you. You just install a client that will send your uptime to the server every 5 minutes and the site will present you with a graph of your uptimes:
http://uptimes-project.org/
i dont think this information is saved between reboots.
if shutting down properly you could run a command on shutdown that saves the uptime, that way you could read it back after booting back up.
Or you can use tuptime https://sourceforge.net/projects/tuptime/ for a total uptime time.
You can use tuptime, a simple command for report the total uptime in linux keeping it betwwen reboots.
http://sourceforge.net/projects/tuptime/
Since I haven't found an answer here that would help retroactively, maybe this will help someone.
kern.log (depending on your distribution) should log a timestamp.
It will be something like:
2019-01-28T06:25:25.459477+00:00 someserver kernel: [44114473.614361] somemessage
"44114473.614361" represents seconds since last boot, from that you can calculate the uptime without having to install anything.
Nagios can make even very beautiful diagrams about this.
Use Syslog
For anyone coming here searching for their past uptime.
The solution of #1800_Information is a good advise for the future, but I needed to find information for my past uptimes on a specific date.
Therefore I used syslog to determine when that day the system was started (first log entry of that day) and when the system was shutdown again.
Boot time
To get the system start time grep for the month and day and show only the first lines:
sudo grep "May 28" /var/log/syslog* | head
Shutdown time
To get the system shutdown time grep for the month and day and show only the last few lines:
sudo grep "May 28" /var/log/syslog* | tail