I would like to take a snapshot of my entire proc file system, and save it in a tarball (or in the worst case concatenate all of the text files together into a single text file).
But when I run:
tar -c /proc
I get a segfault.
What's the best way to do this? Should I set up some kind of recursive walk through each file?
I only have the basic *nix utilities, such as bash, cat, ls, echo, etc. I don't have anything fancy like python or perl or java.
The linux /proc filesystem is actually kernel variables pretending to be a filesystem. There is nothing to save thus nothing to backup. If the system let you, you could rm -rf /proc and it would magically reappear upon the next reboot.
The /dev file system has real i-nodes and they can be backed up. Except they have no contents, just a major and minor number, permissions, and a name. Tools that do backup special device files only record those parameters and never try to open(2) the device. However, since the device major and minor numbers are only meaningful on the precise system they are built for, there is little cause for backing them up.
The reason that trying to tar the /proc pseudo-filesystem causes tar to segfault is because /proc has funny file behavior: things like a write-only pseudo-file may appear to have read permissions, but return an error indication if a program tries to open(2) it for backup. That's sure to drive a naïve tar to get persnickety.
Added in response to comment
It doesn't surprise me that tar had problems reading /proc/kmsg because it has some funny properties:
# strace cat /proc/kmsg
execve("/bin/cat", ["cat", "kmsg"],
open("kmsg", O_RDONLY|O_LARGEFILE) = 3
// ok, no problem opening the file for reading
fstat64(3, { st_mode=S_IFREG|0400, st_size=0,
// looks like a normal file of zero length
// but cat does not pay attention to st_size so it just
// does a blocking read
read(3, "<4>[103128.156051] ata2.00: qc t"..., 32768) = 461
write(1, "<4>[103128.156051] ata2.00: qc t"..., 461) = 461
// ...forever...
read(3, "<6>[103158.228444] ata2.00: conf"..., 32768) = 48
write(1, "<6>[103158.228444] ata2.00: conf"..., 48) = 48
+++ killed by SIGINT +++
Since /proc/kmsg is a running list of kernel messages as they happen, it never returns 0 (EOF) it just keeps going until I get bored and press ^C.
Interestingly, my tar has no trouble with /proc/kmsg:
$ tar --version
tar (GNU tar) 1.22
# tar cf /tmp/junk.tar /proc/kmsg
$ tar tvf /tmp/junk.tar
-r-------- root/root 0 2010-09-01 14:41 proc/kmsg
and if you look at the strace output, GNU tar 1.22 saw that st_length == 0 and didn't even bother opening the file for reading, since there wasn't anything there.
I can imagine that your tar saw the length was 0, allocated that much (none) space using malloc(3) which dutifully handed back a pointer to a zero length buffer. Your tar read from /proc/kmsg, got a non-zero length read, and tried to store it in the zero length buffer and got a segmentation violation.
That is but one rat-hole that awaits tar in /proc. How many more are there? Dunno. Will they behave identically? Probably not. Which of the ~1000 or so files which aren't /proc/<pid> psuedo-files are going to have weird semantics? Dunno.
But perhaps the most telling question: What sense would you make of /proc/sys/vm/lowmem_reserve_ratio, will it be different next week, and will you be able to learn anything from that difference?
While the accepted answer makes a lot of sense if you want to argue the sense of doing something like this, nevertheless there is an answer that works. Here's a script to duplicate the complete /proc file system into /tmp/proc. This can then be tarred and gzipped. I used this to keep a memory of the setup and capabilities (memory, bogomips, default processes, etc.) of my trusty old file server before I replaced it with a new one.
cd /
mkdir /tmp/proc
find /proc -type f | while read F ; do
D=/tmp/$(dirname $F)
test -d $D || mkdir -p $D
test -f /tmp/$F || sudo cat $F > /tmp/$F
done
Notes:
Permissions are not preserved since I have to use cat instead of cp.
cp -a /proc /proccopy doesn't work since it crashes on "kcore" as well. mc (Midnight Commander) succeeds in creating a copy of /proc which you can then tar and gzip, but you have to dismiss thousands of "Cannot read file XYZ" errors and it too crashes on 'kcore' with a Bus Error.
A simple answer:
ls -Rd /proc/* > proc.lst
foreach item (<proc.lst>)
echo "proc_file:$item"
if (-f $item) cat $item
end
Apropos the site advisory protocol:
"But avoid ... Making statements based on opinion..."
(IMHO...based on quite a few years experience) It doesn't require much imagination to think of some good reasons for, at a given point in time, taking a snapshot of selected elements of /proc/* and storing it, or sending it somewhere. Therefore, I would question the usefulness of an 'answer' such as:
The linux /proc filesystem is actually kernel variables pretending to be a filesystem. There is nothing to save thus nothing to backup. If the system let you, you could rm -rf /proc and it would magically reappear upon the next reboot.
...on the grounds that it doesn't answer the question, makes a false assertion, and contains gratuitous information irrelevant to the question.
Related
I have some scripts in my RedHat server which contains Microfocus COBOL programs which generates a huge file of aprox 3GB in a sort of time of 3 hours on average. The programs write their output files directly in the directory /my_test/files/.
The problem is that sometimes (randomly) some files generated contains null character sections in the middle of the file. And when I check them up, if I reexecute the script again (with the same input parameters), the output file is perfectly generated (it doesn't contain any nullchars). I've checked it a lot of times and I'm pretty sure is not the fault of the COBOL programs (they use quite simple operations). The space in use of that folder is 40%.
Some programs updates the database, and if they finish with return code 0, then the changes are commited, and I don't have any backup, so this is the point of what I'm doing.
This is an example of a file declaration of one of the problematic COBOL programs:
FILE-CONTROL.
SELECT MYFILE
ASSIGN TO MYFILE
ORGANIZATION IS SEQUENTIAL
ACCESS MODE IS SEQUENTIAL
FILE STATUS IS FILE-STATUS.
DATA DIVISION.
FILE SECTION.
FD MYFILE
LABEL RECORD STANDARD
RECORDING MODE F.
01 REG-OUTPUT PIC X(400).
I've also checked for the nulls in the COBOL programs before the NULL files, but unfortunately there are no nulls spotted.
Then I thought about creating a crontab which executes the following script each 5 seconds:
if [[ -f /tmp/sorry_im_working ]]; then
exit
fi
trap 'rm -rf /tmp/sorry_im_working' EXIT
touch /tmp/sorry_im_working
lsof | awk 'BEGIN{
sfiles="";
} {
if($1=="PROGRAM" && $9~/my_test\/files/){
sfiles=sfiles" "$9
}
}END{
comm="find "sfiles" -newermt \x27-2 seconds\x27 -exec env LC_ALL=C bash -c \x27grep -Pq \x22\x5Cx00{200}\x22 <(tail -c 1000 {}) && echo {}\x27 \x5C\x3B";
while(comm | getline sout){
print sout;
};
close(comm);
}' >> /home/ouhma/nullfiles.txt
Therefore, I would like to ask you the following questions:
Any idea of what's going on here?
Do you have any other way to trigger the lastest modified files?
What other information of interest could I add to my log?
If you construct a file d with only \x00 :
hexdump -C d
00000000 5c 78 30 30 0a |\x00.|
00000005
and you :
grep -Faq '\x00' d;echo $?
0
But they're no null caracter inside d.
Maybe, is better to use grep -Paq '\x00'
Depending on the configuration and record structure that is used for the file MF will pad different characters with hex null.
Please copy the 'ASSIGN' clause and the 'FD' clause of the COBOL program.
BTW: if your COBOL programs run three ours to do some calculations and write three GB of data back you should investigate the storage and / or get a COBOL programmer to check the programs, sounds much to slow.
I suspect you are have non-printable characters in your file, the null inserts can be controlled, take a look # INSERTNULL file configuration.
Intent:
Does Go have the functionality (package or otherwise) to perform a special file stat on Linux akin to the command file -s <path>
Example:
[root#localhost ~]# file /proc/uptime
/proc/uptime: empty
[root#localhost ~]# file -s /proc/uptime
/proc/uptime: ASCII text
Use Case:
I have a fileglob of files in /proc/* that I need to very quickly detect if they are truly empty instead of appearing to be empty.
Using The os Package:
Code:
result,_ := os.Stat("/proc/uptime")
fmt.Println("Name:",result.Name()," Size:",result.Size()," Mode:",int(result.Mode()))
fmt.Printf("%q",result)
Result:
Name: uptime Size: 0 Mode: 292
&{"uptime" '\x00' 'Ĥ' {%!q(int64=63606896088) %!q(int32=413685520) %!q(*time.Location=&{ [] [] 0 0 <nil>})} {'\x03' %!q(uint64=4026532071) '\x01' '脤' '\x00' '\x00' '\x00' '\x00' '\x00' 'Ѐ' '\x00' {%!q(int64=1471299288) %!q(int64=413685520)} {%!q(int64=1471299288) %!q(int64=413685520)} {%!q(int64=1471299288) %!q(int64=413685520)} ['\x00' '\x00' '\x00']}}
Obvious Workaround:
There is the obvious workaround of the following. But it's a little over the top to need to call in a bash shell in order to get file stats.
output,_ := exec.Command("bash","-c","file -s","/proc/uptime").Output()
//parse output etc...
EDIT/MY PRACTICAL USE CASE:
Quickly determining which files are zero size without needing to read each one of them first.
file -s /cgroup/memory/lsf/<cluster>/*/tasks | <clean up commands> | uniq -c
6 /cgroup/memory/lsf/<cluster>/<jobid>/tasks: ASCII text
805 /cgroup/memory/lsf/<cluster>/<jobid>/tasks: empty
So in this case, I know that only those 6 jobs are running and the rest (805) have terminated. Reading the file works like this:
# cat /cgroup/memory/lsf/<cluster>/<jobid>/tasks
#
or
# cat /cgroup/memory/lsf/<cluster>/<jobid>/tasks
12352
53455
...
I'm afraid you might be confusing matters here: file is special in precisely a way it "knows" a set of heuristics to carry out its tasks.
To my knowledge, Go does not have anything like this in its standard library, and I've not came across a 3rd-party package implementing a file-like functionality (though I invite you to search by relevant keywords on http://godoc.org)
On the other hand, Go provides full access to the syscall interface of the underlying OS so when it comes to querying the OS in a way file does it, there's nothing you could not do in plain Go.
So I suggest you to just fetch the source code of file, learn what it does in its mode turned on by the "-s" command-line option and implement that in your Go code.
We'll try to have you with specific problems doing that — should you have any.
Update
Looks like I've managed to grasp the OP is struggling with: a simple check:
$ stat -c %s /proc/$$/status && wc -c < $_
0
849
That is, the stat call on a file under /proc shows it has no contents but actually reading from that file returns that contents.
OK, so the solution is simple: instead of doing a call to os.Stat() while traversing the subtree of the filesystem one should instead merely attempt to read a single byte from the file, like in:
var buf [1]byte
f, err := os.Open(fname)
if err != nil {
// do something, or maybe ignore.
// A not existing file is OK to ignore
// (the POSIX error code will be ENOENT)
// because after the `path/filepath.Walk()` fetched an entry for
// this file from its directory, the file might well have gone.
}
_, err = f.Read(buf[:])
if err != nil {
if err == io.EOF {
// OK, we failed to read 1 byte, so the file is empty.
}
// Otherwise, deal with the error
}
f.Close()
You might try to be more clever and first obtain the stat information
(using a call to os.Stat()) to see if the file is a regular file—to
not attempt reading from sockets etc.
I have a fileglob of files in /proc/* that I need to very quickly
detect if they are truly empty instead of appearing to be empty.
They are truly empty in some sense (eg. they occupy no space on file system). If you want to check whether any data can be read from them, try reading from them - that's what file -s does:
-s, --special-files
Normally, file only attempts to read and
determine the type of argument files which stat(2) reports are
ordinary files. This prevents problems, because reading special files
may have peculiar consequences. Specifying the -s option causes file
to also read argument files which are block or character special
files. This is useful for determining the filesystem types of the
data in raw disk partitions, which are block special files. This
option also causes file to disregard the file size as reported by
stat(2) since on some systems it reports a zero size for raw disk
partitions.
I have large file with size of 130GB.
# ls -lrth
-rw-------. 1 root root 129G Apr 20 04:25 syslog.log
So I need to reduce file size by deleting line which starts with "Nov 2" , So I have given the following command,
sed -i '/Nov 2/d' syslog.log
So I can't edit file using VIM editor also.
When I trigger SED command , its creating backup file also. But I don't have much space in root. Please try to give alternate solution to delete particular line from this file without increasing space in server.
It does not create a real backup file. sed is a stream editor. When applied to a file with option -i it will stream that file through the sed process, write the output to a new file (a temporary one), when everything is done, it will rename the new file to the original name.
(There are options to create backup files also, but you didn't give them, so I won't mention that further.)
In your case you have a very large file and don't want to create any copy, however temporary. For this you need to open the file for reading and writing at the same time, then your sed process can overwrite the original. After this, you will have to truncate the file at the end of the writing.
To demonstrate how this can be done, we first perform a test case.
Create a test file, containing lots of lines:
seq 0 999999 > x
Now, lets say we want to remove all lines containing the digit 4:
grep -v 4 1<>x <x
This will open the file for reading and writing as STDOUT (1), and for reading as STDIN. The grep command will read all lines and will output only the lines not containing a 4 (option -v).
This will effectively overwrite the beginning of the original file.
You will not know how long the output is, so after the output the original contents of the file will appear:
…
999991
999992
999993
999995
999996
999997
999998
999999
537824
537825
537826
537827
537828
537829
…
You can use the Unix tool truncate to shorten your file manually afterwards. In a real scenario you will have trouble finding the right spot for this, so it makes sense to count the number of bytes written (using wc):
(Don't forget to recreate the original x for this test.)
(grep -v 4 <x | tee /dev/stderr 1<>x) |& wc -c
This will preform the step above and additionally print out the number of bytes written to the terminal, in this example case the output will be 3653658. Now use truncate:
truncate -s 3653658 x
Now you have the result you want.
If you want to do this in a script, i. e. without interaction, you can use this:
length=$((grep -v 4 <x | tee /dev/stderr 1<>x) |& wc -c)
truncate -s "$length" x
I cannot guarantee that this will work for files >2GB or >4GB on your machine; depending on your operating system (32bit?) and the versions of the installed tools you might run into largefile issues. I'd perform tests with large files first (>4GB as this is typically a limit for many things) and then cross your fingers and give it a try :)
Some caveats you have to keep in mind:
Of course, nobody is supposed to append log entries to that log file while the procedure is running.
Also, any abort during the running of the process (power failure, signal caught, etc.) will leave the file in an undefined state. But re-running the command again after such a mishap will in most cases produce the correct output; some lines might be doubled, but not more than a single line should be corrupted then.
The output must be smaller than the input, of course, otherwise the writing will overtake the reading, corrupting the whole result so that lines which should be there will be missing (or truncated at the start).
Is there way to compress the core files during core dump generation?
If the storage space is limited in the system, is there a way of conserving it in case of need for core dump generation with immediate compression?
Ideally the method would work on older versions of linux such as 2.6.x.
The Linux kernel /proc/sys/kernel/core_pattern file will do what you want: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#191
Set the filename to something like |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz and your core files should be saved compressed for you.
For an embedded Linux systems, following script change perfectly works to generate compressed core files in 2 steps
step 1: create a script
touch /bin/gen_compress_core.sh
chmod +x /bin/gen_compress_core.sh
cat > /bin/gen_compress_core.sh #!/bin/sh exec /bin/gzip -f - >"/var/core/core-$1.$2.gz"
ctrl +d
step 2: update the core pattern file
cat > /proc/sys/kernel/core_pattern |/bin/gen_compress_core.sh %e %p ctrl+d
As suggested by other answer, the Linux kernel /proc/sys/kernel/core_pattern file is good place to start: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#141
As documentation says you can specify the special character "|" which will tell kernel to output the file to script. As suggested you could use |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz as name, however it doesn't seem to work for me. I expect that the reason is that on my system kernel doesn't treat the > character as a output, rather it probably passes it as a parameter to gzip.
In order to avoid this problem, like other suggested you can create your file in some location I am using /home//crash/core.sh, create it using the following command, replacing with your user. Alternatively you can also obviously change the entire path.
echo -e '#!/bin/bash\nexec /bin/gzip -f - >"/home/<username>/crashes/core-$1-$2-$3-$4-$5.gz"' > ~/crashes/core.sh
Now this script will take 5 input parameters and concatenate them and add to core-path. The full paths must be specified in the ~/crashes/core.sh. Also the location of this script can be specified. Now lets tell kernel to use tour executable with parameters when generating file:
sudo sysctl -w kernel.core_pattern="|/home/<username>/crashes/core.sh %e %p %h %t"
Again should be replaced (or entire path to match location and name of core.sh script). Next step is to crash some program, lets create example crashing cpp file:
int main (){
int * a = nullptr;
int b = *a;
}
After compiling and running there are 2 options, either we will see:
Segmentation fault (core dumped)
Or
Segmentation fault
In case we see the latter, there are few possible reasons.
ulimit is not set, ulimit -c should specify what is limit for cores
apport or your distro core dump collector is not running, this should be investigated further
there is an error in script we wrote, I suggest than checking some basic dump path to check if the other things aren't reason the below should create /tmp/core.dump:
sudo sysctl -w kernel.core_pattern="/tmp/core.dump"
I know there is already an answer for this question however it wasn't obvious for me why it isn't working "out of the box" so I wanted to summarize my findings, hope it helps someone.
Using utimes, futimes, futimens, etc., it is possible to set the access and modification timestamps on a file.
Modification time is the last time the file data changed. Similarly, "ctime" or change time, is the last time attributes on the file, such as permissions, were changed. (Linux/POSIX maintains three timestamps: mtime and ctime, already discussed, and 'atime', or access time.)
Is there a function to set change timestamps? (Where "change" is the attribute modification or 'ctime', not modification time 'mtime'.) (I understand the cyclic nature of wanting to change the change timestamp, but think archiving software - it would be nice to restore a file exactly as it was.)
Are there any functions at all for creation timestamps? (I realize that ext2 does not support this, but I was wondering if Linux did, for those filesystems that do support it.)
If it's not possible, what is the reasoning behind it not being so?
For ext2/3 and possibly for ext4 you can do this with debugfs tool, assuming you want to change the ctime of file /tmp/foo which resides in disk /dev/sda1 we want to set ctime to 201001010101 which means 01 January 2010, time 01:01:
Warning: Disk must be unmounted before this operation
# Update ctime
debugfs -w -R 'set_inode_field /tmp/foo ctime 201001010101' /dev/sda1
# Drop vm cache so ctime update is reflected
echo 2 > /proc/sys/vm/drop_caches
Information taken from Command Line Kung Fu blog.
I had a similar issue, and wrote my answer here.
https://stackoverflow.com/a/17066309/391040
There are essentially two options:
Slight change in kernel (code included in link)
Change the system clock to the desired ctime, touch the file, then restore current time. (shell script for that included in link).
According to http://lists.gnu.org/archive/html/coreutils/2010-08/msg00010.html ctime cannot be faked (at least it's not intended to be fakeable):
POSIX says that atime and mtime are user-settable to arbitrary times
via the utimensat() family of syscalls, but that ctime must
unfakeably track the current time of any action that changes a file's
metadata or contents.
If you just need to change a file's ctime for some testing/debugging, bindfs might be helpful. It's a FUSE filesystem which mounts one directory into another place, and can do some transformation on the file attributes. With option --ctime-from-mtime the ctime of each file is the same as its mtime, which you can set with touch -t.
The easiest way:
1) change System time
2) copy paste a file on another location.
I tried this on windows 7 and I succeed to change all three timestamps.
The stat command on linux shows that all three timestamps are changed.
The script below automates running debugfs ... set_inode_field ... ctime ... in ismail's answer for many files. It will copy ctime values from files in /media/MYUSER/MYFS/FOO/BAR (recursively) to /media/MYUSER/MYFS2/FOO/BAR, and umount /media/MYUSER/MYFS2 as a side effect. It will work only if the filesystem of /media/MYUSER/MYFS2 is ext2, ext3 or ext4 (because debugfs works only for these filesystems).
mydev2="$(df /media/MYUSER/MYFS2 | perl -ne '$x = $1 if !m#^Filesystem # and m#([^ ]+) #; END { print "$x\n" }')"
cd /media/MYUSER/MYFS
find FOO/BAR -type f | perl -ne 'chomp; my #st = lstat($_); if (#st and -f(_)) { s#"#""#g; print "set_inode_field \"/$_\" ctime \#$st[10]\n" }' >/tmp/sif.out
sudo umount /media/MYUSER/MYFS2 # Repeat until success.
sudo debugfs -w -f /tmp/sif.out /dev/"$mydev2"
It handles filenames with whitespace and special characters correctly.
It works independently of time zones. As a limitation of debugfs, its precision is seconds, it ignores anything smaller (e.g. milliseconds, microseconds, nanoseconds). Depending the version of debugfs used, it may use 32-bit timestamps, thus it works correctly with dates before 2038-01-19.
If the current user doesn't have enough read permissions for /media/MYUSER/MYFS, then the commands above should be run as root (sudo bash).