I am trying to find the virtual file that contains the current users id. I was told that I could find it in the proc directory, but not quite sure which file.
You actually want /proc/self/status, which will give you information about the currently executed process.
Here is an example:
$ cat /proc/self/status
Name: cat
State: R (running)
Tgid: 17618
Pid: 17618
PPid: 3083
TracerPid: 0
Uid: 500 500 500 500
Gid: 500 500 500 500
FDSize: 32
Groups: 10 488 500
VmPeak: 4792 kB
VmSize: 4792 kB
VmLck: 0 kB
VmHWM: 432 kB
VmRSS: 432 kB
VmData: 156 kB
VmStk: 84 kB
VmExe: 32 kB
VmLib: 1532 kB
VmPTE: 24 kB
Threads: 1
SigQ: 0/32268
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 00000003
Mems_allowed: 1
voluntary_ctxt_switches: 0
nonvoluntary_ctxt_switches: 3
You probably want to look at the first numbers on the Uid and Gid lines. You can look up which uid numbers map to what username by looking at /etc/passwd, or calling the relevant functions for mapping uid to username in whatever language you're using.
Ideally, you would just call the system call getuid() to look up this information, doing it by looking at /proc/ is counterproductive.
Why not just use "id -u"?
As far as I know, /proc is specific to Linux, it's not in UNIX in general. If you really just want the current UID, use the getuid() or geteuid() function.
If you know you'll be on Linux only, you can explore the hierarchy under /proc/self/*, it contains various information about the current process. Remember that /proc is "magical", it's a virtual filesystem the kernel serves and the contents is dynamically generated at the point you request it. Therefore it can return information specific for the current process.
For example, try this command: cat /proc/self/status
I'm not sure that can be found in /proc. You could try using the getuid() function or the $USER environment variable.
In /proc/process_id/status (at least on Linux) you'll find a line like this:
Uid: 1000 1000 1000 1000
This tells you the uid of the user under whose account the process is running.
However, to find out the process id of the current process you would need a system call, and then you might as well call getuid to get the uid directly.
Edit: ah, /proc/self/status... learning something new every day!
Most likely, you either want to check the $USER environment variable. Other options include getuid and id -u, but searching /proc is certainly not the best method of action.
The things you are looking for may be in environment variables. You need to be careful about what shell you are using when you check environment variables. bash uses "UID" while tcsh uses "uid" and in *nix case matters. I've also found that tcsh sets "gid" but I wasn't able to find a matching variable in bash.
Related
We develop a Linux application that uses DPDK and which must also be heavily optimised for speed.
We must specify huge pages for use by DPDK and also for general dynamic memory allocation. For the latter we use the libhugetlbfs library.
sudo mount -t hugetlbfs none /mnt/hugetlbfs
We specify the huge pages in the bootcmd line as follows:
hugepagesz=1G hugepages=20 hugepages=0 default_hugepagesz=1G
We are using Centos 7 and full boot cmdline is:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet hugepagesz=1G hugepages=20 default_hugepagesz=1G irqaffinity=0,1 isolcpus=4-23 nosoftlockup mce=ignore_ce idle=poll
These values are fairly arbitrary. With these values, I see:
$ free -m
total used free shared buff/cache available
Mem: 47797 45041 2468 9 287 2393
Swap: 23999 0 23999
So 2.468GB RAM is free out of 48GB. So a very large amount of memory is allocated to huge pages and I want to reduce this.
My question is what would be sensible values for them?
I am confused by the interpretation of the parameters. I see:
$ cat /proc/meminfo
<snip>
HugePages_Total: 43
HugePages_Free: 43
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
and also:
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
43
Why are 43 pages reported when my parameters only specify 20 pages of 1G?
I would like some guidelines on:
huge page size/quantity that I might need for DPDK?
huge page size/quantity that I might need for malloc?
I know these are highly application dependant but some guidelines would be helpful. Also how could I detect if the huge pages were insufficent for the application?
Additional info:
$ cat /proc/mounts | grep huge
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
Update 4 March:
My boot cmdline is now:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet hugepagesz=1G hugepages=20 default_hugepagesz=1G irqaffinity=0,1 isolcpus=4-23 nosoftlockup mce=ignore_ce idle=poll transparent_hugepage=never
and transparent hugepages are disabled (I activated a custom tune profile):
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
but I still see 43 hugepages:
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
43
whereas I have only specified 20 in cmdline. Why is this?
I used gdb to produce core dump file with gcore and used the dump in symbolic execution. However I found it did not include read-only segments in memory, for example, the loaded library libc.so.6. Unfortunately, I need it for symbolic execution.
Is there a way to make core dump also include those read-only segments?
EDIT:
I changed value in /proc//coredump_filter to 0x3c to allow:
" Dump file-backed shared mappings" according to core(5). However, gcore now gives dozens of warnings:
warning: Memory read failed for corefile section, 1048576 bytes at xxxxxx
,where xxxxxx is roughly the memory region where shared libraries (including libc, etc.) lies in the memory. However I want the core file to include shared libraries.
What is wrong?
ps: I can read from memory regions of shared libraries inside gdb normally with a plain gdb command x /16i.
EDIT:
I also checked /proc/<pid>/smaps, inside which, for example, memory region of libc is like:
7f65930f8000-7f65932df000 r-xp 00000000 08:01 789499 /lib/x86_64-linux-gnu/libc-2.27.so
Size: 1948 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 836 kB
Pss: 82 kB
Shared_Clean: 836 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 836 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd ex mr mw me sd
I presume the default value 0x33 in coredump_filter should be sufficient to dump this region.
Also as VM flags do not include dd (do not include area into core dump), it was not prevented from being written to core file, but why gcore did not dump it???
We are running a large MATLAB program. After about 24 hours of calculation it stops with an "Out of Memory" error.
Not long before this error occured the process status was quite healty, the process using about 3 Gb on a 12 Gb RAM machine with 32 Gb swap. We have no reason to believe that the process should behave much different, it was in iteration 3 of similar process where the first three iterations lead to a VmPeak of 2.7 Gb. The process was run without GUI using the -nojvm option, using MATLAB r2013a. The error occured in two different computers. Depending on exact parameterization the error occurs at different places in the program.
My question is: is there in MATLAB/Linux/64 bit another resource than plain RAM which can link to an Out of Memory message? Can it be memory fragmentation related, and how can we measure the current level of memory fragmentation?
Matlab error:
CATCH in vsProcessVideosetByFrame.Error using cat
Out of memory. Type HELP MEMORY for your options.
Error in ttClassDetections/horzcat (line 170)
output.data.(fn{j}) = cat(2,temp{:});
Process status not long before it crashed with an Out of Memory message:
schuttek#pc-07843:~$ cat /proc/14123/status
Name: MATLAB
State: S (sleeping)
Tgid: 14123
Ngid: 0
Pid: 14123
PPid: 1
TracerPid: 0
Uid: 8793 8793 8793 8793
Gid: 100 100 100 100
FDSize: 256
Groups: 4 24 27 30 46 100 108 124 501
NStgid: 14123
NSpid: 14123
NSpgid: 14123
NSsid: 11423
VmPeak: 2744640 kB
VmSize: 2613568 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1045376 kB
VmRSS: 1025692 kB
VmData: 2095720 kB
VmStk: 132 kB
VmExe: 12 kB
VmLib: 179980 kB
VmPTE: 2920 kB
VmPMD: 24 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 18
SigQ: 1/64005
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000008080006
SigIgn: 0000000000000001
SigCgt: 00000001880804ee
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Cpus_allowed: ffffffff
Cpus_allowed_list: 0-31
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 37
nonvoluntary_ctxt_switches: 5
Since this is probably not the OOM killer (as dmesg | grep kill is empty), the error should be internal to MATLAB.
One possible cause is indeed the memory fragmentation; at least some MATLAB routines seem to request explicitly continuous areas of memory that you may run out of. That may be alleviated with pack - a MATLAB function that reconciles the memory.
Here's the documentation page. Make sure to heed the warning about variables larger than 2GB!
https://www.mathworks.com/help/matlab/ref/pack.html
So far running on a Windows 10 64 bit system seems to solve our problem.
I have a program that creates a shared memory object with shm_open that I'm working on. It tries to release the object with shm_unlink, but sometimes a programming error will cause it to crash before it can make that call. In that case, I need to unlink the shared memory object "by hand," and I'd like to be able to do it in a normal shell, without writing any C code -- i.e. with normal Linux utilities.
Can it be done? This question seems to say that using unlink(1) on /dev/shm/path_passed_to_shm_open, but the manpages aren't clear.
Unlinking the file in /dev/shm will delete the shared memory object if no other process has the object mapped.
Unlike SYSV shared memory, which is implemented in the Kernel, POSIX shared memory objects are simply "files in disguise".
When you call shm_open and mmap, you can see the following in the process process map (using pmap -X):
Address Perm Offset Device Inode Mapping
b7737000 r--s 00000000 00:0d 2267945 test_object
The device major and minor number correspond to the tmpfs mounted at /dev/shm (some systems mount this at /run, and then symlink /dev/shm to /run/shm).
A listing of the folder will show the same inode number:
$ ls -li /dev/shm/
2267945 -rw------- 1 mikel mikel 1 Apr 14 13:36 test_object
Like any other inode, the space will be freed when all references are removed. If we close the only program referencing this we see:
$ cat /proc/meminfo | grep Shmem
Shmem: 700 kB
Once we remove the last reference (created in /dev/shm), the space will be freed:
$ rm /dev/shm/test_object
$ cat /proc/meminfo | grep Shmem
Shmem: 696 kB
If you're curious, you can look at the corresponding files in the glibc sources. shm_directory.c and shm_directory.h generate the filename as /dev/your_shm_name. The implementations shm_open and shm_unlink simply open and unlink this file. So it should be easy to see that rm /dev/shm/your_shm_name performs the same operation,
Suppose a binary executable program is running:
For example:
ps -eaf | grep someServer
shows that someServer is running.
Is it possible to tell if someServer executable on disk ( eg /usr/bin/someServer ) matches the program that was actually started ?
Yes: Use soft link /proc/$pid/exe to get the path which was used to load the code.
Look into /proc/$pid/maps. It will look like this (for /sbin/getty):
00400000-00407000 r-xp 00000000 08:01 3145779 /sbin/getty
00606000-00607000 r--p 00006000 08:01 3145779 /sbin/getty
00607000-00608000 rw-p 00007000 08:01 3145779 /sbin/getty
... lots more ...
Filter the file using the path that you got from the soft link to find the lines that are interesting for you.
The last number (3145779) is the inode of the file. When you create a new file on disk, it gets a new inode.
To see the inode of a file, use ls --inode /sbin/getty:
3145779 /sbin/getty
Since the two numbers are still identical, the executable on disk is the same as in RAM.
Background: Linux doesn't load processes into RAM at once. Instead, the executable file is memory-mapped into RAM using the virtual memory subsystem. This means parts of the executable which you never use will never be loaded into memory. It also means that the kernel uses the executable on disk as a "cache".
When you overwrite the executable on disk, the original inode is not changed. Your existing process hangs on to it. Instead, a new inode is created and the directory node (which contains the file name and a pointer to the inode with the data) is updated. This is why you can overwrite files that are currently in use on Linux.
The original inode will be cleaned up when the last process which uses it dies.
I am not sure what do you mean exactly.
use pid matching (compare ps output and getpid() in the server)
use /proc/$pid/environ and find _=path_to_someServer (there is the binary)
use diff /proc/$pid/exe someServer ("type someServer" to get full path)
If you want to know if any running processes were spawned from a particular executable inode:
# find -L /proc/[0-9]*/exe -samefile /usr/bin/someServer
This command will output a list of /proc/<pid>/exe pathnames, one for each process whose executable image was mapped into memory from the same inode as is presently linked at /usr/bin/someServer.
Note that the command will NOT find processes that were spawned from inodes that were formerly linked at /usr/bin/someServer but have since been unlinked, such as if a newer version of the executable has replaced a running version.