For these two days, I have met a weird question.
STAR from https://github.com/alexdobin/STAR is a program used to build suffix array indexes. I have been used this program for years. It works OK until recently.
For these days, when I run STAR, it will always be killed.
root#localhost:STAR --runMode genomeGenerate --runThreadN 10 --limitGenomeGenerateRAM 31800833920 --genomeDir star_GRCh38 --genomeFastaFiles GRCh38.fa --sjdbGTFfile GRCh38.gtf --sjdbOverhang 100
.
.
.
Killed
root#localhost:STAR --runMode genomeGenerate --runThreadN 10 --genomeDir star_GRCh38 --genomeFastaFiles GRCh38.fa --sjdbGTFfile GRCh38.gtf --sjdbOverhang 100
Jun 03 10:15:08 ..... started STAR run
Jun 03 10:15:08 ... starting to generate Genome files
Jun 03 10:17:24 ... starting to sort Suffix Array. This may take a long time...
Jun 03 10:17:51 ... sorting Suffix Array chunks and saving them to disk...
Killed
A month ago, the same command with same inputs and same parameters runs OK. It does cost some memory, but not a lot.
I have tried 3 recently released version of this program, all failed. So I do not think it is the question of STAR program but my sever configuration.
I also tried to run this program as both root and normal user, no lucky for each.
I suspect there is a limitation of memory usage in my server.
But I do not know how the memory is limited? I wonder if some one can give me some hints.
Thanks!
Tong
The following is my debug process and system info.
Command dmesg -T| grep -E -i -B5 'killed process' showing it is Out of memory problem.
But before the STAR program is killed, top command showing only 5% mem is occupied by this porgram.
[一 6 1 23:43:00 2020] [40479] 1002 40479 101523 18680 112 487 0 /anaconda2/bin/
[一 6 1 23:43:00 2020] [40480] 1002 40480 101526 18681 112 486 0 /anaconda2/bin/
[一 6 1 23:43:00 2020] [40481] 1002 40481 101529 18682 112 485 0 /anaconda2/bin/
[一 6 1 23:43:00 2020] [40482] 1002 40482 101531 18673 111 493 0 /anaconda2/bin/
[一 6 1 23:43:00 2020] Out of memory: Kill process 33822 (STAR) score 36 or sacrifice child
[一 6 1 23:43:00 2020] Killed process 33822 (STAR) total-vm:23885188kB, anon-rss:10895128kB, file-rss:4kB, shmem-rss:0kB
[三 6 3 10:02:13 2020] [12296] 1002 12296 101652 18681 113 486 0 /anaconda2/bin/
[三 6 3 10:02:13 2020] [12330] 1002 12330 101679 18855 112 486 0 /anaconda2/bin/
[三 6 3 10:02:13 2020] [12335] 1002 12335 101688 18682 112 486 0 /anaconda2/bin/
[三 6 3 10:02:13 2020] [12365] 1349 12365 30067 1262 11 0 0 bash
[三 6 3 10:02:13 2020] Out of memory: Kill process 7713 (STAR) score 40 or sacrifice child
[三 6 3 10:02:13 2020] Killed process 7713 (STAR) total-vm:19751792kB, anon-rss:12392428kB, file-rss:0kB, shmem-rss:0kB
--
[三 6月 3 10:42:17 2020] [ 4697] 1002 4697 101526 18681 112 486 0 /anaconda2/bin/
[三 6月 3 10:42:17 2020] [ 4698] 1002 4698 101529 18682 112 485 0 /anaconda2/bin/
[三 6月 3 10:42:17 2020] [ 4699] 1002 4699 101532 18680 112 487 0 /anaconda2/bin/
[三 6月 3 10:42:17 2020] [ 4701] 1002 4701 101534 18673 110 493 0 /anaconda2/bin/
[三 6月 3 10:42:17 2020] Out of memory: Kill process 21097 (STAR) score 38 or sacrifice child
[三 6月 3 10:42:17 2020] Killed process 21097 (STAR) total-vm:19769500kB, anon-rss:11622928kB, file-rss:884kB, shmem-rss:0kB
Command free -hl showing I have enough memory.
total used free shared buff/cache available
Mem: 251G 10G 11G 227G 229G 12G
Low: 251G 240G 11G
High: 0B 0B 0B
Swap: 29G 29G 0B
Also as showed by ulimit -a, no virtual memory limitation is set.
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1030545
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1030545
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Here is the version of my Centos and Kernel (output by hostnamectl):
hostnamectl
Static hostname: localhost.localdomain
Icon name: computer-server
Chassis: server
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
Architecture: x86-64
Here shows the content of cat /etc/security/limits.conf:
#* soft core 0
#* hard rss 10000
##student hard nproc 20
##faculty soft nproc 20
##faculty hard nproc 50
#ftp hard nproc 0
##student - maxlogins 4
* soft nofile 65536
* hard nofile 65536
##intern hard as 162400000
##intern hard nproc 150
# End of file
As suggested, I have updated the output of df -h:
Filesystem All Used Available (Used)% Mount
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 1.3M 126G 1% /dev/shm
tmpfs 126G 4.0G 122G 4% /run
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/mapper/cl-root 528G 271G 257G 52% /
/dev/sda1 492M 246M 246M 51% /boot
tmpfs 26G 0 26G 0% /run/user/0
tmpfs 26G 0 26G 0% /run/user/1002
tmpfs 26G 0 26G 0% /run/user/1349
tmpfs 26G 0 26G 0% /run/user/1855
ls -a /dev/shm/
. ..
grep Shmem /proc/meminfo
Shmem: 238640272 kB
Several tmpfs costs 126G memory. I am googleing this, but still not sure what should be done?
That is the problem of shared memory due to program terminated abnormally.
ipcrm is used to clear all shared memory and then STAR running is fine.
$ ipcrm
.....
$ free -h
total used free shared buff/cache available
Mem: 251G 11G 226G 3.9G 14G 235G
Swap: 29G 382M 29G
It looks like the problem is with shared memory: you have 227G of memory eaten up by shared objects.
Shared memory files are persistent. Have a look in /dev/shm and any other tmpfs mounts to see if there are large files that can be removed to free up more physical memory (RAM+swap).
$ ls -l /dev/shm
...
$ df -h | grep '^Filesystem\|^tmpfs'
...
When I run a program called STAR, it will always be killed.
It probably has some memory leak. Even old programs may have residual bugs, and they could appear in some very particular cases.
Check with strace(1) or ltrace(1) and pmap(1). Learn also to query /proc/, see proc(5), top(1), htop(1). See LinuxAteMyRam and read about memory over-commitment and virtual address space and perhaps a textbook on operating systems.
If you have access to the source code of your STAR, consider recompiling it with all warnings and debug info (with GCC, you would pass -Wall -Wextra -g to gcc or g++) then use valgrind and/or some address sanitizer. If you don't have legal access to the source code of STAR, contact the entity (person or organization) which provided it to you.
You could be interested in that draft report and by the Clang static analyzer or Frama-C (or coding your own GCC plugin).
So I do not think it is the question of STAR program but my server configuration.
I recommend using valgrind or gdb and inspect your /proc/ to validate that optimistic hypothesis.
Related
According to:
$ ls -l /sys/kernel/mm/hugepages
drwxr-xr-x 2 root root 0 Dec 6 10:38 hugepages-1048576kB
drwxr-xr-x 2 root root 0 Dec 6 10:38 hugepages-2048kB
There is a choice of 2 MB and 1 GB sizes of huge pages on my system which is running a 5.4.17 kernel
However according to:
$ cpuid | grep -i tlb |sort| uniq
0x03: data TLB: 4K pages, 4-way, 64 entries
0x63: data TLB: 2M/4M pages, 4-way, 32 entries
0x76: instruction TLB: 2M/4M pages, fully, 8 entries
0xb5: instruction TLB: 4K, 8-way, 64 entries
0xc3: L2 TLB: 4K/2M pages, 6-way, 1536 entries
cache and TLB information (2):
data TLB: 1G pages, 4-way, 4 entries
L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
the TLBs on my Skylake also support 4 MB pages. The same information can be found at
https://en.wikichip.org/wiki/intel/microarchitectures/skylake_(server)
So the question is: can I really have 4 MB pages, and if so what do I need to do to set up my system to have that option?
The best answer is probably to install and/or use libhugetlbfs
If it's already installed, you can check status of huge pages in the OS with a command like:
$ hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 0 1 257388 *
4194304 0 0 128694
8388608 0 0 64347
16777216 0 0 32173
33554432 0 0 16086
67108864 0 0 8043
134217728 0 0 4021
268435456 0 0 2010
536870912 0 0 1005
1073741824 0 0 502
2147483648 0 0 251
The same hugeadm command can also be run as sudo with various options to configure the available huge memory pools. See the hugeadm man page for details.
*docker system prune doesn't work!!!!
i keep getting the following error:
ERROR: write /var/lib/docker/tmp/GetImageBlob892028471: no space left on device
my docker info:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 119K 308 119K 1% /dev
tmpfs 123K 505 122K 1% /run
/dev/xvda1 1000K 137K 864K 14% /
tmpfs 123K 1 123K 1% /dev/shm
tmpfs 123K 3 123K 1% /run/lock
tmpfs 123K 18 123K 1% /sys/fs/cgroup
/dev/loop0 474 474 0 100% /snap/snapd/11588
/dev/loop1 16 16 0 100% /snap/amazon-ssm-agent/3552
/dev/loop2 11K 11K 0 100% /snap/core18/1997
/dev/loop3 474 474 0 100% /snap/snapd/12704
/dev/loop4 11K 11K 0 100% /snap/core18/2074
/dev/loop5 16 16 0 100% /snap/amazon-ssm-agent/4046
tmpfs 123K 11 123K 1% /run/user/1000
I wanted to change the path of docker-storage as it was recommended somewhere but the path /etc/sysconfig/ doesn't exist and that's where the file is meant to me. Could also be that I have taken up too much space in the ubuntu machine and need help clearing it out.
It can happen that a process that was not killed gracefully is holding onto a large amount of disk space and it is not reported in df a reboot will force the process to let go of the disk space allowing you to utilise the space again.
I ended up creating a very heavy-handed function for this; note that it will burninate all your containers and their vnets!
docker_cleanup() {
set -x
echo "pruning containers"
docker container prune -f
# https://stackoverflow.com/a/42371013/
docker system prune -f
set +x
}
I am using Yocto and it's wic tool to build my embedded Linux image.
The wic configuration file looks like this:
part /boot --source bootimg-partition --ondisk mmcblk --fstype=msdos --label boot --align 1024 --fixed-size 64
part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root_a --fixed-size 256 --active
part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root_b --fixed-size 256
part /permanent-storage --ondisk mmcblk --fstype=ext4 --label permanent-storage --fixed-size 300
part swap --ondisk mmcblk --size 64 --label swap --fstype=swap
I burn the resulting image to my SD Card and boot successfully, and there is an unexpected small ( 1K ) partition:
root#eval:/dev# ls -lrt /dev/mmcblk0*
brw-rw---- 1 root disk 179, 0 Feb 27 21:55 /dev/mmcblk0
brw-rw---- 1 root disk 179, 4 Feb 27 21:55 /dev/mmcblk0p4
brw-rw---- 1 root disk 179, 2 Feb 27 21:55 /dev/mmcblk0p2
brw-rw---- 1 root disk 179, 3 Feb 27 21:55 /dev/mmcblk0p3
brw-rw---- 1 root disk 179, 5 Feb 27 21:55 /dev/mmcblk0p5
brw-rw---- 1 root disk 179, 1 Feb 27 21:55 /dev/mmcblk0p1
brw-rw---- 1 root disk 179, 6 Feb 27 21:55 /dev/mmcblk0p6
root#eval:/dev# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 59.6G 0 disk
|-mmcblk0p1 179:1 0 64M 0 part
|-mmcblk0p2 179:2 0 256M 0 part /
|-mmcblk0p3 179:3 0 256M 0 part
|-mmcblk0p4 179:4 0 1K 0 part
|-mmcblk0p5 179:5 0 300M 0 part
`-mmcblk0p6 179:6 0 64M 0 part
Why is wic creating this partition and how can I get rid of it with my wic file? Thanks.
The mmcblk0p4 (1K) partition is an extended partition. When using a master boot record (MBR) to partition storage into more than 4 partitions one must use 3 primary partitions and 1 extended partition. This is because there is a maximum of 4 primary partitions. The extended partition may hold multiple logical partitions.
mmcblk0 <- Entire Storage
|--mmcblk0p1 <- Primary Partition 1
|--mmcblk0p2 <- Primary Partition 2
|--mmcblk0p3 <- Primary Partition 3
|--mmcblk0p4 <- Extended Partition
|--mmcblk0p5 <- Logical Partition 1
|--mmcblk0p6 <- Logical Partition 2
This is not Yocto specific. I use Buildroot and have a similar layout. The commonality is the disk partition method not the Linux distribution.
Wikipedia: Disk Partitioning
Serverfault: Primary vs extended partitions on Linux
I'm running Ubuntu Trusty 14.04 on a new machine with 8GB of RAM, and it seems to be locking up periodically and nothing is in syslog file. I've installed Nagios and have been watching the graphs, and it looks like memory is going high from 7% to 72% in just a span of 10 mins. Only node process are running on server. In top I found all process are running very normal memory consumption. Even after stopping node process. Memory remains with same utilization.
free agrees, claiming I'm using more than 5.7G of memory:
free -h
total used free shared buffers cached
Mem: 7.8G 6.5G 1.3G 2.2M 233M 612M
-/+ buffers/cache: 5.7G 2.1G
Swap: 2.0G 0B 2.0G
This other formula for totaling the memory roughly agrees:
# ps -e -orss=,args= | sort -b -k1,1n | awk '{total = total + $1}END{print total}'
503612
If the processes only total 500 MiB, where's the rest of the memory going?
I've got solution on this... so just wanna to update the same...
echo 2 > /proc/sys/vm/drop_caches
This resolved my issue. So I have added the same in my cron for every 5 mins on each of ubuntu server
I was checking my server resource usage and noticed that the "cma" process is using a lot of RAM.
top - 15:04:54 up 127 days, 21:00, 1 user, load average: 0.27, 0.33, 0.24
Tasks: 157 total, 1 running, 156 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.9%us, 0.3%sy, 0.0%ni, 92.6%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4043700k total, 4006616k used, 37084k free, 146968k buffers
Swap: 1052248k total, 1052240k used, 8k free, 1351364k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4308 root 16 0 2080m 977m 4708 S 0.0 24.8 0:00.02 cma
4396 root 15 0 2080m 977m 4708 S 0.0 24.8 0:00.10 cma
4397 root 16 0 2080m 977m 4708 S 0.0 24.8 3:47.36 cma
4398 root 15 0 2080m 977m 4708 S 0.0 24.8 2:31.40 cma
4399 root 15 0 2080m 977m 4708 S 0.0 24.8 0:00.34 cma
4400 root 18 0 2080m 977m 4708 S 0.0 24.8 0:00.00 cma
4403 root 15 0 2080m 977m 4708 S 0.0 24.8 0:47.36 cma
4404 root 18 0 2080m 977m 4708 S 0.0 24.8 0:00.07 cma
4405 root 18 0 2080m 977m 4708 S 0.0 24.8 0:00.04 cma
4406 root 15 0 2080m 977m 4708 S 0.0 24.8 0:12.14 cma
4408 root 19 0 2080m 977m 4708 S 0.0 24.8 0:00.00 cma
I found this forum post from last year and apparently these processes have to do with McAfee virus scanning.
I ran pmap on one of the processes and this is the last line of output:
mapped: 2130892K writeable/private: 2113632K shared: 40K
Is this process really using 2.1GB of memory? Is Top reporting the memory usage accurately>
Thanks!
The VIRT column tells you the total size of the virtual memory segments mapped into the process - this includes the executable itself, libraries, data segments, stack, heap, memory mapped files, etc. In a sense, it is the total amount of memory that the process currently has permission to touch in one way or another (read, write, execute). The process is not necessarily using all of that, which is one of several reasons that the RES column reports a smaller number. RES is the total size of the subset of the VIRT size that is actually currently in physical memory at the moment. It is a better (but still not great) measure of how much memory the process is actually using - the fact that it is in memory indicates that it has been or is currently being actively used. However, if your system has lots of memory, a portion of that RES number may have been touched 3 days ago, and not since, so it may not be actively in use. Conversely, if you are short on memory, the process may be trying to actively use more than RES currently indicates, which will result in paging/swapping activity and performance issues.
Then there's the tendency for some types of memory (executables, libraries) to be shared between multiple instances of a program, the existence of IPC-type shared memory, and several other things that all factor into "how much memory is this process using?"...
In other words, it's not as simple a question as you might imagine...