We have a legacy device with following configurations:
Chipset Architecture : Intel NM10 express
OS : Yocto warrior
CPU : Atom D2250 Dual Core
Volatile Memory : 2GB DDR3
CPU core : 4
After installing yocto sato image on device, we are seeing sluggish behavior. We are unsure whether it is because of hardware limitations or anything else.
If this slow behavior is because of yocto sato image then,
Is there something which can be done to improve performance?
Additional information:
Output(only first core) of cat /proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 54
model name : Intel(R) Atom(TM) CPU D2550 # 1.86GHz
stepping : 1
microcode : 0x10d
cpu MHz : 1861.758
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fdiv_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm arat
bugs :
bogomips : 3723.54
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Output of cat /proc/meminfo:
MemTotal: 917184 kB
MemFree: 700044 kB
MemAvailable: 761264 kB
Buffers: 5928 kB
Cached: 74112 kB
SwapCached: 0 kB
Active: 129856 kB
Inactive: 44956 kB
Active(anon): 95096 kB
Inactive(anon): 5720 kB
Active(file): 34760 kB
Inactive(file): 39236 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 102980 kB
HighFree: 220 kB
LowTotal: 814204 kB
LowFree: 699824 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 16 kB
Writeback: 0 kB
AnonPages: 94816 kB
Mapped: 42624 kB
Shmem: 6048 kB
Slab: 17692 kB
SReclaimable: 9124 kB
SUnreclaim: 8568 kB
KernelStack: 976 kB
PageTables: 1676 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 458592 kB
Committed_AS: 348404 kB
VmallocTotal: 122880 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 528 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 22520 kB
DirectMap2M: 884736 kB
Output of cat /proc/uptime:
00:44:08 up 29 min, load average: 0.51, 0.78, 0.67
Edit1 : lubuntu is running as expected i.e. No sluggish behavior is seen on device
Edit2 : Graphics gets faster after removing libglamoregl.so. What is there in this library which is causing graphics performance issue.
Related
I use Android P-OS. and kernel version is msm-4.14
oom invoked and killing process since booting up. However, memory is abundant.
My memory size is 8GByte, Swap is 1GByte.
I'm not even using a swap.
[ 59.901334] Killing 'ndroid.keychain' (2011), adj 906,\x0a to free 87268kB on behalf of 'Binder:883_2' (938)\x0a Free CMA is 246200kB\x0a Total reserve is 242332kB\x0a Total free pages is 5100764kB\x0a Total file cache is 978224kB
[ 59.903948] Killing 'Jit thread pool' (2016), adj 906,\x0a to free 88676kB on behalf of 'ActivityManager' (960)\x0a Free CMA is 246200kB\x0a Total reserve is 242332kB\x0a Total free pages is 5100764kB\x0a Total file cache is 978224kB
[ 60.007328] oom_reaper: reaped process 2011 (ndroid.keychain), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
$ free
total used free shared buffers
Mem: 7842377728 3144630272 4697747456 2084864 14852096
-/+ buffers/cache: 3129778176 4712599552
Swap: 1073737728 0 1073737728
$ meminfo
MemTotal: 7658572 kB
MemFree: 4589120 kB
MemAvailable: 5800580 kB
Buffers: 14416 kB
Cached: 1415944 kB
SwapCached: 0 kB
Active: 630232 kB
Inactive: 1299508 kB
Active(anon): 501820 kB
Inactive(anon): 1876 kB
Active(file): 128412 kB
Inactive(file): 1297632 kB
Unevictable: 2888 kB
Mlocked: 2888 kB
SwapTotal: 1048572 kB
SwapFree: 1048572 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 502148 kB
Mapped: 745728 kB
Shmem: 2036 kB
Slab: 520776 kB
SReclaimable: 130688 kB
SUnreclaim: 390088 kB
KernelStack: 28336 kB
PageTables: 40972 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4877856 kB
Committed_AS: 95986336 kB
VmallocTotal: 263061440 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
CmaTotal: 303104 kB
CmaFree: 246200 kB
I don't understand this situation.
Why does this happen? Is there a way to avoid it?
Answer Myself
It is not Out Of Memory.
Just oom_reaper is working by sigkill signal.
Memory is enough
Thanks
I have inherited a server with some performance issues. It is running node js, nginx, basic MEAN stack. (DB on another server though)
Whenever I copied a file (log file with size of around 150MB) or vim a file with that size, the output of "iostat -x 1" will be like below
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 8137.62 0.00 49.50 0.00 29924.75 604.48 17.32 123.54 16.50 81.68
avg-cpu: %user %nice %system %iowait %steal %idle
1.59 0.00 24.34 0.00 0.00 74.07
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 39.39 0.00 36606.06 929.23 2.42 351.64 1.87 7.37
avg-cpu: %user %nice %system %iowait %steal %idle
2.78 0.00 24.44 0.00 0.00 72.78
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The main reason I am bringing this up is because sometimes a simple RESTFUL API that my nodejs is serving will respond slowly (from 10ms to 500ms) and I am not sure what to look for or check what is causing this.
The same codebase copied to another server will run smoothly without issues, the problem mentioned above is the only lead I can find that there might be something wrong with the server but I am not sure what is it.
The codes are like below:
In one file statistic/index.js:
var tracker = function (data) {
piwik.tracker(data);
};
exports.tracker = tracker;
In another file statistic/piwik.js:
exports.tracker = function (data) {
var params = {};
/** Assign params with data - just static string **/
/** API_URL is another machine in same LAN **/
needle.post(API_URL, params, function (err, response, body) {
if (err || (response.statusCode !== 200 && response.statusCode !== 204)) {
util.error(err);
}
});
};
In the file that is calling the above route/getuser.js:
exports.getUser = function (req, res) {
async.auto({
get_user: function (cb) {
/** Read user data from DB **/
cb();
},
record_statistic: ['get_user', function (cb) {
statistic.tracker({ /** Pass static string data **/});
cb();
}]
}, function (err) {
if (err) {
res.json(err);
} else {
res.json();
}
});
};
The reason I am mentioning the above codes is because when I remarked out statistic.tracker({ /** Pass static string data **/}); The function will response within 50ms, but if I include it most of the time it will respond between 100ms - 500ms. I have even put a timestamp check for the "needle" HTTP post, and it respond within 10 - 20ms.
When I copy a file (cp -p x.txt y.txt) especially when it is a > 100MB file, it will also slow down my node js. But even when I am not doing anything on the server besides nginx and node js listening for request the codes below will respond slowly. (If I didn't remark out the code)
I am suspecting IO but where else to check? or what to look out for?
Below are some info about the server:
[ec2-user#tlp-backend logs]$ uname -a
Linux tlp-backend 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user#tlp-backend logs]$ cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: VMware, Model: VMware Virtual S Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
[ec2-user#tlp-backend logs]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30502
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[ec2-user#tlp-backend logs]$ more /proc/meminfo
MemTotal: 3918960 kB
MemFree: 392260 kB
Buffers: 296116 kB
Cached: 1205652 kB
SwapCached: 364 kB
Active: 1725084 kB
Inactive: 1155564 kB
Active(anon): 949492 kB
Inactive(anon): 430528 kB
Active(file): 775592 kB
Inactive(file): 725036 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4095992 kB
SwapFree: 4092872 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 1378528 kB
Mapped: 29860 kB
Shmem: 1140 kB
Slab: 588628 kB
SReclaimable: 461108 kB
SUnreclaim: 127520 kB
KernelStack: 2296 kB
PageTables: 16940 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6055472 kB
Committed_AS: 1829320 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 288456 kB
VmallocChunk: 34359446140 kB
HardwareCorrupted: 0 kB
AnonHugePages: 507904 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10240 kB
DirectMap2M: 4184064 kB
[ec2-user#tlp-backend logs]$ more /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 2
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
stepping : 2
cpu MHz : 2400.000
cache size : 12288 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch
_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_
spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lah
f_lm ida arat
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
top - 18:02:58 up 26 days, 18:49, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 176 total, 1 running, 175 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3918960k total, 3522368k used, 396592k free, 295340k buffers
Swap: 4095992k total, 3120k used, 4092872k free, 1201660k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 19340 1248 1040 S 0.0 0.0 0:03.45 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:02.55 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:01.57 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root RT 0 0 0 0 S 0.0 0.0 0:18.87 migration/1
7 root 20 0 0 0 0 S 0.0 0.0 0:01.34 ksoftirqd/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
9 root RT 0 0 0 0 S 0.0 0.0 0:01.90 migration/2
10 root 20 0 0 0 0 S 0.0 0.0 0:01.30 ksoftirqd/2
11 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
Apparently, the main problem is coming from the code itself. When you are mixing async.auto with needle package, you need to explicitly stating { connection: 'keep-alive' } in HTTP header
See more info here: https://github.com/tomas/needle/issues/148
I am mmapping a block as:
mapAddr = mmap((void*) 0, curMapSize, PROT_NONE, MAP_LOCKED|MAP_SHARED, fd, curMapOffset);
if this does not fail (mapAddr != MAP_FAILED) I query mincore as:
err = mincore((char*) mapAddr, pageSize, &mincoreRet);
to find out whether it is in RAM. In case it is in RAM (err == 0 && mincoreRet & 0x01) I mmap it again for reading as:
copyAddr = mmap((void*) 0, curMapSize, PROT_READ, MAP_LOCKED|MAP_SHARED, fd, curMapOffset);
and then I try to copy it out to my buffer as:
memcpy(data, copyAddr, pageSize);
everything works fine except in the last memcpy once in a while I get SIGBUS. When I check /proc/ /smaps at the time of the failure I notice that it has Rss as well as Locked fields as 0 as listed below:
7f4a4c118000-7f4a4c119000 r--s 00326000 00:17 6 <file name>
Size: 4 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
Any thoughts? This is happening on ubuntu 12.0.4 with kernel version 3.5.0-36.
I have a little question regarding VmSize, in the documentation it's supposed to be the application's usage of memory.
However on my system:
VmSize = physical memory + swap
VmHWM seems more like what the application actually would be using.
[root#sun ~]# free -m
total used free shared buffers cached
Mem: 12012 9223 2788 0 613 1175
-/+ buffers/cache: 7434 4577
Swap: 3967 0 3967
[root#sun ~]# cat /proc/8268/status
Name: mysqld
State: S (sleeping)
Tgid: 8268
Pid: 8268
PPid: 1
TracerPid: 0
Uid: 89 89 89 89
Gid: 89 89 89 89
FDSize: 512
Groups: 89
VmPeak: 15878128 kB
VmSize: 15878128 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 7036312 kB
VmRSS: 7036312 kB
VmData: 15839272 kB
VmStk: 136 kB
VmExe: 10744 kB
VmLib: 6356 kB
VmPTE: 16208 kB
VmSwap: 0 kB
Threads: 265
SigQ: 0/96048
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000087007
SigIgn: 0000000000001000
SigCgt: 00000001800066e9
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: fff
Cpus_allowed_list: 0-11
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 2567
nonvoluntary_ctxt_switches: 77
Any idea of why?
I try to get the usage of memory for this application in particular but this result doesn't really make sense.
Thanks.
VMsize is the "address space" that the process has in use: the number of available adresses. These addresses do not have to have any physical memory attached to them. (Attached physical memory is the RSS figure)
You can verify this by allocating a chunk of memory with p = malloc(4 * 1024 * 1024);, and not doing anything to *p: the VmSize will increase by 1K pages, but the RSS will be (about) the same. (your program will have more adressable memory, but it does not address it, so the memory does not need to be attached )
VmSize is the sum of all mapped memory (/proc/pid/maps)
I was confused about the Pss column in /proc/pid/smaps, so I wrote a program to test it:
void sa();
int main(int argc,char *argv[])
{
int fd;
sa();
sleep(1000);
}
void sa()
{
char *pi=new char[1024*1024*10];
for(int i=0;i<4;++i) {
for(int j=0;j<1024*1024;++j){
*pi='o';
pi++;
}
}
int cnt;
for(int i=0;i<6;++i) {
for(int j=0;j<1024*1024;++j){
cnt+=*pi;
pi++;
}
}
printf("%d",cnt);
}
$cat /proc/`pidof testprogram`/smaps
08838000-0885b000 rw-p 00000000 00:00 0 [heap]
Size: 140 kB
Rss: 12 kB
Pss: 12 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 12 kB
Referenced: 12 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
b6dcd000-b77d0000 rw-p 00000000 00:00 0
Size: 10252 kB
Rss: 10252 kB
Pss: 4108 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4108 kB
Referenced: 4108 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Here I found Pss equal to Private_Dirty, but I wonder why.
BTW: Is there any detailed documentation for smaps?
Quoting from lwn.net
The "proportional set size" (PSS) of a process is the count of pages
it has in memory, where each page is divided by the number of
processes sharing it. So if a process has 1000 pages all to itself,
and 1000 shared with one other process, its PSS will be 1500
From Linux Kernel Documentation,
The /proc/PID/smaps is an extension based on maps, showing the memory
consumption for each of the process's mappings. For each of mappings there
is a series of lines such as the following:
08048000-080bc000 r-xp 00000000 03:02 13130 /bin/bash
Size: 1084 kB
Rss: 892 kB
Pss: 374 kB
Shared_Clean: 892 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 892 kB
Anonymous: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 374 kB
The first of these lines shows the same information as is displayed
for the mapping in /proc/PID/maps. The remaining lines show the size
of the mapping (size), the amount of the mapping that is currently
resident in RAM (RSS), the process' proportional share of this mapping
(PSS), the number of clean and dirty private pages in the mapping.
Note that even a page which is part of a MAP_SHARED mapping, but has
only a single pte mapped, i.e. is currently used by only one process,
is accounted as private and not as shared. "Referenced" indicates the
amount of memory currently marked as referenced or accessed.
"Anonymous" shows the amount of memory that does not belong to any
file. Even a mapping associated with a file may contain anonymous
pages: when MAP_PRIVATE and a page is modified, the file page is
replaced by a private anonymous copy. "Swap" shows how much
would-be-anonymous memory is also used, but out on swap.
This Question on Unix and Linux Stackexchange covers almost the topic. See Mat's excellent answer which will surely clear all your doubts.