NUMA support on which CPU? What are the current server configuration of this kind of CPU? Linux NUMA commands regarding what, how to open NUMA?
This is going to depend of your server, if it's using a multicore cpu that support Numa affinity. Type numactl --hardware and you'll check how it's the current configuration, for example:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 32733 MB
node 0 free: 4027 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 32767 MB
node 1 free: 20898 MB
node distances:
node 0 1
0: 10 21
1: 21 10
If you want to check performance with your application, just make sure that it's using the CPUs from the same numa node. You can check this using ps -aux ortop commands.
Related
I am getting this error in my debug file.If anybody knows this error please help to solve it. Frustated with this error
ERROR [epollEventLoopGroup-2-51] 2017-11-09 16:09:21,495 Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.
I am using G1 garbage collector instead of CMS collector
I have 4 servers
x.x.x.1 contains-------------------------------------------
MAX_HEAP_SIZE="8G"
HEAP_NEWSIZE="2000M"
OS: CentOS - 7
RAM: 142 GB
Swap: 3 GB
Processor: Intel(R) Xeon(R) CPU E5-2630 v4 # 2.20GHz
Core: 40 Core
Disk: 2.5T
x.x.x.2 contains--------------------------------------------
MAX_HEAP_SIZE="16G"
HEAP_NEWSIZE="4000M"
OS: CentOS - 7
RAM: 125 GB
Swap: 3 GB
Processor: Intel(R) Xeon(R) CPU E5-2630 v4 # 2.20GHz
Core: 40 Core
Disk: 2.2T
x.x.x.3 contains---------------------------------------------
MAX_HEAP_SIZE="16G"
HEAP_NEWSIZE="4000M"
OS: CentOS - 7
RAM: 125 GB
Swap: 3 GB
Processor: Intel(R) Xeon(R) CPU E5-2630 v4 # 2.20GHz
Core: 40 Core
Disk: 2 TB
x.x.x.4 contains-----------------------------------------
MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="1200M"
OS: CentOS - 7
RAM: 125 GB
Swap: 3 GB
Processor: Intel(R) Xeon(R) CPU E5-1650 v3 # 3.50GHz
Core: 12 Core
Disk: 2.7 TB
jvm options are like this----------------------
-XX:InitiatingHeapOccupancyPercent=70
-XX:ParallelGCThreads=16
-XX:ConcGCThreads=16
log options are like this -----------------------
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-XX:PrintFLSStatistics=1
-Xloggc:/var/log/cassandra/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M
but thing is one of the server is always turns down.
Thanks and Regards
pavs
I am running a CPU benchmark tool (LINPACK) on a PV virtual machine inside Xen 4.5.1 on Ubuntu 15.10 x64 on an IBM x3550 M4 server. This tool should consume all possible CPU cycles available. I allocate 4 vCPUs by defining this in the Xen PV (test.cfg). However, LINPACK detects only 1 core and 4 threads while it should detect at least 4 cores:
CPU frequency: 2.494 GHz
Number of CPUs: 1
Number of cores: 1
Number of threads: 4
This is what lscpu says inside this Xen PV VM:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 4
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2609 v2 # 2.50GHz
Stepping: 4
CPU MHz: 2500.062
BogoMIPS: 5000.12
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-3
Other platforms such as Docker and HVM DO get cores allocated inside the virtual node (see below). These nodes have significant better performance then the Xen PV virtual node.
CPU frequency: 2.499 GHz
Number of CPUs: 2
Number of cores: 8
Number of threads: 4
This is the DomU Xen host machine lscpu:
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2609 v2 # 2.50GHz
Stepping: 4
CPU MHz: 2500.062
BogoMIPS: 5000.12
Hypervisor vendor: Xen
Virtualization type: none
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-7
Xen vCPU list:
xl vcpu-list
Domain-0 0 0 7 -b- 10.1 all / all
Domain-0 0 1 2 -b- 6.5 all / all
Domain-0 0 2 4 r-- 2.6 all / all
Domain-0 0 3 0 r-- 3.9 all / all
Domain-0 0 4 3 -b- 4.4 all / all
Domain-0 0 5 6 -b- 2.6 all / all
Domain-0 0 6 5 -b- 4.7 all / all
Domain-0 0 7 7 -b- 2.9 all / all
test 3 0 1 -b- 1.5 0-3 / all
test 3 1 0 -b- 1.8 0-3 / all
test 3 2 0 -b- 0.7 0-3 / all
test 3 3 2 -b- 0.6 0-3 / all
xen DomU PV VM config:
cat test.cfg
bootloader = '/usr/lib/xen-4.5/bin/pygrub'
vcpus = '4'
memory = '2048'
cpus = "0-3"
Is there any options to give the paravirtualized guest the host-cpu topology? In other words, how do I get Xen to use more vCPU cores/ vCPUs?
Thanks!
I have two servers running the exact same nodejs application. I am doing load testing and I can't figure out why one of my servers will not utilize more CPU and RAM.
It is much slower when load testing yet it is not even close to utilizing all the free CPU and memory.
If I run top during the load test, these are the numbers I am getting
PID User PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND
1308 ubuntu 20 0 1002524 87508 9788 S 5.3 4.3 0:03.06 nodejs
1307 ubuntu 20 0 925540 75288 9436 S 5.0 3.7 0:02.17 nodejs
1308 ubuntu 20 0 992076 77068 9788 S 14.0 3.8 0:03.48 nodejs
1307 ubuntu 20 0 937140 86904 9436 S 2.7 4.3 0:02.25 nodejs
1308 ubuntu 20 0 1012936 98000 9788 S 14.3 4.8 0:03.91 nodejs
1307 ubuntu 20 0 942940 92644 9436 S 1.0 4.5 0:02.28 nodejs
1307 ubuntu 20 0 943204 92976 9436 S 6.0 4.6 0:02.46 nodejs
1308 ubuntu 20 0 1011764 96804 9788 S 6.0 4.7 0:04.09 nodejs
1307 ubuntu 20 0 933644 83388 9436 S 8.6 4.1 0:02.72 nodejs
1308 ubuntu 20 0 1008720 93556 9788 S 5.3 4.6 0:04.25 nodejs
1308 ubuntu 20 0 1000184 85256 9788 S 8.6 4.2 0:04.51 nodejs
1307 ubuntu 20 0 944092 93988 9436 S 7.6 4.6 0:02.95 nodejs
1307 ubuntu 20 0 941748 91816 9436 S 15.0 4.5 0:03.40 nodejs
1308 ubuntu 20 0 1004832 90008 9788 S 1.3 4.4 0:04.55 nodejs
1307 ubuntu 20 0 933460 82632 9436 S 9.0 4.1 0:03.67 nodejs
Running two processes I don't see memory getting above 4.7% and CPU is at 14%.
It is taking twice as long to serve the exact same resources as a machine with one core and half the memory.
My other server is using %52 of CPU. Granted it has one core and the above has two, but it doesn't seem like that would make the difference.
I downloaded cpufrequtils and set the GOVERNOR to performance but I don't think it is working. This is what I get when I run cpufreq-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
maximum transition latency: 4294.55 ms.
analyzing CPU 1:
no or unknown cpufreq driver is active on this CPU
maximum transition latency: 4294.55 ms.
Here is the CPU
Intel(R) Core(TM)2 CPU 6300 # 1.86GHz
Any ideas or hints would be appreciated
If both servers are running the same node.js application, then you may want to
compare the other settings on the machines, are they the same? ulimit -a
Also for dual/multicore core machines, node.js is single threaded, it will not benefit from dual/multicores unless you use cluster to make use of it.
ive recently activated opcache but it doesn't appear to be working.
It's confirmed activated via phpinfo()
As you can see
0 hits
1 miss
1 cached script (opcached gui)
What am I missing?
Server is a Linux server centos 6.5 vps
PHP 5.5
A bit more info about opcache configuration
opcache_enabled true
cache_full false
restart_pending false
restart_in_progress false
used_memory 8.54 MB
free_memory 503.46 MB
wasted_memory 0 bytes
current_wasted_percentage 0.00%
buffer_size 4194304
used_memory 446.41 kB
free_memory 3.56 MB
number_of_strings 4895
num_cached_scripts 1
num_cached_keys 1
max_cached_keys 65407
hits 0
start_time Sat, 26 Jul 14 23:20:32 +0000
last_restart_time never
oom_restarts 0
hash_restarts 0
manual_restarts 0
misses 1
blacklist_misses 0
blacklist_miss_ratio 0.00%
opcache_hit_rate 0.00%
This looks like you are using cgi rather than mod_php5. The shared memory area (SMA) is used for both, but it only persists request-to-request for the latter.
I had this issue on a WHM/cPanel server today. As TerryE suggests, you are probably running CGI or suPHP. Change to DSO.
Hi I just installed Cloudera Manager on my cluster, 1 namenode and 4 datanodes, each data nodes has 64 GB RAM, 24 cores Xeon CPU, 16 1T disks SAS..etc.
I installed brand new Redhat Linux and upgraded to 6.5, each disk has been logically set up as RAID0 since there is no JBOD option available on the array controller.
I am running a hive query and here is the top command on the data node. I am so confused and wondering if some experienced hadoop admin could help me understand if my cluster is working fine.
Why there is only 1 task running out of 897 while the other 896 sleeping? There are 2271 mappers for that hive query and it is only 80% on the mapper side.
The load average is 8.66, I read from here that if you computer is working hard, the load average should be around the number of cores. Is my datanode working hard enought?
List item 69/70 memory has been "used", seems like the active yarn process is fairly low memory cost, how could those 64GB memory be so easily used up?
Here is the top output:
top - 22:50:24 up 1 day, 8:24, 3 users, load average: 8.66, 8.50, 7.95
Tasks: 897 total, 1 running, 896 sleeping, 0 stopped, 0 zombie
Cpu(s): 32.3%us, 5.2%sy, 0.0%ni, 62.3%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 70096068k total, 69286800k used, 809268k free, 222268k buffers
Swap: 4194296k total, 0k used, 4194296k free, 61468376k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
439 yarn 20 0 1417m 591m 19m S 193.9 0.9 1:06.12 java
561 yarn 20 0 1401m 581m 19m S 193.2 0.8 0:19.75 java
721 yarn 20 0 1415m 561m 19m S 172.0 0.8 0:08.54 java
611 yarn 20 0 1415m 574m 19m S 127.0 0.8 0:16.87 java
354 yarn 20 0 1428m 595m 19m S 121.4 0.9 0:35.96 java
27418 yarn 20 0 1513m 483m 18m S 13.6 0.7 18:26.14 java
16895 hdfs 20 0 1438m 410m 18m S 9.6 0.6 103:23.70 java
3726 hdfs 20 0 860m 249m 21m S 1.7 0.4 2:12.28 java
I am fairly new at system admin and any metric tool or common sense will be much appreciated! Thanks!