How to know the connections number of MQTT adapter? - eclipse-hono
We want to do a performance test on Hono to verify that it supports 12000 device connections (MQTT), but I can't find the number of device connections in grafana's dashboard. I also checked the prometheus data endpoint of the MQTT adapter and found no metrics related to the number of connections. The metrics of the mqtt adapter are as follows
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 30.0
# HELP hono_connections_authenticated
# TYPE hono_connections_authenticated gauge
hono_connections_authenticated{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 0.0
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="debug",} 813043.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="info",} 1276.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="trace",} 0.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="warn",} 1.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="error",} 8.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 2.987608E7
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.6964352E7
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 315472.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 6350352.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.046592E7
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 1260928.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} 5.6632848E7
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 7935840.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.591340143952E9
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 9043.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="timed-waiting",} 2.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="terminated",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="waiting",} 22.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="blocked",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="new",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="runnable",} 6.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_memory_used_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 5.034343E7
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 32.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1048576.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 7.3979744768E10
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 30.0
# HELP hono_connections_unauthenticated
# TYPE hono_connections_unauthenticated gauge
hono_connections_unauthenticated{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.1365048E8
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_count_buffers{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 9.0
# HELP hono_connections_authenticated_duration_seconds_max
# TYPE hono_connections_authenticated_duration_seconds_max gauge
hono_connections_authenticated_duration_seconds_max{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 20.0
# HELP hono_connections_authenticated_duration_seconds
# TYPE hono_connections_authenticated_duration_seconds summary
hono_connections_authenticated_duration_seconds_count{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 39.0
hono_connections_authenticated_duration_seconds_sum{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 554.666
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.077
jvm_gc_pause_seconds_count{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 12.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.14
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 4483.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.652
jvm_gc_pause_seconds_count{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.12
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
jvm_gc_pause_seconds_max{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.002
jvm_gc_pause_seconds_max{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 345543.191
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.546644844517185E-4
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 4.8328704E7
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.7891328E7
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 2424832.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 7077888.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.0747904E7
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} 5.8368E7
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 1.9464192E7
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.036333878887070375
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.7826992E7
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.93
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 197.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.146486784E9
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 1.146486784E9
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 5.7278464E7
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 5828608.0
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 4.58620928E8
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 5.0343429E7
How should I know how many devices the MQTT adapter has connected?
In general, how many connections can a 1c2G MQTT POD support?
Hono's components report several metrics like number of connected devices, messages transferred etc.
These metrics are reported to a monitoring backend which usually implements a time series data store. Hono by default supports Prometheus for that purpose. You should be able to retrieve the relevant information from the Prometheus server using its query API.
For a pod with 1 CPU and 2GB of memory available, I would think that you should be able to connect at least 25000 devices. However, note that the number of connections is not only limited by CPU and RAM but also by the number of file descriptors available to the OS and maybe other factors.
Related
Papi_avail no events available - Unknown libpfm4 related error
I just made a fresh install of Ubuntu 20 on an AMD Ryzen 5000 series. I installed papi tools 6 by following the install instructions. The installation was succsessful and I verified the installed version: $ sudo papi_version PAPI Version: 6.0.0.0 Then, when I verified the available events, I got none: $ sudo papi_avail Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI version : 6.0.0.0 Operating system : Linux 5.8.0-55-generic Vendor string and code : AuthenticAMD (2, 0x2) Model string and code : AMD Ryzen 9 5900 12-Core Processor (33, 0x21) CPU revision : 0.000000 CPUID : Family/Model/Stepping 25/33/0, 0x19/0x21/0x00 CPU Max MHz : 6084 CPU Min MHz : 2200 Total cores : 24 SMT threads per core : 2 Cores per socket : 12 Sockets : 1 Cores per NUMA region : 24 NUMA regions : 1 Running in a VM : no Number Hardware Counters : 0 Max Multiplex Counters : 384 Fast counter read (rdpmc): yes -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 No No Level 1 data cache misses PAPI_L1_ICM 0x80000001 No No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 No No Level 1 cache misses PAPI_L2_TCM 0x80000007 No No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 No No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 No No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 No No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 No No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a No No Unconditional branch instructions PAPI_BR_CN 0x8000002b No No Conditional branch instructions PAPI_BR_TKN 0x8000002c No No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d No No Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e No No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 No No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 No No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 No No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 No No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b No No Total cycles PAPI_LST_INS 0x8000003c No No Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 No No Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 No No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c No No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 No No Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 No No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b No No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 0 are available, of which 0 are derived. No events detected! Check papi_component_avail to find out why. So I followed the instructions and I ran the papi_component_avail: papi_component_avail Available components and hardware information. -------------------------------------------------------------------------------- PAPI version : 6.0.0.0 Operating system : Linux 5.8.0-55-generic Vendor string and code : AuthenticAMD (2, 0x2) Model string and code : AMD Ryzen 9 5900 12-Core Processor (33, 0x21) CPU revision : 0.000000 CPUID : Family/Model/Stepping 25/33/0, 0x19/0x21/0x00 CPU Max MHz : 6084 CPU Min MHz : 2200 Total cores : 24 SMT threads per core : 2 Cores per socket : 12 Sockets : 1 Cores per NUMA region : 24 NUMA regions : 1 Running in a VM : no Number Hardware Counters : 0 Max Multiplex Counters : 384 Fast counter read (rdpmc): yes -------------------------------------------------------------------------------- Compiled-in components: Name: perf_event Linux perf_event CPU counters \-> Disabled: Unknown libpfm4 related error Name: perf_event_uncore Linux perf_event CPU uncore and northbridge \-> Disabled: No uncore PMUs or events found Active components: -------------------------------------------------------------------------------- I tried applying the fix explained in the stack overflow question papi_avail: No events available - 32308175 which in the past has worked on my Ubuntu 18 running an Intel 6700HQ, but with no avail here. Of course, the problem descriptions were different.I am uncertain on how to resolve the "Unknown libpfm4 related error" error. I also tried installing the current availabe version of Papi using the "apt get install papi tools" sudo apt-get update sudo apt-get install papi-tools The installation went fine but I keep getting the same errore messages. What have I missed?
Your CPU is part of the AMD ZEN3 family. libpfm4 seems to support this CPU architecture family from v4.12.0. If you have papi version 6.0.0.1 or less, it comes with a built-in version of libpfm4 that is older than v4.12.0 so I suppose papi is expected not to work in that case? Nevertheless, we are still experiencing the same issue when installing the master branch of papi (that comes with a libpfm4 version that should support AMD ZEN3 according to documentation).
mpm configuration for httpd
I run a website with 5 httpd servers(Centos 7) in EC2, type is m3.2xlarge. The servers are configured with load balancer. Gradually the server memory keeps going higher in all the instances. For example: Memory usage in few seconds after restarting the httpd service: [centos#ip-10-0-1-77 ~]$ while sleep 1; do free -m; done total used free shared buff/cache available Mem: 29741 2700 26732 36 307 26728 Swap: 0 0 0 total used free shared buff/cache available Mem: 29741 2781 26651 36 307 26647 Swap: 0 0 0 total used free shared buff/cache available Mem: 29741 2820 26613 36 307 26609 Swap: 0 0 0 [centos#ip-10-0-1-77 ~]$ . . . This is what i see after an hour: [centos#ip-10-0-1-77 ~]$ free -m total used free shared buff/cache available Mem: 29741 29092 363 41 284 346 Swap: 0 0 0 Like above it goes and consumes all the memory(30GB) within an hour. To avoid this I started using worker mpm configuration. The following configuration is what I have added at the bottom of /etc/httpd/httpd.conf. <IfModule mpm_worker_module> MaxRequestWorkers 2500 MaxSpareThreads 250 MinSpareThreads 75 ServerLimit 100 StartServers 3 ThreadsPerChild 25 </IfModule> Can Someone help and suggest me the right configuration to utilize the RAM memory properly in all the instances?
A standard Apache process takes up about 12 MB of RAM. If you have 30 GB reserved for Apache you will never reach that with a serverlimit of 100 (=100*12MB=1200MB=1,2GB). So I assume that Apache isn't taking up all that memory. Is there an application that is involved or a DB? Those can take up larger amounts of RAM. For your servertuning.conf (or httpd.conf since you put it there): <IfModule mpm_worker_module> #max amount of requests one worker handles before it's forced to close, 2,5k seems almost a little low MaxRequestsperChild 2500 #maximum number of worker threads which are kept spare #250 seems quite high, but really depends on the traffic you are experiencing, we normally don't use more than 50-75 MaxSpareThreads 250 #minimum number of worker threads which are kept spare #really high, only useful if you often experience higher bursts of traffic #otherwise you should be fine with 15-30, maybe 50 if you experience higher fluctuation -> bigger bursts of requests MinSpareThreads 75 #upper limit on the configurable number of threads per child process #you have to increase this if you want more than 64 ThreadsPerChild ThreadLimit 64 #maximum number of simultaneous client connections #that is really low! --> increase that, definitely! We run on 1000, so about 12GB max MaxClients 100 #initial number of server processes to start #3 is really low, if you expected your server to be flodded with requests the second you start it #maybe turn it up a little to around 20 or even 50 if you receive lots of traffic right after a restart StartServers 3 #number of worker threads created by each child proces #25 threads per worker is not tooo much, but at some point the administration of xx threads gets more expensive than creating new ones #would suggest to leave it at 25 or turn it up to around 40 ThreadsPerChild 25 </IfModule> Notice that I changed ServerLimit to MaxClients and MaxRequestWorkers to MaxRequestsPerChild, because as far as I know those are the terms used in mpm-worker. Additionally you can change following variables: #KeepAlive: Whether or not to allow persistent connections (more than #one request per connection). Set to "Off" to deactivate. #if it's on, leave it there KeepAlive On #MaxKeepAliveRequests: The maximum number of requests to allow #during a persistent connection. Set to 0 to allow an unlimited amount. #We recommend you leave this number high, for maximum performance. #default=100, but you can turn that up if your sites contain a lot of item (img, css, ...) #we are using about 20*<average object-count per site> = 600 MaxKeepAliveRequests 600 #KeepAliveTimeout: Number of seconds to wait for the next request from the #same client on the same connection. #would recommend to decrease that, otherwise you could become a victim of slow-dos attacks #default is 15, we are running just fine on 5 KeepAliveTimeout 5 To further prevent slow-dos or piling up of open sessions, you can use mod_reqtimeout: <IfModule mod_reqtimeout.c> # allow 10s timeout for the headers and allow 1s more until 20s upon receipt of 1000 bytes. # almost the same with the body, except that it is tricky to # limit the request timeout within the body at all - it may take time to generate the body. # below are the default values #RequestReadTimeout header=10-20,MinRate=1000 body=20,MinRate=1000 # and this is how I'd set them with today's internet speed # deduct the according numbers from explanation above... RequestReadTimeout header=2-5,MinRate=100000 body=5-10,MinRate=1000000 </IfModule> If that's not enough to help your RAM-issues (if they are really caused by Apache), use the tools of your server's OS accordingly to find out what is taking up all the RAM --> TOOLS
Excessive Gen 2 Free Blocks in Crash Dump
On inspecting a crash dump file for an out of memory exception reported by a client the results of !DumpHeap -stat showed that 575MB of memory is being taken up by 45,000 objects of type "Free" most of which I assume would have to reside in Gen 2 due to the size. The first places I looked for problems were the large object heap (LOH) and pinned objects. The large object heap with free space included was only 70MB so that wasn't the issue and running !gchandles showed: GC Handle Statistics: Strong Handles: 155 Pinned Handles: 265 Async Pinned Handles: 8 Ref Count Handles: 163 Weak Long Handles: 0 Weak Short Handles: 0 Other Handles: 0 which is a very small number of handles (around 600) compared to the number of free objects (45,000). To me this rules out the free blocks being caused by pinning. I also looked into the free blocks themselves to see if maybe they had a consistent size, but on inspection the sizes varied widely and went from just short of 5MB to only around 12 bytes or so. Any help would be appreciated! I am at a loss since there is fragmentation but no signs of it being cause by the two places that I know to look which are the large object heap (LOH) and pinned handles.
In which generation are the free objects? I assume would have to reside in Gen 2 due to the size Size is not related to generations. To find out in which generation the free blocks reside, you can follow these steps: From !dumpheap -stat -type Free get the method table: 0:003> !dumpheap -stat -type Free total 7 objects Statistics: MT Count TotalSize Class Name 00723538 7 100 Free Total 7 objects From !eeheap -gc, get the start addresses of the generations. 0:003> !eeheap -gc Number of GC Heaps: 1 generation 0 starts at 0x026a1018 generation 1 starts at 0x026a100c generation 2 starts at 0x026a1000 ephemeral segment allocation context: none segment begin allocated size 026a0000 026a1000 02731ff4 0x00090ff4(593908) Large object heap starts at 0x036a1000 segment begin allocated size 036a0000 036a1000 036a3250 0x00002250(8784) Total Size 0x93244(602692) ------------------------------ GC Heap Size 0x93244(602692) Then dump the free objects only of a specific generation by passing the start and end address (e.g. of generation 2 here): 0:003> !dumpheap -mt 00723538 0x026a1000 0x026a100c Address MT Size 026a1000 00723538 12 Free 026a100c 00723538 12 Free total 2 objects Statistics: MT Count TotalSize Class Name 00723538 2 24 Free Total 2 objects So in my simple case, there are 2 free objects in generation 2. Pinned handles 600 pinned objects should not cause 45.000 free memory blocks. Still from my experience, 600 pinned handles are a lot. But first, check in which generation the free memory blocks reside.
How could uptime command show a cpu time greater than 100%?
Running an application with an infinite loop (no sleep, no system calls inside the loop) on a linux system with kernel 2.6.11 and single core processor results in a 98-99% cputime consuming. That's normal. I have another single thread server application normally with an average sleep of 98% and a maximum of 20% of cputime. Once connected with a net client, the sleep average drops to 88% (nothing strange) but the cputime (1 minute average) rises constantly but not immediately over the 100%... I saw also a 160% !!?? The net traffic is quite slow (one small packet every 0.5 seconds). Usually the 15 min average of uptime command shows about 115%. I ran also gprof but I did not find it useful... I have a similar scenario: IDLE SCENARIO %time name 59.09 Run() 25.00 updateInfos() 5.68 updateMcuInfo() 2.27 getLastEvent() .... CONNECTED SCENARIO %time name 38.42 updateInfo() 34.49 Run() 10.57 updateCUinfo() 4.36 updateMcuInfo() 3.90 ... 1.77 ... .... None of the listed functions is directly involved with client communication How can you explain this behaviour? Is it a known bug? Could be the extra time consumed in kernel space after a system call and lead to a calculus like %=100*(apptime+kerneltime)/(total apps time)?
How to find the processor queue length in linux
Trying to determine the Processor Queue Length (the number of processes that ready to run but currently aren't) on a linux machine. There is a WMI call in Windows for this metric, but not knowing much about linux I'm trying to mine /proc and 'top' for the information. Is there a way to determine the queue length for the cpu? Edit to add: Microsoft's words concerning their metric: "The collection of one or more threads that is ready but not able to run on the processor due to another active thread that is currently running is called the processor queue."
sar -q will report queue length, task list length and three load averages. Example: matli#tornado:~$ sar -q 1 0 Linux 2.6.27-9-generic (tornado) 01/13/2009 _i686_ 11:38:32 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 11:38:33 PM 0 305 1.26 0.95 0.54 11:38:34 PM 4 305 1.26 0.95 0.54 11:38:35 PM 1 306 1.26 0.95 0.54 11:38:36 PM 1 306 1.26 0.95 0.54 ^C
vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 256368 53764 75980 220564 2 28 60 54 774 1343 15 4 78 2 The first column (r) is the run queue - 2 on my machine right now Edit: Surprised there isn't a way to just get the number Quick 'n' dirty way to get the number (might vary a little on different machines): vmstat|tail -1|cut -d" " -f2
The metrics you seek exist in /proc/schedstat. The format of this file is described in sched-stats.txt in the kernel source. Specifically, the cpu<N> lines are what you want: CPU statistics -------------- cpu<N> 1 2 3 4 5 6 7 8 9 First field is a sched_yield() statistic: 1) # of times sched_yield() was called Next three are schedule() statistics: 2) This field is a legacy array expiration count field used in the O(1) scheduler. We kept it for ABI compatibility, but it is always set to zero. 3) # of times schedule() was called 4) # of times schedule() left the processor idle Next two are try_to_wake_up() statistics: 5) # of times try_to_wake_up() was called 6) # of times try_to_wake_up() was called to wake up the local cpu Next three are statistics describing scheduling latency: 7) sum of all time spent running by tasks on this processor (in jiffies) 8) sum of all time spent waiting to run by tasks on this processor (in jiffies) 9) # of timeslices run on this cpu In particular, field 8. To find the run queue length, you would: Observe field 8 for each CPU and record the value. Wait for some interval. Observe field 8 for each CPU again, and calculate how much the value has increased. Dividing that difference by the length of the time interval waited (the documentation says it's in jiffies, but it's actually in nanoseconds since the addition of CFS), by Little's Law, yields the mean length of the scheduler run queue over the interval. Unfortunately, I'm not aware of any utility to automate this process which is usually installed or even packaged in a Linux distribution. I've not used it, but the kernel documentation suggests http://eaglet.rain.com/rick/linux/schedstat/v12/latency.c, which unfortunately refers to a domain that is no longer resolvable. Fortunately, it's available on the wayback machine. Why not sar or vmstat? These tools report the number of currently runnable processes. Certainly if this number is greater than the number of CPUs, some of them must be waiting. However, processes can still be waiting even when the number of processes is less than the number of CPUs, for a variety of reasons: A process may be pinned to a particular CPU. The scheduler may decide to schedule a process on a particular CPU to make better utilization of cache, or for NUMA optimization reasons. The scheduler may intentionally idle a CPU to allow more time to a competing, higher priority process on another CPU that shares the same execution core (a hyperthreading optimization). Hardware interrupts may be processable only on particular CPUs for a variety of hardware and software reasons. Moreover, the number of runnable processes is only sampled at an instant in time. In many cases this number may fluctuate rapidly, and the contention may be occurring between the times the metric is being sampled. These things mean the number of runnable processes minus the number of CPUs is not a reliable indicator of CPU contention.
uptime will give you the recent load average, which is approximately the average number of active processes. uptime reports the load average over the last 1, 5, and 15 minutes. It's a per-system measurement, not per-CPU. Not sure what the processor queue length in Windows is, hopefully it's close enough to this?