Related
I have deployed OpenStack and configured OVS-DPDK on compute nodes for high-performance networking. My workload is a general-purpose workload like running haproxy, mysql, apache, and XMPP etc.
When I did load-testing, I found performance is average and after 200kpps packet rate I noticed packet drops. I heard and read DPDK can handle millions of packets but in my case, it's not true. In guest, I am using virtio-net which processes packets in the kernel so I believe my bottleneck is my guest VM.
I don't have any guest-based DPDK application like testpmd etc. Does that mean OVS+DPDK isn't useful for my cloud? How do I take advantage of OVS+DPDK with a general-purpose workload?
Updates
We have our own loadtesting tool which generate Audio RTP traffic which is pure UDP based 150bytes packets and noticed after 200kpps audio quality go down and choppy. In short DPDK host hit high PMD cpu usage and loadtest showing bad audio quality. when i do same test with SRIOV based VM then performance is really really good.
$ ovs-vswitchd -V
ovs-vswitchd (Open vSwitch) 2.13.3
DPDK 19.11.7
Intel NIC X550T
# ethtool -i ext0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80000d63, 18.8.9
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
In the following output what does these queue-id:0 to 8 and why only
the first queue is in use but not others, they are always zero. What
does this mean?
ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 2:
isolated : false
port: vhu1c3bf17a-01 queue-id: 0 (enabled) pmd usage: 0 %
port: vhu1c3bf17a-01 queue-id: 1 (enabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 2 (disabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 3 (disabled) pmd usage: 0 %
pmd thread numa_id 1 core_id 3:
isolated : false
pmd thread numa_id 0 core_id 22:
isolated : false
port: vhu1c3bf17a-01 queue-id: 3 (enabled) pmd usage: 0 %
port: vhu1c3bf17a-01 queue-id: 6 (enabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 0 (enabled) pmd usage: 54 %
port: vhu6b7daba9-1a queue-id: 5 (disabled) pmd usage: 0 %
pmd thread numa_id 1 core_id 23:
isolated : false
port: dpdk1 queue-id: 0 (enabled) pmd usage: 3 %
pmd thread numa_id 0 core_id 26:
isolated : false
port: vhu1c3bf17a-01 queue-id: 2 (enabled) pmd usage: 0 %
port: vhu1c3bf17a-01 queue-id: 7 (enabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 1 (disabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 4 (disabled) pmd usage: 0 %
pmd thread numa_id 1 core_id 27:
isolated : false
pmd thread numa_id 0 core_id 46:
isolated : false
port: dpdk0 queue-id: 0 (enabled) pmd usage: 27 %
port: vhu1c3bf17a-01 queue-id: 4 (enabled) pmd usage: 0 %
port: vhu1c3bf17a-01 queue-id: 5 (enabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 6 (disabled) pmd usage: 0 %
port: vhu6b7daba9-1a queue-id: 7 (disabled) pmd usage: 0 %
pmd thread numa_id 1 core_id 47:
isolated : false
$ ovs-appctl dpif-netdev/pmd-stats-clear && sleep 10 && ovs-appctl
dpif-netdev/pmd-stats-show | grep "processing cycles:"
processing cycles: 1697952 (0.01%)
processing cycles: 12726856558 (74.96%)
processing cycles: 4259431602 (19.40%)
processing cycles: 512666 (0.00%)
processing cycles: 6324848608 (37.81%)
Does processing cycles mean my PMD is under stress? but i am only
hitting 200kpps rate?
This is my dpdk0 and dpdk1 port statistics
sudo ovs-vsctl get Interface dpdk0 statistics
{flow_director_filter_add_errors=153605,
flow_director_filter_remove_errors=30829, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64338613,
rx_1_to_64_packets=367, rx_256_to_511_packets=116298,
rx_512_to_1023_packets=31264, rx_65_to_127_packets=6990079,
rx_broadcast_packets=0, rx_bytes=12124930385, rx_crc_errors=0,
rx_dropped=0, rx_errors=12, rx_fcoe_crc_errors=0, rx_fcoe_dropped=12,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=367,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=128, rx_management_dropped=35741,
rx_management_packets=31264, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71512362,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1096,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=6990079, tx_128_to_255_packets=64273778,
tx_1_to_64_packets=128, tx_256_to_511_packets=43670294,
tx_512_to_1023_packets=153605, tx_65_to_127_packets=881272,
tx_broadcast_packets=10, tx_bytes=25935295292, tx_dropped=0,
tx_errors=0, tx_management_packets=0, tx_multicast_packets=153,
tx_packets=109009906}
stats
sudo ovs-vsctl get Interface dpdk1 statistics
{flow_director_filter_add_errors=126793,
flow_director_filter_remove_errors=37969, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64435459,
rx_1_to_64_packets=107843, rx_256_to_511_packets=230,
rx_512_to_1023_packets=13, rx_65_to_127_packets=7049788,
rx_broadcast_packets=199058, rx_bytes=12024342488, rx_crc_errors=0,
rx_dropped=0, rx_errors=11, rx_fcoe_crc_errors=0, rx_fcoe_dropped=11,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=107843,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=1906, rx_management_dropped=0,
rx_management_packets=13, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71593333,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1131,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=7049788, tx_128_to_255_packets=102664472,
tx_1_to_64_packets=1906, tx_256_to_511_packets=68008814,
tx_512_to_1023_packets=126793, tx_65_to_127_packets=1412435,
tx_broadcast_packets=1464, tx_bytes=40693963125, tx_dropped=0,
tx_errors=0, tx_management_packets=199058, tx_multicast_packets=146,
tx_packets=172252389}
Update - 2
dpdk interface
# dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:3b:00.1 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe
0000:af:00.1 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe
Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno1 drv=tg3 unused=vfio-pci
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno2 drv=tg3 unused=vfio-pci
0000:3b:00.0 'Ethernet Controller 10G X550T 1563' if=int0 drv=ixgbe unused=vfio-pci
0000:af:00.0 'Ethernet Controller 10G X550T 1563' if=int1 drv=ixgbe unused=vfio-pci
OVS
# ovs-vsctl show
595103ef-55a1-4f71-b299-a14942965e75
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port vxlan-0a48042b
Interface vxlan-0a48042b
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.72.4.44", out_key=flow, remote_ip="10.72.4.43"}
Port vxlan-0a480429
Interface vxlan-0a480429
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.72.4.44", out_key=flow, remote_ip="10.72.4.41"}
Port vxlan-0a48041f
Interface vxlan-0a48041f
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.72.4.44", out_key=flow, remote_ip="10.72.4.31"}
Port vxlan-0a48042a
Interface vxlan-0a48042a
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.72.4.44", out_key=flow, remote_ip="10.72.4.42"}
Bridge br-vlan
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port br-vlan
Interface br-vlan
type: internal
Port dpdkbond
Interface dpdk1
type: dpdk
options: {dpdk-devargs="0000:af:00.1", n_txq_desc="2048"}
Interface dpdk0
type: dpdk
options: {dpdk-devargs="0000:3b:00.1", n_txq_desc="2048"}
Port phy-br-vlan
Interface phy-br-vlan
type: patch
options: {peer=int-br-vlan}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port vhu87cf49d2-5b
tag: 7
Interface vhu87cf49d2-5b
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhu87cf49d2-5b"}
Port vhub607c1fa-ec
tag: 7
Interface vhub607c1fa-ec
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhub607c1fa-ec"}
Port vhu9a035444-83
tag: 8
Interface vhu9a035444-83
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhu9a035444-83"}
Port br-int
Interface br-int
type: internal
Port int-br-vlan
Interface int-br-vlan
type: patch
options: {peer=phy-br-vlan}
Port vhue00471df-d8
tag: 8
Interface vhue00471df-d8
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhue00471df-d8"}
Port vhu683fdd35-91
tag: 7
Interface vhu683fdd35-91
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhu683fdd35-91"}
Port vhuf04fb2ec-ec
tag: 8
Interface vhuf04fb2ec-ec
type: dpdkvhostuserclient
options: {vhost-server-path="/var/lib/vhost_socket/vhuf04fb2ec-ec"}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
ovs_version: "2.13.3"
I have created guest vms using openstack and they can see them they are connected using vhost socket (Ex: /var/lib/vhost_socket/vhuf04fb2ec-ec)
When I did load-testing, I found performance is average and after 200kpps packet rate I noticed packet drops. In short DPDK host hit high PMD cpu usage and loadtest showing bad audio quality. when i do same test with SRI
[Answer] this observation is not true based on the live debug done so far. The reason as stated below
qemu launched were not pinned to specific cores.
comparison done against PCIe pass-through (VF) against vhost-client is not apples to apples comparison.
with OpenStack approach, there are at least 3 bridges before the packets to flow through before reaching VM.
OVS threads were not pinned which led to all the PMD threads running on the same core (causing latency and drops) in each bridge stage.
To have a fair comparison against SRIOV approach, the following changes have been made with respect to similar question
External Port <==> DPDK Port0 (L2fwd) DPDK net_vhost <--> QEMU (virtio-pci)
Numbers achieved with iperf3 (bidirectional) is around 10Gbps.
Note: requested to run trex, pktgen to try out Mpps. Expectation is to reach minimum 8 MPPS with the current setup.
Hence this is not DPDK, virtio-client, qemu-kvm or SRIOV related issue, instead a configuration or platform setup issue.
I am parsing chunks of pos-tagged text in JupyterLabs.
NLTK can run on Python<3.5 and >3.8 according to its faq.
I can return pos-tagged text just fine. But when I want to return parsed chunks, it crashes python.
I am running MacOS 11.1, Python 3.8.6, Jupyterlab 3.0.0, and nltk 3.5
from nltk import *
text = """The Buddha, the Godhead, resides quite as comfortably in the circuits of a digital
computer or the gears of a cycle transmission as he does at the top of a mountain
or in the petals of a flower. To think otherwise is to demean the Buddha...which is
to demean oneself."""
sentence_re = r'''(?x)
([A-Z])(\.[A-Z])+\.?
| \w+(-\w+)*
| \$?\d+(\.\d+)?%?
| \.\.\.
| [][.,;"'?():-_`]
'''
grammar = r"""
NBAR:
{<NN.*|JJ>*<NN.*>} # Nouns and Adjectives, terminated with Nouns
NP:
{<NBAR>}
{<NBAR><IN><NBAR>} # Above, connected with in/of/etc...
"""
chunker = RegexpParser(grammar)
toks = word_tokenize(text)
postoks = pos_tag(toks)
All is fine until I want to parse the chunks with RegexpParser. At which point, it crashes the kernel.
chunker.parse(postoks)
beginning of the crash report:
Process: Python [1549]
Path: /usr/local/Cellar/python#3.8/3.8.6_2/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python
Identifier: org.python.python
Version: 3.8.6 (3.8.6)
Code Type: X86-64 (Native)
Parent Process: Python [1522]
Responsible: Terminal [416]
User ID: 501
Date/Time: 2020-12-29 15:34:13.546 -0500
OS Version: macOS 11.1 (20C69)
Report Version: 12
Anonymous UUID: 8EEE2257-0986-3569-AA83-52641AF02282
Time Awake Since Boot: 1200 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
abort() called
end of the crash report:
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 4
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 4785
thread_create: 0
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=835.0M resident=0K(0%) swapped_out_or_unallocated=835.0M(100%)
Writable regions: Total=1.5G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.5G(100%)
VIRTUAL REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Activity Tracing 256K 1
Dispatch continuations 64.0M 1
Kernel Alloc Once 8K 1
MALLOC 150.1M 37
MALLOC guard page 24K 5
MALLOC_MEDIUM (reserved) 960.0M 8 reserved VM address space (unallocated)
STACK GUARD 72K 18
Stack 86.6M 18
VM_ALLOCATE 182.5M 359
VM_ALLOCATE (reserved) 128.0M 2 reserved VM address space (unallocated)
__DATA 16.1M 460
__DATA_CONST 11.8M 200
__DATA_DIRTY 509K 87
__FONT_DATA 4K 1
__LINKEDIT 506.1M 241
__OBJC_RO 60.5M 1
__OBJC_RW 2452K 2
__TEXT 329.5M 432
__UNICODE 588K 1
mapped file 51.5M 9
shared memory 40K 4
=========== ======= =======
TOTAL 2.5G 1888
TOTAL, minus reserved VM space 1.4G 1888
Model: iMac18,2, BootROM 429.60.3.0.0, 4 processors, Quad-Core Intel Core i7, 3.6 GHz, 32 GB, SMC 2.40f1
Graphics: kHW_AMDRadeonPro560Item, Radeon Pro 560, spdisplays_pcie_device, 4 GB
Memory Module: BANK 0/DIMM0, 16 GB, DDR4 SO-DIMM, 2400 MHz, 0x802C, 0x313641544632473634485A2D3247334232202020
Memory Module: BANK 1/DIMM0, 16 GB, DDR4 SO-DIMM, 2400 MHz, 0x802C, 0x313641544632473634485A2D3247334232202020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x16E), Broadcom BCM43xx 1.0 (7.77.111.1 AirPortDriverBrcmNIC-1675.1)
Bluetooth: Version 8.0.2f9, 3 services, 27 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en1
USB Device: USB 3.0 Bus
USB Device: AS2105
USB Device: USB 2.0 BILLBOARD
USB Device: Bluetooth USB Host Controller
USB Device: FaceTime HD Camera (Built-in)
USB Device: Scarlett 2i4 USB
USB Device: My Passport 0827
Thunderbolt Bus: iMac, Apple Inc., 41.4
I'm unable to use uWSGI's busyness cheaper algorithm although it appears to be pre-loaded in my installation. Does it still merit an explicit install?
Is so, where can I download the standalone plugin package from?
Any help is appreciated, thank you.
uWSGI Information
uwsgi --version
2.0.19.1
uwsgi --cheaper-algos-list
*** uWSGI loaded cheaper algorithms ***
busyness
spare
backlog
manual
--- end of cheaper algorithms list ---
uWSGI Configuration File
[uwsgi]
module = myapp:app
socket = /path/to/myapp.sock
stats = /path/to/mystats.sock
chmod-socket = 766
socket-timeout = 60 ; Set internal sockets timeout
logto = /path/to/logs/%n.log
log-maxsize = 5000000 ; Max size before rotating file
disable-logging = true ; Disable built-in logging
log-4xx = true ; But log 4xx
log-5xx = true ; And 5xx
strict = true ; Enable strict mode (placeholder cannot be used)
master = true ; Enable master process
enable-threads = true ; Enable threads
vacuum = true ; Delete sockets during shutdown
single-interpreter = true ; Do not use multiple interpreters (single web app per uWSGI process)
die-on-term = true ; Shutdown when receiving SIGTERM (default is respawn)
need-app = true ; Exit if no app can be loaded
harakiri = 300 ; Forcefully kill hung workers after desired time in seconds
max-requests = 1000 ; Restart workers after this many requests
max-worker-lifetime = 3600 ; Restart workers after this many seconds
reload-on-rss = 1024 ; Restart workers after this much resident memory (this is per worker)
worker-reload-mercy = 60 ; How long to wait for workers to reload before forcefully killing them
cheaper-algo = busyness ; Specify the cheaper algorithm here
processes = 16 ; Maximum number of workers allowed
threads = 4 ; Number of threads per worker allowed
thunder-lock = true ; Specify thunderlock activation
cheaper = 8 ; Number of workers to keep idle
cheaper-initial = 8 ; Workers created at startup
cheaper-step = 4 ; Number of workers to spawn at once
cheaper-overload = 30 ; Check the busyness of the workers at this interval (in seconds)
uWSGI Log
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x15e8f10 pid: 14479 (default app)
spawned uWSGI master process (pid: 14479)
spawned uWSGI worker 1 (pid: 14511, cores: 4)
spawned uWSGI worker 2 (pid: 14512, cores: 4)
spawned uWSGI worker 3 (pid: 14516, cores: 4)
spawned uWSGI worker 4 (pid: 14520, cores: 4)
spawned uWSGI worker 5 (pid: 14524, cores: 4)
spawned uWSGI worker 6 (pid: 14528, cores: 4)
spawned uWSGI worker 7 (pid: 14529, cores: 4)
spawned uWSGI worker 8 (pid: 14533, cores: 4)
THIS LINE --> unable to find requested cheaper algorithm, falling back to spare <-- THIS LINE
OS Information
Red Hat Enterprise Linux Server release 7.7 (Maipo)
Other Details
uWSGI was installed using pip
The comments were messing things up for me - the below format fixed the issue
########################################################
# #
# Cheaper Algo and Worker Count #
# #
########################################################
cheaper-algo = busyness
processes = 16
threads = 4
thunder-lock = true
cheaper = 8
cheaper-initial = 8
cheaper-step = 4
cheaper-overload = 30
While testing a simple node server (written with Hapi.js):
'use strict';
var Hapi = require("hapi");
var count = 0;
const server = Hapi.server({
port: 3000,
host: 'localhost'
});
server.route({
method: 'GET',
path: '/test',
handler: (request, h) => {
count ++;
console.log(count);
return count;
}
});
const init = async () => {
await server.start();
};
process.on('unhandledRejection', (err) => {
process.exit(1);
});
init();
start the server:
node ./server.js
run the Apache ab tool:
/usr/bin/ab -n 200 -c 30 localhost:3000/test
Env details:
OS: CentOS release 6.9
Node: v10.14.1
Hapi.js: 17.8.1
I found unexpected results in case of multiple concurrent requests (-c 30): the request handler function has been called more times than the number of requests to be performed (-n 200).
Ab output example:
Benchmarking localhost (be patient)
Server Software:
Server Hostname: localhost
Server Port: 3000
Document Path: /test
Document Length: 29 bytes
Concurrency Level: 30
Time taken for tests: 0.137 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 36081 bytes
HTML transferred: 6119 bytes
Requests per second: 1459.44 [#/sec] (mean)
Time per request: 20.556 [ms] (mean)
Time per request: 0.685 [ms] (mean, across all concurrent requests)
Transfer rate: 257.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 0
Processing: 15 17 1.5 16 20
Waiting: 2 9 3.9 9 18
Total: 15 17 1.5 16 21
Percentage of the requests served within a certain time (ms)
50% 16
66% 16
75% 17
80% 18
90% 20
95% 20
98% 21
99% 21
100% 21 (longest request)
And the node server print out 211 log lines. During various tests the mismatch is variable but always present:
-n 1000 -c 1 -> 1000 log
-n 1000 -c 2 -> ~1000 logs
-n 1000 -c 10 -> ~1001 logs
-n 1000 -c 70 -> ~1008 logs
-n 1000 -c 1000 -> ~1020 logs
It seems that as concurrency increases, the mismatch increases.
I couldn't figure out whether the ab tool performs more http requests or the node server responds more times than necessary.
Could you please help?
Its very strange and I don't get the same results as you on my machine. I would be very surprised if it was ab that was issuing different numbers of actual requests.
Things i would try:
Write a simple server using express rather than hapi. If the issue still occurs you at least know its not a problem with hapi.
Intercept the network calls using fiddler
ab -X localhost:8888 -n 100 -c 30 http://127.0.0.1:3000/test will use the fiddler proxy which will then let you see the actual calls across the network interface. more details
wireshark if you need more power and your feeling brave (I'd only use it if fiddler has let you down)
If after all these you are still finding an issue then it has been narrowed down to an issue with node, I'm not sure what else it could be. Try using node 8 rather than 10.
Using the Fiddler proxy I found that AB tool runs more times than the number of requests to be performed (example: -n 200).
By running a series of consecutive tests:
# 11 consecutive times
/usr/bin/ab -n 200 -c 30 -X localhost:8888 http://localhost:3000/test
Both the proxy and the node server report a total of 2209 requests. It looks like that AB is less imprecise with the proxy in the middle, but still imprecise.
In general, and more important, I never found mismatches between the requests passed through the proxy and the requests received by the node server.
Thanks!
I have a pair of computers running arch linux with rabbitmq message queues and I want to use a shovel to move messages from the queue on the first computer to the queue on the second. Unfortunately I can't seem to create a shovel, or even verify that my rabbitmq.config file is being read.
Computer 1 has an ip address of 192.168.6.66
/etc/rabbitmq/rabbitmq-env.conf
NODENAME=bunny
NODE_IP_ADDRESS=192.168.6.66
NODE_PORT=5672
LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia
RABBITMQ_PLUGINS_DIR=/usr/lib/rabbitmq/lib/rabbitmq_server-2.7.1/plugins
/etc/rabbitmq/rabbitmq.conf
[ {mnesia, [{dump_log_write_threshold, 100}]},
{bunny, [{vm_memory_high_watermark, 0.3}]},
{rabbitmq_shovel,
[{shovels,
[{test_shovel,
[{sources, [{broker, "amqp://shoveluser:shoveluser#192.168.6.64:5672/"}]},
{destinations, [{broker, ""}]},
{queue, <<"observation2">>}
]
}]
}]
}
].
Computer 2 has an ip address of 192.168.6.64
/etc/rabbitmq/rabbitmq-env.conf
NODENAME=bunny
NODE_IP_ADDRESS=0.0.0.0
NODE_PORT=5672
LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia
RABBITMQ_PLUGINS_DIR=/usr/lib/rabbitmq/lib/rabbitmq_server-2.7.1/plugins
When I restart the rabbitmq-server on computer 1 this is the output:
[root#test_toshiba ~]# /etc/rc.d/rabbitmq-server restart
:: Stopping rabbitmq-server daemon [BUSY] Stopping and halting node bunny#localhost ...
...done.
[DONE]
:: Starting rabbitmq-server daemon [BUSY] Activating RabbitMQ plugins ...
********************************************************************************
********************************************************************************
9 plugins activated:
* amqp_client-2.7.1
* erlando-2.7.1
* mochiweb-1.3-rmq2.7.1-git
* rabbitmq_management-2.7.1
* rabbitmq_management_agent-2.7.1
* rabbitmq_mochiweb-2.7.1
* rabbitmq_shovel-2.7.1
* rabbitmq_shovel_management-2.7.1
* webmachine-1.7.0-rmq2.7.1-hg
I expected to see this "config file(s) : /etc/rabbitmq/rabbitmq.config" given the description of the config file documentation here
And after the rabbitmq-server has started I run this command and don't see a shovel:
[root#test_toshiba ~]# rabbitmqctl eval 'rabbit_shovel_status:status().'
[]
...done.
Here is the rabbitmq status
[root#test_toshiba ~]# rabbitmqctl status
Status of node bunny#localhost ...
[{pid,14225},
{running_applications,
[{rabbitmq_shovel,"Data Shovel for RabbitMQ","2.7.1"},
{erlando,"Syntax extensions for Erlang","2.7.1"},
{rabbitmq_shovel_management,"Shovel Status","2.7.1"},
{rabbitmq_management,"RabbitMQ Management Console","2.7.1"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","2.7.1"},
{amqp_client,"RabbitMQ AMQP Client","2.7.1"},
{rabbit,"RabbitMQ","2.7.1"},
{os_mon,"CPO CXC 138 46","2.2.9"},
{sasl,"SASL CXC 138 11","2.2.1"},
{rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.7.1"},
{webmachine,"webmachine","1.7.0-rmq2.7.1-hg"},
{mochiweb,"MochiMedia Web Server","1.3-rmq2.7.1-git"},
{inets,"INETS CXC 138 49","5.9"},
{mnesia,"MNESIA CXC 138 12","4.7"},
{stdlib,"ERTS CXC 138 10","1.18.1"},
{kernel,"ERTS CXC 138 10","2.15.1"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,18530752},
{processes,6813815},
{processes_used,6813800},
{system,11716937},
{atom,428361},
{atom_used,414658},
{binary,182176},
{code,8197217},
{ets,911776}]},
{vm_memory_high_watermark,0.39999999942574066},
{vm_memory_limit,417929625}]
...done.
The logs at /var/log/rabbitmq didn't have any error messages in them.
How can I verify that my config file is being used, and why won't my shovel start?
You need to define a destination for the shovel.
[ {mnesia, [{dump_log_write_threshold, 100}]},
{bunny, [{vm_memory_high_watermark, 0.3}]},
{rabbitmq_shovel,
[{shovels,
[{test_shovel,
[{sources, [{broker, "amqp://shoveluser:shoveluser#192.168.6.64:5672/"}]},
{destinations, [{broker, "amqp://shoveluser:shoveluser#192.168.6.66:5672/"}]},
{queue, >}
]
}]
}]
}
].
I think that "" is not sufficient to specify the localhost broker (in the 'destinations') term. I believe that "amqp://" is required instead.
So, your destinations term should read:
{destinations, [{broker, "amqp://"}]}
Specifying the local broker's username and hostname (the other answer) may also work, but I don't think that's necessary.