I think I am doing something wrong with HAProxy conf because my throughput drops to 25% in a real-world test done with HAProxy and one single AWS instance. Following is my relevant (extremely simple) configuration:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 30000
frontend localnodes
bind *:80
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
hash-type consistent
option httpchk /health
server w1 xx.xx.xx.xx:80 check id 1
I had enabled logging. A typical entry in log looks like this:
Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43908
[02/Dec/2016:09:29:05.940] localnodes nodes/w1 38/0/0/1/41 200 130 - -
---- 36/36/12/2/0 0/0 "GET /ep?key=123&message=XXQSYI HTTP/1.1" Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43920
[02/Dec/2016:09:29:05.941] localnodes nodes/web01 39/0/0/0/40 200 160
- - ---- 35/35/11/0/0 0/0 "GET /q1?key=123&val=123 HTTP/1.1" Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43933
[02/Dec/2016:09:29:05.955] localnodes nodes/web01 24/0/0/1/26 200 134
- - ---- 34/34/11/1/0 0/0 "GET /q1?key=123&val=123 HTTP/1.1"
My throughput is 25% of what a direct traffic to my instance would be. This is terrible performance. Am I doing something really wrong?
EDIT
Going down the log, some logs clearly show that time taken to reach server from HAProxy is too high
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39789 [02/Dec/2016:10:56:58.729] main app/app1 0/0/1000/1/1002 200 449 - - ---- 13/13/13/7/0 0/0 "GET / HTTP/1.1"
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39803 [02/Dec/2016:10:56:58.730] main app/app1 0/0/999/1/1000 200 377 - - ---- 12/12/12/7/0 0/0 "GET / HTTP/1.1"
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39804 [02/Dec/2016:10:56:58.730] main app/app1 0/0/999/1/1000 200 277 - - ---- 11/11/11/7/0 0/0 "GET / HTTP/1.1"
From your log, most of your time is being spent connecting to the server. For example, you spend 1000, 999 and 999 milliseconds connecting. This may have to do with that you are closing the connection to the server immediately after each transaction by using option http-server-close. So, the TCP connection has to be re-established each time (if this is the same client between requests).
Overall, it looks like you're spending about 1 second per request, which doesn't sound horrible to me. What were you seeing before using HAProxy?
Related
I want that in the HAProxy server,
when the client sends a request to the server with port 8080, the HAProxy server redirects the port that the client requested with to the IP of the same HAProxy server with port 2095.
I use the following code but it doesn't work.
please guide me
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 50s
timeout server 500s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen http80new
bind *:8080
mode http
acl path_check1 path
server webserver1 ip:2095
Edit: I found this to be a networking issue, but I don't have an answer on how to repair it yet, so hopefully someone else knows anything about it:
When i'm inside the Nginx container, I can query node.js like this:
curl http://192.168.65.2:3001/api/getTest
and that works, but has the same erratic behaviour as Nginx has below. So it indeed does mostly time-out on the backend for some networking reason I do not understand.
So when I run ab from the Nginx container:
> ab -n 10000 -c 5 http://192.168.65.2:3001/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.65.2 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16 requests completed
Which indeed is the same behaviour as I saw from Nginx -> backend. Considering node.js works fine (seeing from the below ab on node.js directly; I even ran both simultaneously; the node.js version always finished with 0 errors, the ab from Docker never finishes correctly as shown above and below).
------- Old question, needed to understand the full case:
I have a standard nginx docker image:
image: nginx
port mapping:
- "8080:80"
I have a node server running outside docker with node v17.3.0 on port 3001.
I proxy pass requests from nginx 8080 to 3001 with the following configuration;
upstream backend {
server host.docker.internal:3001;
keepalive 32;
}
location /api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://backend;
proxy_set_header Host $http_host;
}
When I do requests with curl or via the web frontend, it works on both 8080 and 3001 as expected. However, I noticed that on :8080, even with very little subsequent requests , Nginx hangs; average when I manually just try curl requests, after average 10 in a row Nginx hangs (it seems to be at around 10 subsequent request 'blocks' as ab hangs always at around 50 when concurrency=5, 30 when currency is 3 etc completed requests and manually, it always hangs after 10 completed requests). And it hangs longer than 30s, always.
So I get try:
> curl http://localhost:8080/api/getTest
<myjsonbody>
> curl http://localhost:3001/api/getTest
<myjsonbody>
But sometimes(!), simply:
> curl http://localhost:8080/api/getTest
And nothing for 30s+ seconds after which it sometimes does return the correct result and sometimes not:
> time curl http://localhost:8080/api/getTest
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.21.4</center>
</body>
</html>
0.01s user 0.01s system 0% cpu 1:00.04 total
Now I look at the node.js logs, when sending requests to nginx and the ones that hang do not arrive at node.js until 30-60s later or, in the latter case, it doesn't arrive at all at node.js.
So, to test I try the following ab version:
> ab -n 10000 -c 5 http://localhost:8080/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 48 requests completed
And, directly to node.js:
ab -n 10000 -c 5 http://localhost:3001/api/getTest
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:
Server Hostname: localhost
Server Port: 3001
Document Path: /api/getTest
Document Length: 41017 bytes
Concurrency Level: 5
Time taken for tests: 2.965 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 412610000 bytes
HTML transferred: 410170000 bytes
Requests per second: 3372.14 [#/sec] (mean)
Time per request: 1.483 [ms] (mean)
Time per request: 0.297 [ms] (mean, across all concurrent requests)
Transfer rate: 135877.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 0 1 3.7 1 118
Waiting: 0 1 3.1 1 118
Total: 0 1 3.7 1 119
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 2
95% 2
98% 3
99% 4
100% 119 (longest request)
All requests succeed directly to node.js while nginx almost immediately hangs. I tried direct node.js with many more requests concurrently and in total; it simply never hangs. Nginx almost immediately always hangs.
Any ideas? I wouldn't know how to even debug it. I used Wireshark to trace the data and it goes into Nginx when it fails, but often doesn't actually gets sent to node.js (and then it gives the above gateway error) while when it does reach node.js, it is as:
Request received by Nginx
40s nothing
Request received by Node.js from Nginx
Immediate response by Node.js to Nginx
Nginx delivers endresult after 40s+some ms.
I tried a lot of configs but it doesn't fix the problem.
Could it be a networking problem? It feels like it but how would I find out further than what I have tried already?
There is a shared printer at my workplace. We send jobs and then go to the printer and authenticate, so printer prints your documents only when you present at it. Periodically, we change domain passwords, so I also have to change it in /etc/cups/printers.conf (windows users just change domain password). So, that's how it works.
But, suddenly, it stop receive my jobs. When I send job I have no errors and have this:
sudo tail /var/log/cups/access_log
localhost - - [14/Apr/2015:12:15:14 +0300] "POST /printers/Generic-PCL-6-PCL-XL HTTP/1.1" 200 499 Create-Job successful-ok
localhost - - [14/Apr/2015:12:15:14 +0300] "POST /printers/Generic-PCL-6-PCL-XL HTTP/1.1" 200 1273674 Send-Document successful-ok
localhost - - [14/Apr/2015:12:17:59 +0300] "POST / HTTP/1.1" 200 183 Renew-Subscription successful-ok
On cups page in browser it shows state for job - "Pending since (date/time)".
It seems like job was sent successfully, but when I came to printer I've got nothing and no job in my queue. Our IT support fix problems only for Windows users and who on Linux - on their own. So, I don't know what to do and what logs I should inspect. Please, help.
Probably, some updates broke it down. But I have found another solution - I add printer not via samba, but via lp and it doesn't ask username/password:
cat /etc/cups/printers.conf
# Printer configuration file for CUPS v1.5.3
# Written by cupsd
# DO NOT EDIT THIS FILE WHEN CUPSD IS RUNNING
<DefaultPrinter KonicaMinolta>
UUID urn:uuid:0f60c08a-ecfb-326a-421c-86aa3519147b
Info MyCompany Office printer
Location WestCorridor
MakeModel Generic PostScript Printer Foomatic/Postscript (recommended)
DeviceURI lpd://Company_printer_server_address/lp
State Idle
StateTime 1429265417
Type 8433692
Accepting Yes
Shared Yes
JobSheets none none
QuotaPeriod 0
PageLimit 0
KLimit 0
OpPolicy default
ErrorPolicy stop-printer
</Printer>
If somebody can provide another solution or some explanation why it is so, I will be glad to see.
As far as debugging you can view more data in your CUPS logs if you edit your /etc/cups/cupsd.conf file, find the section "loglevel" change "info" to "debug"
Then you should restart CUPS with:
/etc/init.d/cups restart
Then your log will be in
/var/log/cups/error_log
I want to bulkload my cassandra data from node A to node B.
when I set the 'listen_address' of each cassandra.yaml file to localhost,
they do not show error on console but the data is never transmitted.
when I set each node's listen address to their own local network[eth1 ipv4]address (192.168....), I get the following error.
I can read from this error log that the application is trying to access to port 1..4
and I do not have no idea what on earth is going on.
each node is on the virtual machine on the Virtual Box Hypervisor. Both OS is centOS.
[vagrant#localhost conf]$ ../bin/sstableloader -v -d 192.168.33.12 -p 9160 /db/data/m
oomin/hoahoa2/
Streaming revelant part of /db/data/moomin/hoahoa2/moomin-hoahoa2-hf-69-Data.db to [/192.168.33.12]
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:42,655 Failed attempt 1 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:46,658 Failed attempt 2 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:54,666 Failed attempt 3 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 16000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)]
Here is my cassandra.yaml (the cassandra.yaml of target file is also configured the same way)
# communicate!
#
# Leaving it blank leaves it up to InetAddress.getLocalHost(). This
# will always do the Right Thing *if* the node is properly configured
# (hostname, name resolution, etc), and the Right Thing is to use the
# address associated with the hostname (it might not be).
#
# Setting this to 0.0.0.0 is always wrong.
listen_address: 192.168.33.12
#listen_address: localhost
rpc_address: 0.0.0.0
# port for Thrift to listen for clients on
rpc_port: 9160
# enable or disable keepalive on rpc connections
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
emory usage without a impact on performance.
index_interval: 128
Can anybody give me advice? I am really suffering as hell.
I'm trying to start up HAProxy on Cygwin. When I do so, I get the following response:
$ /usr/local/sbin/haproxy -f /usr/local/sbin/haproxy.cfg
[ALERT] 313/180006 (4008) : cannot change UNIX socket ownership
(/tmp/haproxy.socket). Aborting.
[ALERT] 313/180006 (4008) : [/usr/local/sbin/haproxy.main()]
Some protocols failed to start
their listeners! Exiting.
It looks like it's due to the following line in my config file, when I rip this it starts up:
stats socket /tmp/haproxy.socket uid haproxy mode 770 level admin
The entire config:
global
log 127.0.0.1 local0 info
stats socket /tmp/haproxy.socket uid haproxy mode 770 level admin
maxconn 1000
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 1000
timeout connect 5s
timeout client 120s
timeout server 120s
listen rabbitmq_local_cluster 127.0.0.1:5555
mode tcp
balance roundrobin
server rabbit_0 127.0.0.1:5673 check inter 5000 rise 2 fall 3
server rabbit_1 127.0.0.1:5674 check inter 5000 rise 2 fall 3
listen private_monitoring 127.0.0.1:8100
mode http
option httplog
stats enable
stats uri /stats
stats refresh 5s
Any ideas would be appreciated, Thanks!
Simple answer, as I expected. My user "haproxy" which is referenced in the problematic line:
stats socket /tmp/haproxy.socket uid haproxy mode 770 level admin
Did not have necessary permissions on the local machine. Once this was set up, it started up fine.
Nice to know that it still works on cygwin, what version of haproxy is this ? I did not know that UNIX sockets were supported on windows BTW. Or maybe they're emulated via named pipes ?