One of our linux servers today experienced problems opening outbound requests.
I've reviewed this answer, Increasing the maximum number of tcp/ip connections in linux and it appears as though we are well within the maximum limits.
At the time, netstat -an showed approximately 700 established connections.
Any new socket connections would fail, but nothing would be written to /var/log.
All connections are long-term, and usually open for several hours at a time.
Is there any logging that would help determine what configuration parameter we are bumping against?
nf_conntrack_tcp_timeout_established
It turns out that there’s another timeout value you need to be concerned with. The established connection timeout. Technically this should only apply to connections that are in the ESTABLISHED state, and a connection should get out of this state when a FIN packet goes through in either direction. This doesn’t appear to happen and I’m not entirely sure why.
So how long do connections stay in this table then? The default value for nf_conntrack_tcp_timeout_established is 432000 seconds. I’ll wait for you to do the long division…
Fun times.
I changed the timeout value to 10 minutes (600 seconds) and in a few days time I noticed conntrack_count go down steadily until it sat at a very manageable level of a few thousand.
We did this by adding another line to the sysctl file:
net.netfilter.nf_conntrack_tcp_timeout_established=600
Related
Mongodb docs suggests to reduce tcp keepalive time for better performance:
If you experience socket errors between clients and servers or between members of a sharded cluster or replica set that do not have other reasonable causes, check the TCP keepalive value (for example, the tcp_keepalive_time value on Linux systems). A common keepalive period is 7200 seconds (2 hours); however, different distributions and macOS may have different settings.
However it does not explain why this will help, how it improves performance. From my (limited)understanding, connections created by mongo shards and replicas will have their own keep alive time, which might be way shorter than linux global keep-alive values. so Mongo might break the connection as par it's config and creating new connection should ideally not take too much time.
How will it improve performance by reducing linux tcp keep alive setting?
I thank the shorter keepalive setting on DB side or client side will keep less total connections but with higher percentage of active connections between server-client or server-server, also less connections (pool) will use less resource on server and client side.
I'm running into a lot of trouble when trying to seed a lot of torrents( > 10k) with libtorrent.
They include:
Choking my network connection
Tracker requests timing out(libtorrent tracker error)
When using auto-manage(they go from checking to seeding very slowly, even when my active_seeding is set to unlimited.
I used to let them be automanaged, but I'd find that it makes nearly all of them unavailable.
Here are my current settings:
sessionSettings.setActiveDownloads(5);
sessionSettings.setActiveLimit(-1);
sessionSettings.setActiveSeeds(-1);
sessionSettings.setActiveDHTLimit(5);
sessionSettings.setPeerConnectTimeout(25);
sessionSettings.announceDoubleNAT(true);
sessionSettings.setUploadRateLimit(0);
sessionSettings.setDownloadRateLimit(0);
sessionSettings.setHalgOpenLimit(5);
sessionSettings.useReadCache(false);
sessionSettings.setMaxPeerlistSize(500);
My current method is to loop over all my 10k+ torrents, and run torrent.resume(). When using automanage, this basically only starts ~ 50 of the torrents, and the others start about at a rate of 1 torrent per 10 minutes, which wouldn't work. When not using automanage, it chokes my connection.
BUT, when I do only 30 of them, they all seem to seed correctly, so my next plan is try to resume() them in groupings either with a time delay, or after they've received a tracker_reply.
I tried to garner what I could from this, but don't know what my settings should be specifically:
http://blog.libtorrent.org/2012/01/seeding-a-million-torrents/
I'd really appreciate someone sharing their settings for seeding thousands of torrents,
When not using automanage, it chokes my connection.
Since you say it can run either on a hosted server or domestic internet connection then you will have not much of a choice but to throttle torrent startups. Domestic internet connections are generally behind consumer grade routers and possibly CGNAT, both of which have fairly small NAT tables that will eventually choke from concurrently established TCP connections (peer-peer connections, tracker announces) or UDP pseudo-connections (UDP trackers, µTP, DHT)
So to run many torrents at once you will have to limit all active maintenance traffic of that kind so that the torrents are only started to listen passively for incoming connections.
I have create a tcp listener in C#. I have set timeout to 0 with unlimited time. If there is no activity about 50 hours like on weekends between listener and client, but not disconnect. It dies
Please advice how can i fix this issue.
Thanks in advance.
Routers will typically kill idle TCP connections after a preset amount of time. If your connection is going over the internet you have no control over this. To prevent the problem programmatically detect your idle connections and send a very small amount of information, i.e. byte[] { 0 }; every 5 minutes or so.
This adds no overhead as if the connection is active you do not send the 'keep alive packet'.
Another option is to set the 'KeepAlive' option on the underlying sockets using `Socket.SetSocketOption'. This should perform the same function but I've always found a custom solution more reliable.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I apologize in advance for the length of this question, but I wanted to make it clear what I have already attempted.
Setup:
4 t1.micro EC2 instances (clients)
1 c1.medium EC2 instance (server) in a VPC (behind an Amazon Elastic Load Balancer (ELB))
1 simple node.js server running on the c1.medium (listening on http port 3000, returns simple "hello" html string)
4 node.js servers (1 on each t1.micro) doing distributed load testing with a custom benchmarking suite against the c1.medium
*Clients and server are running Ubuntu and have their file descriptor limits raised to 102400.
Run Case:
The 4 clients try to make n number of connections (simple http get request) a second, ranging from 400 to 1000, until 80,000 requests are made. The server has a hard response wait time, y, tested at 500, 1000, 2000, and 3000 milliseconds before it responds with "hello".
Problem:
At anything more than 500 connections/second, there is a several second (up to 10 or 15) halt, where the server no longer responds to any of the clients, and clients are left idle waiting for the responses. This is consistently at exactly 31449 requests. The clients show the appropriate amount of ESTABLISHED connections (using netstat) holding during this time. Meanwhile, the server shows around 31550 TIME_WAIT connections. After a few seconds this number reported by the server begins to drop, and eventually it starts responding to the clients again. Then, the same issue occurs at some later total request count, e.g. 62198 (though this is not as consistent). The file descriptor count for that port also drops to 0.
Attempted Resolutions:
Increasing ephemeral port range. Default was 32768-61000, or about 30k. Note that despite coming from 4 different physical clients, the traffic is routed through the local ip of the ELB and all ports are thus assigned to that ip. Effectively, all 4 clients are treated as 1 instead of the usually expected result of each of them being able to use the full port range. So instead of 30k x 4 total ports, all 4 are limited to 30k.
So I increased the port range to 1024-65535 with net.ipv4.ip_local_port_range, restarted the server and the following was observed:
The new port range is used. Ports as low as in the 1000's and as high as 65000's are observed being used.
The connections are still getting stuck at 31449.
Total port's in TIME_WAIT state are observed going as high as 50000, after getting stuck around 31550 for 10-15 seconds.
Other tcp configurations were also changed, independent of each other and in conjuction with each other such as tc_fin_timeout, tcp_tw_recycle, tcp_tw_reuse, and several others without any sizable improvement. tcp_tw_recycle seems to help the most, but it makes the status results on the clients print out strangely and in the wrong order, and still doesn't guarantee that the connections don't get stuck. Also I understand that this is a dangerous option to enable.
Question:
I simply want to have as many connections as possible, so that the real server that gets put on the c1.medium has a high baseline when it is benchmarked. What else can I do to avoid hitting this 31449 connection wall short of recompiling the kernel or making the server unstable? I feel like I should be able to go much higher than 500/s, and I thought that increasing the port range alone should have shown some improvement, but I am clearly missing something else.
Thanks!
I'm trying to get an HTTP server I'm writing on to behave well when under heavy load, but I'm getting some weird behavior that I cannot quite understand.
My testing consists of using ab (the Apache benchmark program) over the loopback interface at a concurrency level of 1000 (ab -n 50000 -c 1000 http://localhost:8080/apa), while straceing the server process. Strace both slows processing down well enough for the problem to be readily reproducible and allows me to debug the server internals post completion to some extent. I also capture the network traffic with tcpdump while the test is running.
What happens is that ab stops running a while into the test, complaining that a connection returned ECONNRESET, which I find a bit weird. I could easily buy into a connection timing out since the server might simply not have the bandwidth to process them all, but shouldn't that reasonably return ETIMEDOUT or even ECONNREFUSED if not all connections can be accepted?
I used Wireshark to extract the packets constituting the first connection to return ECONNRESET, and its brief packet list looks like this:
(The entire tcpdump file of this connection is available here.)
As you can see from this dump, the connection is accepted (after a few SYN retransmissions), and then the request is retransmitted a few times, and then the server resets the connection. I'm wondering, what could cause this to happen? Normally, Linux' TCP implementation ACKs data before the reading process even chooses to receive it so long as their is space in the TCP window, so why doesn't it do that here? Are there some kind of shared buffers that are running out? Most importantly, why is the kernel responding with a RST packet all of a sudden instead of simply waiting and letting the client re-transmit further?
For the record, the strace of the process indicates that it never even accepts a connection from the port in this connection (port 56946), so this seems to be something Linux does on its own. It is also worth noting that the server works perfectly well as long as ab's concurrency level is low enough (it works perfectly well up to about 100, and then starts failing intermittently somewhere between 100-500), and that its request throughput is rather constant regardless of the concurrency level (it processes somewhere between 6000-7000 requests per second as long as it isn't being straced). I have not found any particular correlation between the frequency of the problem occurring and my backlog setting to listen() (I'm currently using 128, but I've tried up to 1024 without it seeming to make a difference).
In case it matters, I'm running Linux 3.2.0 on this AMD64 box.
The backlog queue filled up: hence the SYN retransmissions.
Then a slot became available: hence the SYN/ACK.
Then the GET was sent, followed by four retransmissions, which I can't account for.
Then the server gave up and reset the connection.
I suspect you have a concurrency or throughput problem in your server which is preventing you from accepting connections rapidly enough. You should have a thread that is dedicated to doing nothing else but calling accept() and either starting another thread to handle the accepted socket or else queueing a job to handle it to a thread pool. I would then speculate that Linux resets connections on connections which are in the backlog queue and which are receiving I/O retries, but that's only a guess.