In our project, the front end UI makes a lot of http request using httpclient to the backend REST service.
I noticed sometime a http request was never even made to the server (using tcpdump)
Is there some kind of limit in Linux that limits the total tcp socket connection one could have ?
I was playing with lsof, but can't seem to make much out of it...
Sorry for the poorly phrased question.
Related
Some self-written java app is consuming external web services via SOAP requests (JAX-WS).
To minimize negative effects when communicating with those external services, I would like to make everything as fail-proof as possible when talking to missbehaving endpoints.
While most of that can be handled pretty easily, there is one particular scenario, that turns out to be quite tricky: Timeout on a non-responsive endpoint.
(non-responsive = The endpoint builds up the TCP connection, but is not responding to any other requests)
To test this scenario, I require some simple TCP endpoint, which would mimic this behavior - accept TCP connections and stay quiet afterwards (thus, not terminating the connection!).
As this endpoint does not require any particular protocol or service functionality, it can be possibly achieved with some standard tools, or even some few lines of code.
Any advise appreciated.
Looks like this is just as simple as using NetCat with nc -l 4711 to open a port (4711).
It will graciously accept any traffic sent, but does not respond with anything.
I am having trouble understanding the difference between net.createserver and http.createserver in node.js.
I have read the documentation for both methods located at these two urls
https://nodejs.org/api/net.html#/net_net,
https://nodejs.org/api/http.html#/http_class_http_server.
I understand that http.createserver creates an http server. However, the documentation says that net.createserver creates a tcp server. I understand that tcp is the transmission protocol that http is on top of and that http servers are set up to read http request headers. I also understand the concept of even emitters in node.js pretty well. However, I don't understand this notion of a tcp server and why one would be made in node.js. The context is I am coding a chat application example in the "node.js in action" book.
http.createServer() sets up a server that handles the HTTP protocol, which is indeed transmitted over tcp. net.createServer() creates a server that simply understands when a TCP connection has happened, and data has been transmitted, and so on, but doesn't know anything about whether a valid HTTP request has been received, etc.
If you are writing a web server, favor http.createServer() over net.createServer() as it will save you a lot of work. If you are writing some other kind of server, do not use http.createServer().
I don't know much of a Node.js, but I know something about networks. HTTP is a protocol that works on 7th (Application) layer of model OSI. TCP is protocol that works on 4th (Transport) layer of model OSI. As you said, yes HTTP works on top of the TCP. The option of creating HTTP server by http.createServer() is there so you don't have to implement it by yourself by using net.createServer(). The protocol TCP might by used by lot of applications, you might create your own, or implement some different protocol than HTTP, for example: FTP, DNS, SMTP, Telnet and much much more.
Straight from the Node Net documentation. NET is the basic bare-bones server you can create. It's particularly useful for setting up a cluster of servers and allows simple connections but on that you'll want communication protocols, namely HTTP, which HTTP is in fact a NET server at it's core.
The net module provides an asynchronous network API for creating stream-based TCP or IPC servers (net.createServer()) and clients (net.createConnection()).
And from the HTTP documentation. HTTP is the common way to transmit large sets of data as requested by the client and then a response is generated. It's the standard way of communicating over the internet and introduces the concept of handshakes and is done through REST protocol, you know the usual request and response way of communicating.
The HTTP interfaces in Node.js are designed to support many features of the protocol which have been traditionally difficult to use. In particular, large, possibly chunk-encoded, messages. The interface is careful to never buffer entire requests or responses — the user is able to stream data.
Websockets are an upgrade over the HTTP headers and offer low latency and less server load and are a much more minimal conversation. If you're talking peer to peer communication, that's the way you'll want to go.
I am using Socket.IO with a MEAN stack and it's been excellent for low latency and bidirectional communication, but what would be the major draw back for using it for relatively static data as well as dynamic?
My assumption is that it would be more apt for sending more dynamic content. That being said, once a socket connection is established, how relevant is the amount of communication being done? Is there a time where it would be more appropriate to use http instead when a connection is constantly established throughout the user's direct interaction with the application?
Thanks!
WebSockets are a bidirectional data exchange within a HTTP connection. So the question is not if you use HTTP or WebSockets, because there is no WebSockets without HTTP. WebSockets are often confused with simple (BSD) sockets, but WebSockets are actually a socket-like layer inside a HTTP connection which is inside a TCP connection which uses "real" sockets. Or for anybody familiar with OSI layers: it as a layer 4 (transport) encapsulated inside layer 7 (application) and the main reason for doing it this strange way instead of using layer 4 directly is that plain sockets to ports outside of HTTP, SMTP and a few other protocols are no longer possible because of all the port blocking firewalls.
So the question should be more if you use simple HTTP or if you need to use WebSockets (inside HTTP).
With simple HTTP the client sends a request and the server sends the response back. The format is well defined and browser and server transparently support compression, caching and other optimizations. But this simple request-response pattern is limited, because there is no way to push data from server to client or to have a more (BSD) socket like behavior where both client and server can send any data at any time. There are various more or less good workarounds for this, like long polling.
WebSockets gives you a bidirectional communication, which makes it possible for the server to push data to the client or to send data in both directions at any time. And once the WebSocket connection is established by upgrading an existing HTTP connection the overhead for the data itself is very small, much smaller then with a full new HTTP request. While this sounds good you loose all the advantages of simple request-response HTTP like caching at the client or in proxies. And because client and server need resources to keep the underlying TCP connection open it needs more resources, which can be relevant for a busy server. Also, WebSockets might give you more trouble with middleboxes (like proxies or firewalls) then simple HTTP does.
In summary: if you don't need the advantages of WebSockets stay with simple request-response HTTP.
I am having a trouble using HTTP POST when cloudflare is enabled.
It keeps returning 524 timeout.
Failed to load resource: the server responded with a status of 524 (Origin Time-out)
But when I disabled cloudflare, the HTTP POST works fine.
Any idea what might caused this?
UPDATE
I am using AJAX POST, does this got anything to do with ajax?
Thanks.
General causes for a CloudFlare 524 error.
Support should be able to provide more detailed troubleshooting.
Console utility "netstat" shows that some connections from CloudFlare are in CLOSE_WAIT state. Pointing that server just sits without correctly closing connections. Looking to the TCP traffic of my web server with Message Analyzer, I found several connections that was established and http request was sent but that wasn't ever processed by my server.
So we get an answer: the number of simultaneously established connections outnumbered available Accept() calls. So TCP stack connects and wait while application will handle it's connection. Depending on the situation this can never happen, so the client side just drops this connection after a 30 sec timeout without getting any response.
To fix this, you must increase the number of outstanding possible accepts. This parameter can be named as "Max simultaneous connections number" or something similar. Check your web server documentation \ ask the support to find it out.
Also, as an experiment, you can force your server to reply with the "Connection:close" header to each request. This may prevent reaching the active connections limit problem because CloudFlare keep-alive them just way too long.
Also, the more simultaneous requests you do, the more probability to get in troubles. You can try to set some small webserver-side timeout for idle connections.
P.S.: Illustration of CloudFlare's connections number after one client loaded a page:
(http://i.imgur.com/IgwGLCf.png)
Firstly, I'm relatively new to network programming. I want to intercept and delay HTTP traffic before it gets to the server application. I've delved into libnetfilter_queue which gives me all the information I need to delay suitably, but at too low a level. I can delay traffic there, but unless I accept the IP datagrams almost immediately (so sending them up the stack when I want to delay them), they will get resent (when no ACK arrives), which isn't what I want.
I don't want or need to have to deal with TCP, just the payloads it delivers. So my question is how do I intercept traffic on a particular port before it reaches its destination, but after TCP has acknowledged and checked it?
Thanks
Edit: Hopefully it's obvious from the tag and libnetfilter_queue - this is for Linux
Hijack the connections through an HTTP proxy. Google up a good way to do this if you can't just set HTTP_PROXY on the client, or set up your filter running with the IP and port number of the current server, moving the real server to another IP.
So the actual TCP connections are between the client and you, then from you to the server. Then you don't have to deal with ACKs, because TCP always sees mission accomplished.
edit: I see the comments on the original already came up with this idea using iptables to redirect the traffic through your transparent proxy process on the same machine.
Well I've done what I suggested in my comment, and it works, even if it did feel a long-winded way of doing it.
The (or a) problem is that the web server now, understandably, thinks that every request comes from localhost. Really I would like this delay to be transparent to both client and server (except in time of course!). Is there anything I can do about this?
If not, what are the implications? Each HTTP session happens through a different port - is that enough for them to be separated completely as they should be? Presumably so considering it works when behind a NAT where the address for many sessions is the same.