Some self-written java app is consuming external web services via SOAP requests (JAX-WS).
To minimize negative effects when communicating with those external services, I would like to make everything as fail-proof as possible when talking to missbehaving endpoints.
While most of that can be handled pretty easily, there is one particular scenario, that turns out to be quite tricky: Timeout on a non-responsive endpoint.
(non-responsive = The endpoint builds up the TCP connection, but is not responding to any other requests)
To test this scenario, I require some simple TCP endpoint, which would mimic this behavior - accept TCP connections and stay quiet afterwards (thus, not terminating the connection!).
As this endpoint does not require any particular protocol or service functionality, it can be possibly achieved with some standard tools, or even some few lines of code.
Any advise appreciated.
Looks like this is just as simple as using NetCat with nc -l 4711 to open a port (4711).
It will graciously accept any traffic sent, but does not respond with anything.
Related
Problem Description/Context
I have a nodeJS-based application using Axios to make HTTP requests (Outbound REST API calls) against a web service (say https://any.example.restapis.com). And these HTTP requests occasionally used to take > 1-minute latency. After some debugging - when we tried httpsAgent property to keep the HTTP connections live (persistent) it did the trick and now the APIs are taking < 1 second and the application is working OK. Basically, my understanding is with this property the TCP connections used by the HTTP calls are persistent now and the httpsAgent is opening multiple socket connections against the web service (i.e; it's keeping the connections alive based on default configs and opening multiple TCP connections based on the load as required - basically maintaining a pool of connections)
httpsAgent: new https.Agent({ keepAlive: true }),
Question
We are not yet sending the full traffic 100% to the micro-service (just 1%). So I would like to understand in detail what is happening underneath to make sure the fix is indeed complete and my micro-service will scale to full traffic.
So, can anyone please let me know after SSH into the pod's container how I can check if my node JS application is indeed making number of TCP (socket) connections against the web service rather than just using single TCP connection but keeping it alive (I tried to use netstat -atp command like below - however I'm not able to make the connection). So, it will great if anyone help me with how to check the number of TCP connections made by my micro-service.
// example cmd -
// Looking at cmds like netstat, lsof as they may (hoping!) give me details that I want!
netstat -atp | grep <my process ID>
In a microservices architecture, the number of server to server connections increases dramatically compared to alternative setups. Interactions which would traditionally have been an in-memory process in one application now often rely on remote calls to other REST based services over HTTP, meaning it is more important than ever to ensure these remote calls are both fast and efficient.
The netstat command is used to show network status.
# netstat -at : To list all tcp ports.
# netstat -lt : To list only the listening tcp ports.
It is used more for problem determination than for performance measurement. However, the netstat command can be used to determine the amount of traffic on the network to ascertain whether performance problems are due to network congestion.
I am having trouble understanding the difference between net.createserver and http.createserver in node.js.
I have read the documentation for both methods located at these two urls
https://nodejs.org/api/net.html#/net_net,
https://nodejs.org/api/http.html#/http_class_http_server.
I understand that http.createserver creates an http server. However, the documentation says that net.createserver creates a tcp server. I understand that tcp is the transmission protocol that http is on top of and that http servers are set up to read http request headers. I also understand the concept of even emitters in node.js pretty well. However, I don't understand this notion of a tcp server and why one would be made in node.js. The context is I am coding a chat application example in the "node.js in action" book.
http.createServer() sets up a server that handles the HTTP protocol, which is indeed transmitted over tcp. net.createServer() creates a server that simply understands when a TCP connection has happened, and data has been transmitted, and so on, but doesn't know anything about whether a valid HTTP request has been received, etc.
If you are writing a web server, favor http.createServer() over net.createServer() as it will save you a lot of work. If you are writing some other kind of server, do not use http.createServer().
I don't know much of a Node.js, but I know something about networks. HTTP is a protocol that works on 7th (Application) layer of model OSI. TCP is protocol that works on 4th (Transport) layer of model OSI. As you said, yes HTTP works on top of the TCP. The option of creating HTTP server by http.createServer() is there so you don't have to implement it by yourself by using net.createServer(). The protocol TCP might by used by lot of applications, you might create your own, or implement some different protocol than HTTP, for example: FTP, DNS, SMTP, Telnet and much much more.
Straight from the Node Net documentation. NET is the basic bare-bones server you can create. It's particularly useful for setting up a cluster of servers and allows simple connections but on that you'll want communication protocols, namely HTTP, which HTTP is in fact a NET server at it's core.
The net module provides an asynchronous network API for creating stream-based TCP or IPC servers (net.createServer()) and clients (net.createConnection()).
And from the HTTP documentation. HTTP is the common way to transmit large sets of data as requested by the client and then a response is generated. It's the standard way of communicating over the internet and introduces the concept of handshakes and is done through REST protocol, you know the usual request and response way of communicating.
The HTTP interfaces in Node.js are designed to support many features of the protocol which have been traditionally difficult to use. In particular, large, possibly chunk-encoded, messages. The interface is careful to never buffer entire requests or responses — the user is able to stream data.
Websockets are an upgrade over the HTTP headers and offer low latency and less server load and are a much more minimal conversation. If you're talking peer to peer communication, that's the way you'll want to go.
I'm writing a piece of P2P software, which requires a direct connection to the Internet. It is decentralized, so there is no always-on server that it can contact with a request for the server to attempt to connect back to it in order to observe if the connection attempt arrives.
Is there a way to test the connection for firewall status?
I'm thinking in my dream land where wishes were horses, there would be some sort of 3rd-party, public, already existent servers to whom I could send some sort of simple command, and they would send a special ping back. Then I could simply listen to see if that arrives and know whether I'm behind a firewall.
Even if such a thing does not exist, are there any alternative routes available?
Nantucket - does your service listen on UDP or TCP?
For UDP - what you are sort of describing is something the STUN protocol was designed for. It matches your definition of "some sort of simple command, and they would send a special ping back"
STUN is a very "ping like" (UDP) protocol for a server to echo back to a client what IP and port it sees the client as. The client can then use the response from the server and compare the result with what it thinks its locally enumerated IP address is. If the server's response matches the locally enumerated IP address, the client host can self determinte that it is directly connected to the Internet. Otherwise, the client must assume it is behind a NAT - but for the majority of routers, you have just created a port mapping that can be used for other P2P connection scenarios.
Further, you can you use the RESPONSE-PORT attribute in the STUN binding request for the server to respond back to a different port. This will effectively allow you to detect if you are firewalled or not.
TCP - this gets a little tricky. STUN can partially be used to determine if you are behind a NAT. Or simply making an http request to whatismyip.com and parsing the result to see if there's a NAT. But it gets tricky, as there's no service on the internet that I know of that will test a TCP connection back to you.
With all the above in mind, the vast majority of broadband users are likely behind a NAT that also acts as a firewall. Either given by their ISP or their own wireless router device. And even if they are not, most operating systems have some sort of minimal firewall to block unsolicited traffic. So it's very limiting to have a P2P client out there than can only work on direct connections.
With that said, on Windows (and likely others), you can program your app's install package can register with the Windows firewall so your it is not blocked. But if you aren't targeting Windows, you may have to ask the user to manually fix his firewall software.
Oh shameless plug. You can use this open source STUN server and client library which supports all of the semantics described above. Follow up with me offline if you need access to a stun service.
You might find this article useful
http://msdn.microsoft.com/en-us/library/aa364726%28v=VS.85%29.aspx
I would start with each os and ask if firewall services are turned on. Secondly, I would attempt the socket connections and determine from the error codes if connections are being reset or timeout. I'm only familiar with winsock coding, so I can't really say much for Linux or mac os.
I'm going through the Whitepaper by RADVISION on NAT/Firewall traversal for H.323 endpoints.
It is suggested there to use ITU-T H.460.18,17 and 19.
460.17 is very clear way for NAT traversal, but I'm not so clear about the 460.18.
Both present a clear solution for Firewall, but how is 460.18 a solution for NAT traversal?
Regards,
H.460.18 works by opening pinholes when moving from one protocol/network connection to the next.
H.323 works in the following classic way to connect a call:
RAS is used over UDP to register to the gatekeeper
Q.931 is used over TCP (usually) to initiate a call
H.245 is used to negotiate media capabilities and open media channels
RTP/RTCP is used to send actual media
Now, to be able to open up Q.931 and H.245, you need the endpoint to be listening on a TCP address for incoming connections. If the endpoint is behind a NAT - that will be impossible to achieve.
So H.460.18 adds special messages to get these TCP connections from the inside out (=reverse).
On RAS, when a new TCP connection needs to be opened for Q.931, a RAS SCI (ServiceControlIndication) message will be sent to the endpoint so that the endpoint will open up the TCP connection for Q.931 instead of just waiting to get an incoming connection.
On Q.931, when a new H.245 connection needs to be opened, it is initiated today already on Q.931; but now it will always be done from the endpoint behind the NAT to a public address.
To sum it up:
H.460.17 uses a single connection outbound from the endpoint to the gatekeeper and then just tunnels everything on top of it.
H.460.18 just opens up a new pinhole from one protocol to the next by having the endpoint behind a NAT do the connecting instead of doing the listening.
The problem with H.460.17 is that virtually no H.323 equipment supports it.
H.460.18 works nicely, even across vendors. It lets the endpoint behind the firewall poke a whole and then uses that whole for both ways of communication. Its rather simple when you read though the standards document. But beware that it is patented by Tandberg, so you have to get a (free) license before you can implement it.
You can look at the GNU Gatekeeper to see the details how H.460.18 gets through the firewall.
I understand the need for putting a web server in a DMZ and blocking inbound traffic to all ports except 80 and 443. I can also see why you should probably also block most outbound traffic in case the server is compromised.
But is it necessary to block outbound HTTP traffic over port 80? If so, why? A lot of web applications these days rely on sending/retrieving data from external web services and APIs, so blocking outbound traffic over port 80 would prevent this capability. Is there a security concern that's valid enough to justify this?
The only reason I can think of is if your machine is somehow compromomised remotely then it won't be able to DDoS another website on port 80. It's not something I normally do though.
Rather then blocking it, throttle it. Use iptables -m limit.
I have several web apps that invoke external web services, so I would say it's a bad idea to block output HTTP traffic. If you're concerned with security, you could block it and allow for only certain destinations.
Depending on your SQL version, you could have certificate authentication time out issues with SQL server 2005.
First - I agree with #vartec on throttling "Rather then blocking it, throttle it. Use iptables -m limit" as at least part of the solution.
However I can offer another reason to not block port 80 outbound at all times. If you have automatic security updates turned on the server can't reach out to PPAs over port 80 to initiate a security update. Thus if you have automatic security updates set up they won't run. On ubuntu auto-security updates are turned on in 14.04 LTS with:
sudo apt-get install unattended-upgrades update-notifier-common && \
sudo dpkg-reconfigure -plow unattended-upgrades
(then select "YES")
More graceful solutions would be ansible scripts opening the port automatically, possibly also modifying an AWS security group rule via the CLI in addition to iptables if you are at AWS. I prefer modifying my outbound rules temporarily via AWS CLI initiated by a stealth box. This forces logging the update up in my AWS S3 log buckets but never shows up in the logs on the server itself. Further the server that initiates the update doesn't even have to be in the private subnet ACL.
Maybe do both? You have to figure at times an attack is going to relay off an internal IP in your subnet so there is merit to doubling down while preserving the ability to automate backups and security updates.
I hope this helps. If not reply and provide more code examples to be more specific and exact. #staysafe !
If the machine is compromised and outbound traffic on port 80 is allowed, it would make it easier for intruders to send back harvested data to themselves. Allowing outbound traffic means you can initiate a connection from your machine to the outside world. A better approach would be allowing outbound traffic only to certain web sites/addresses that you trust (i.e. Microsoft Windows Update, Google reCAPTCHA) rather than any destination in the world.
what do you mean with blocking outbound traffic over port 80.
You have two possibilities. Gernerate Dynamic Rules which allow communication from client to your webserver for this session. Search for Stateful firewall rules.
Or you generally allow established Connections to communicate in and outgoing with each other.
If you generally block all outbound traffic over Port 80 your Webserver could not reply to any client.
The other way around, if your Webserver needs to get some API, e.g. a jquery library he wont use port 80 as his Port to communicate with the Webserver who holds the API.
Your Webserver would normally choose a port > 1024 and use it for his request to get the API from the remote Server.
So blocking all traffic over port 80 (as your port you connecting from) would not prevent your Server from sending any requests for apis and such things. because he doesnt use port 80 when he acts as a client.