The company I work for has a web application running solidly in our Hosted Datacentre for years, no hiccups. Using route 53, NGINX, etc.
We started building in Azure recently, and are noticing weird connection drops in random times. No pattern we can find except that, the drops only occur with the Azure firewall involved.
Has anyone encountered this? The client-side traffic flow is as follows:
Client Machine --> Route 53 --> Azure Firewall --> NGINX server --> Azure application server
We've done multiple connection tests with our apps - to keep it short, internally within the Azure environment, there's no problems.
Connection tests involved just the app server stack, just the internal NGINX server, route 53 + nginx server (bypassing firewall) is fine.
It seems something specific with the firewall and how it keeps connections. I could provide some scrubbed logs, but not sure where to look.
I've found this, not sure it could be related:
https://github.com/wbuchwalter/azure-content/blob/master/includes/guidance-tcp-session-timeout-include.md
since the FW was only being used for port forwarding, we've bypassed it and that solved the issue. i suspect nginx didn't like the requests or didn't know how to forward back through to the firewall because, of, ip encapsulation from the firewall. there may be a way to solve that but we didn't investigate further.
Related
We have a Nodejs server that communicates with the websocket protocol (WebRTC, socket.io).
During our development on Heroku, we did not encounter any particular problem.
However, we encountered problems during the deployment of our application on Azure:
The client / server communication is unstable, after analysis we noticed that there is a failure to communicate in websocket and that the transport protocol used is 'polling':
websocket.js:112 WebSocket connection to
'wss://hote.fr/socket.io/?EIO=3&transport=websocket&sid=EgjKLAtp89wrBKMzAAAG'
failed: Error during WebSocket handshake: Unexpected response code:
503
When you activate the "Websocket" function on the administration, the site becomes even more unstable and very long and communication is impossible. The problem of websocket communication is not solved with this function. Example
After having informed us we did disable the websockets in the web.config but without change.
Note that with the same code, everything works well under heroku and the protocol used is websocket.
Has anyone ever encountered this problem, and has a solution?
I'm sorry about my English.
Thank you in advance for your time.
You need do the following to make it work on Azure App Service.
Enable Web sockets via the Azure portal.
Disable the IIS WebSocket module to allow Node.js to provide its own
WebSocket implementation by add this to your web.config file:
<webSocket enabled="false" />
Tell Socket.IO to use WebSocket only instead of a few XHR requests by adding this to the Node.js server:
io.set('transports', ['websocket']);
And on the client add this:
var socket = io({transports: ['websocket']});
Try the suggestions outlined by Aaron to narrow the issue, I would like to highlight a few restrictions on the Azure Sandbox to help isolate the issue further:
1.In App Service, limits are enforced for the maximum number of outbound connections that can be made for each VM instance.
As mentioned in the document Cross-VM numerical limits:
“These limits apply only for customers of Basic or higher plans; in other words, customers running on their own dedicated VMs. These limits are there to protect the entire VM even though one particular site may be with its limits described above. The limits are different depending on the size of VM configured.”
This error also might occur if you try to access a local address from your application.
As mentioned in the document Local address requests:
“Connection attempts to local addresses (e.g. localhost, 127.0.0.1) and the machine's own IP will fail, except if another process in the same sandbox has created a listening socket on the destination port.
Rejected connection attempts, such as the following example which attempts to connect to 127.0.0.1:80, from .NET will result in the following exception:
Exception Details: System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions 127.0.0.1:80.”
3.For more information about outbound connections in your web app, see the blog post about outgoing connections to Azure websites.
so I have 0% experience with web programming, and the project I am working on doesn't have anything to do with it, but I hit a small roadblock and need to solve a small port problem.
So we want to build a cluster of GPU machines on Azure for some Deep Learning calculations, and want to install some applications on them and let our scientists use the app' GUIs to launch and monitor their jobs. The problem is that an app A for example runs on port 5050, but our firewall doesn't let us communicate to unusual ports. The problem is easy to fix from the Azure side, but our IT team won't let us modify our security policies.
That's why I need to find a hacky and fast solution to overcome this, I don't want to spend my whole internship doing something complicated for it, just something that does the job.
What I thought about was to have some kind of server running on the machines (let's say Machine A has public IP address ipbA and private IP ipvA) that when we type "http://ipba/app1" on our browser, the server on A will fetch the page "http://ipva:5050" (or "----://ipba/app2" -> "----://ipva:5051") and display it, but does this work if the page needs to be interactive because we would like to launch jobs?
I have no clue how to do this, if you could please just tell me what I should look into, google and read about, or if there is an easier way to handle it, (maybe some VPN stuff, which I don't prefer since we're moving towards a hybrid cloud architecture and I don't think we would want to VPN into all the different cloud platforms) that would be awesome :)
Two common solutions for your problem:
Set-up a reverse proxy on a standard port (such as 80 or 443 if you want some SSL certificates headaches).
All your domain names will point to the reverse proxy (single IP) but the reverse proxy will forward the traffic transparently to the real servers on their special ports.
https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html
https://www.digitalocean.com/community/tutorials/how-to-use-apache-http-server-as-reverse-proxy-using-mod_proxy-extension
For the technical details, in short: you keep in file(s) the configuration for each domain or subdomain and where they should be forwarded.
Chain of events:
User types http://interface-1.company.com
Browser resolves interface-1.company.com (DNS: IP Reverse Proxy)
Browser connects on reverse proxy (port 80)
Reverse proxy reads configuration which says where to forward
Proxy forwards request to realserver.company.com:5050
Realserver relays response to reverse proxy
reverse proxy relays response to browser
I think that is what you are trying to achieve.
Set-up a VPN service which will be connectible through the proxy of your company and provide VPN clients to the end-users. OpenVPN clients can use an HTTPS proxy connection (your company proxy) to establish connexion to a remote VPN.
Once connected on the VPN, everyone uses the VPN's IP address + firewall policy, and are therefore no more restricted by the company's firewalling policy. Any kind of traffic can also be forwarded. This is harder to set up and your security team might not accept it. However, it's a fully functional solution and it can also offer additional security features if implemented properly.
I do not recommend going this way for all the paperwork that would involve.
I have a windows server 2012 VPS running a web app behind Cloudflare. The app needs to initiate outbound connections based on user actions (eg upload image from URL). The problem is that this 'leaks' my server's IP address and increases risk of DDOS attacks.
So I would like to prevent my server's IP from being discovered by setting up a forward proxy. So far my research has shown that this is no simple task, and would involve setting up another VPS to act as a proxy.
Does this extra forward proxy VPS have to be running windows ? Are their any paid services that could act as a forward proxy for my server (like cloudflare's reverse proxy system)?
Also, it seems that the suggested IIS forward proxy plugin, Application Request Routing, does not work for HTTPS.
Is there a solution for both types of outgoing (HTTPS + HTTP) requests?
I'm really lost here, so any help or suggestions would be appreciated.
You are correct in needing a "Forward Proxy". A good analogy for this is the proxy settings your browser has for outbound requests. In your case, the web application behaves like a desktop browser and can be configured to make the resource request through a proxy.
Often you can control this for individual requests at the application layer. An example of doing so with C#: C# Connecting Through Proxy
As far as the actual proxy server: No, it does not need to run Windows or IIS. Yes, you can use a proxy service. The vast majority of proxy services are targeted towards consumers and are used for personal privacy or to get around network restrictions. As such, I have no direct recommendations.
Cloudflare actually has recommendations regarding this: https://blog.cloudflare.com/ddos-prevention-protecting-the-origin/.
Features like "upload from URL" that allow the user to upload a photo from a given URL should be configured so that the server doing the download is not the website origin server.
This may be a more comfortable risk mitigator, as it wouldn't depend on a third party proxy service. A request for upload could be handled as a web service call to a dedicated "file downloader" server. Keep in mind that if you have a queued process for another server to do the work, and that server is hosted in the same infrastructure, both might be impacted by a DDoS, depending on the type of DDoS.
Your question implies that you may be comfortable using a non-windows server. Many softwares exist that can operate as a proxy(most web servers), but suffer from the same problem as ARR - lack of support for the HTTP "CONNECT" verb, which is used by modern browsers to start an HTTPS connection before issuing a "GET". SQUID is very popular, open source, and supports everything to connect to.. anything. It's not trivial to set up. Apache also has support for this in "mod_proxy_connect", but I have no experience in that and the online documentation isn't very robust. It's Apache, though, so it may be worth the extra investigation.
I'm running a nodejs webserver on azure using the express library for http handling. We've been attempting to enable cloudflare protection on the domains pointing to this box, but when we turn cloudflare proxying on, we see cycling periods of requests succeeding, and requests failing with a 524 error. I understand this error is returned when the server fails to respond to the connection with an HTTP response in time, but I'm having a hard time figuring out why it is
A. Only failing sometimes as opposed to all the time
B. Immediately fixed when we turn cloudflare proxying off.
I've been attempting to confirm the TCP connection using
tcpdump -i eth0 port 443 | grep cloudflare (the request come over https) and have seen curl requests fail seemingly without any traffic hitting the box, while others do arrive. For further reference, these requests should be and are quite quick when they succeed, so I'm having a hard time believe the issue is due to a long running process stalling the response.
I do not believe we have any sort of IP based throttling or firewall (at least not intentionally?)
Any ideas greatly appreciated, thanks
It seems that the issue was caused by DNS resolution.
On Azure, you can configure a custom domain name for your created webapp. And according to the CloudFlare usage, you need to switch the DNS resolution to CloudFlare DNS server, please see more infomation for configuring domain name https://azure.microsoft.com/en-us/documentation/articles/web-sites-custom-domain-name/.
You can try to refer to the faq doc of CloudFlare How do I enter Windows Azure DNS records in CloudFlare? to make sure the DNS settings is correct.
Try clearing your cookies.
Had a similar issue when I changed cloudflare settings to a new host but cloudflare cookies for the domain was doing something funky to the request (I am guessing it might be trying to contact the old host?)
I understand the need for putting a web server in a DMZ and blocking inbound traffic to all ports except 80 and 443. I can also see why you should probably also block most outbound traffic in case the server is compromised.
But is it necessary to block outbound HTTP traffic over port 80? If so, why? A lot of web applications these days rely on sending/retrieving data from external web services and APIs, so blocking outbound traffic over port 80 would prevent this capability. Is there a security concern that's valid enough to justify this?
The only reason I can think of is if your machine is somehow compromomised remotely then it won't be able to DDoS another website on port 80. It's not something I normally do though.
Rather then blocking it, throttle it. Use iptables -m limit.
I have several web apps that invoke external web services, so I would say it's a bad idea to block output HTTP traffic. If you're concerned with security, you could block it and allow for only certain destinations.
Depending on your SQL version, you could have certificate authentication time out issues with SQL server 2005.
First - I agree with #vartec on throttling "Rather then blocking it, throttle it. Use iptables -m limit" as at least part of the solution.
However I can offer another reason to not block port 80 outbound at all times. If you have automatic security updates turned on the server can't reach out to PPAs over port 80 to initiate a security update. Thus if you have automatic security updates set up they won't run. On ubuntu auto-security updates are turned on in 14.04 LTS with:
sudo apt-get install unattended-upgrades update-notifier-common && \
sudo dpkg-reconfigure -plow unattended-upgrades
(then select "YES")
More graceful solutions would be ansible scripts opening the port automatically, possibly also modifying an AWS security group rule via the CLI in addition to iptables if you are at AWS. I prefer modifying my outbound rules temporarily via AWS CLI initiated by a stealth box. This forces logging the update up in my AWS S3 log buckets but never shows up in the logs on the server itself. Further the server that initiates the update doesn't even have to be in the private subnet ACL.
Maybe do both? You have to figure at times an attack is going to relay off an internal IP in your subnet so there is merit to doubling down while preserving the ability to automate backups and security updates.
I hope this helps. If not reply and provide more code examples to be more specific and exact. #staysafe !
If the machine is compromised and outbound traffic on port 80 is allowed, it would make it easier for intruders to send back harvested data to themselves. Allowing outbound traffic means you can initiate a connection from your machine to the outside world. A better approach would be allowing outbound traffic only to certain web sites/addresses that you trust (i.e. Microsoft Windows Update, Google reCAPTCHA) rather than any destination in the world.
what do you mean with blocking outbound traffic over port 80.
You have two possibilities. Gernerate Dynamic Rules which allow communication from client to your webserver for this session. Search for Stateful firewall rules.
Or you generally allow established Connections to communicate in and outgoing with each other.
If you generally block all outbound traffic over Port 80 your Webserver could not reply to any client.
The other way around, if your Webserver needs to get some API, e.g. a jquery library he wont use port 80 as his Port to communicate with the Webserver who holds the API.
Your Webserver would normally choose a port > 1024 and use it for his request to get the API from the remote Server.
So blocking all traffic over port 80 (as your port you connecting from) would not prevent your Server from sending any requests for apis and such things. because he doesnt use port 80 when he acts as a client.