We have 2 Windows 2016 Servers and on both we found that our apps, that run on IIS, started to have suspicious behavior e.g. started to timeout, slowing down etc.
After doing more research we found that some processes have requests to same domain, that requests spend most bandwidth and they never go off (as you would expect).
Here are screenshots:
Server 1: https://gyazo.com/8bd433b2a2e6e3091885f9bf095ce0be
Server 2: https://gyazo.com/6a18d04d836adafc6952be212413eb16
Any idea how to resolve that?
since looking up for the domain didnt show any intresting result ,
in order to investigate the activity you need to download and install wireshark .
use it to sniff your hosts (install it on them ofcorse) and then submit the traffic samples of the HTTP activity of yhfund.dadcaptive.co.uk to the answer
Related
I'm using servlets 3 with jetty 8.1.1 and the SslContextFactory on an amazon ec2 machine (m1-small).
The first HTTPS request from localhost (of the amazone machine) is about 150ms and further
requests seem to get faster (down to ~40ms) but not as close as to
the HTTP response time of only 20ms - why? Is encryption really that
slow?
Also when comparing HTTPS and HTTP from outside of the amazon cloud
the difference gets even worse: HTTPS requests are at least 400ms
slower!? How can that be? Is the encrypted content also bigger? And
how can I debug it or make all faster?
Some more informations: all 'measurements' are unscientificly done via time curl http://mydomain.com/ping but are reproducable. Also there is an ec2 load balancer in between. I'm sure I've configured something wrong or there is a big misunderstanding from me. Let me know!
update to 8.1.7
check the time from localhost on the aws machine for reference
check using the IP vs DNS, quite often those sorts of long pauses involve dns issues
set your /etc/hosts to bypass a DNS look for host as a test as well
-Dorg.eclipse.jetty.LEVEL=DEBUG on the server side to enable debug, should help your correlate the roundtrip inside of jetty and compare to actual network results
ssl decryption does incur some performance hit, hard to say that that would be all of your differences here though
odds are this is not specific to jetty but something in the environment, which hopefully some bullet above will help steer you in the right direction
I need to find out how to enable SSL sessions. For this I've created a new question as it is unclear how to turn on in jetty and how to handle on the client side
While profiling my application I've noticed that in the Firebug Net panel, the "Connecting" time—that is the time waiting for a TCP connection—is consistently around 70–100ms. See image below:
Of course in the grand scheme of things, 100ms is not long, but I have seen other services that respond with 0ms Connect time. So if other servers can, I should be able to as well.
Any thoughts on how I might even beging to troubleshoot this?
I would start with looking to see if iptables is doing anything that may get in the way. Also, if you were working with an ELB load balancer, (or any other load balancing), I'd remove it from the mix and see if you are still having the longer than expected connect time.
You could also separately install lighttpd or Apache and see what happens. If you get a lower connect time, than that would point to your Node.js build. Although not definitively.
I would suggest a simple test to check if this problem is related to your server:
Launch another instance in the same availability zone as your server.
Benchmark your server with Apache Benchmark from the second instance:
ab -c 1 -n 20000 http://<private_server_instance_ip>:<port>/<URL>
It is important to put private IP here, not private or public DNS to sweep aside domain name resolution effects.
Check the average time taken per request: If it will be around 1 ms - the problem described is not of your server.
Benchmarking with FireFox BTW may not be the best idea because the results might depend on a number of circumstances.
I'd like to see how many requests varnish is sending to each backend via round-robin director.
i haven't found out how so far. thanks for any idea
You can't get this numbers from varnish directly, but if youre using round-robin director I'd guess the traffic to each backend will be nearly the 1/n of total ammount where n is number of backends. This is true only if backends all healthy and not skipped from round.
You can still get theese stattistics from backend access log or server-status page or similar, depending on http daemon used.
I'm a bit rusty on the varnish log file, but you should be able to use varnishlog and sample it.
http://linux.die.net/man/1/varnishlog
The sample, should give you an idea of the distribution.
Another option, would be to do a netstat and see how many connections are open to the background server.
-daniel
I'm trying to write a small website monitoring program, that can check my web hosts to see whether they are down or not, and to calculate the uptime or warn me if it's down. It's going to be a standalone app.
I wanted to know whether pinging is a good way of finding out whether a site is down or not?
Thanks in advance.
That's one thing that you can do but it's by no means a certainty either way.
Some sites will ignore ICMP packets so that no ping response is given. Some sites will respond to pings even when the web server (or whatever service you're after) is down.
The only way you can be certain that a given site will provide a service is to, well, use that service. Nothing else will be as accurate.
A better method would be to provide a series of steps which would detect where a fault lay, at least in the infrastructure that you can control. For example:
allow pings to be received and acted upon.
have a static web page in the web server.
have a dynamic page in the application server which delivers static content.
have a dynamic page in the application server which uses the database.
Then your tester client would simply attempt to "contact" those four points and report on the success. Since you would expect your site to be up most of the time, I'd just check the fourth option to see if everything was okay, and do the other checks only if a problem were found.
It depends on how you define ping. If you're talking about the "low level" ICMP echo, then no it isn't likely to be a good indicator of whether or not your site is down. You would be better off to actually have an application pull a page down from your site to ensure that the HTTP server is running. There are plenty of services for this and likely some code you could download from google as well. http://www.dailyblogtips.com/test-if-a-website-is-down-for-everyone-or-just-for-your/
ICMP can prove the server is alive.
TCP checking can show the web server is working, but not the site.
To perform site checking, you should do http GET request(even HEAD doesn't work sometimes) to make sure the page was fine.(return status 200)
You can write your own checking system or use some third party site like http://allping.net/
ping gives you insight in latency from a specific location and also points to possible network issues (packet loss). As said in a previous answer, some servers don't respond to ping requests in which case ping is useless.
To check a server with ping from over 50 locations worldwide have a look at this free tool: http://just-ping.com/
How can I make that a site automagically show a nice "Currently Offline" page when the server is down (I mean, the full server is down and the request can't reach IIS)
Changing the DNS manually is not an option.
Edit: I'm looking to some kind of DNS trick to redirect to other server in case the main server is down. I can make permanent changes to the DNS, but not manually as the server goes down.
I have used the uptime services at DNSMadeEasy to great success. In effect, they set the DNS TTL to a very low number (5 minutes). They take care of pinging your server.
In the event of outage, DNS queries get directed to the secondary IP. An excellent option for a "warm spare" in small shops with limited DNS requirements. I've used them for 3 years with not a single minute of downtime.
EDIT:
This allows for geographically redundant failover, which the NLB solution proposed does not address. If the network connection is down, both servers in a standard NLB configuration will be unreachable.
Some server needs to dish out the "currently offline page", so if your server is completely down, there will have to be some other server serving the file(s), so either you can set up a cluster of servers (even if just 2) and while the first one is down, the 2nd is configured only to return the "currently offline page". Once the 1st server is back up, you can take down the 2nd safetly (as server 1 will take all the load).
You probably need a second server with 100% uptime and then add some kind of failover load balancer. to it, and if the main server is online redirect to that and if it isn't redirect to itself showing a page saying server is down
I believe that if the server is down, there is nothing you can do.
The request will send up a 404 network error because when the web address is resolved to an IP, the IP that is being requested does not exist (because the server is down). If you can't change the DNS entry, then the client browser will continue to hit xxx.xxx.xxx.xxx and will never get a response.
If the server is up, but the website is down, you have options.
EDIT
Your edit mentions that you can make a permanent change the IP. But you would still need a two server setup in order to achieve what you are talking about. You can direct the DNS to a load balancer which would be able to direct the request to a server that is currently active. However, this still requires 100% uptime for the server that the DNS points to.
No matter what, if the server that the DNS is pointing to (which you must control, in order to redirect the traffic) is down, then all requests will receive a 404 network error.
EDIT Thanks to brian for pointing out my 404 error error.
Seriously, DNS is not the right answer to server load-balancing or fail-over. Too many systems (including stub clients and ISP recursive resolve) will cache records for much longer than the specified TTL.
If both servers are on the same network, use routing protocols to achieve fail-over by having both servers present the same IP address to the network, but where the fail-over server only takes over if it detects that the (supposedly) live server is offline.
If the servers are Unix, this is easily done by running Quagga on each server, and then using OSPF as the local routing protocol. I've personally used this for warm standby servers where the redundant system was actually in another data center, albeit one that was connected via a direct link to the main data center.
Certain DNS providers, such as AWS's Route 53, have a health-check option, which can be used to re-route to a static page. AWS has a how-to guide on setting this up.
I'm thinking if the site is load balanced the load balancer itself would detect that the web servers it's trying to redirect clients to are down, therefore it would send the user to a backup server with a message dictating technical problems.
Other than that.....
The only thing I can think is to control the calling page. Obviously that won't work in all circumstances... but if you know that most of your hits to this server will come from a particular source, then you could add a java script test to the source, and redirect to a "server down" page that is generated on a different server.
But if you are trying to handle all hits, from all sources (some of which you can't control), then I think you are out of luck. As other folks are saying - when a server is down, the browser gets a 404 error when it attempts a connection.
... perhaps there would be a way at a point in between to detect 404 errors being returned by servers and replacing them with a "server is down" web page. You'd need something like an HTML firewall or some other intermediate network gear between the server and the web client.