I am having trouble using varnish as backend to another varnish server. The front-end varnish uses Disk based caching and backend uses malloc which load-balances multiple backend servers. When I hit the backend server with www.example.com it retrieves pages perfectly, when I hit the front-end server I am receiving error: Error 200 OK OK Guru Meditation: XID: 1692612819 Varnish cache server. Is there any configuration changes to be made to use varnish caching server behind another one?
It is probably due to the inconsistency between cache copies. Try applying HTTP/1.1 to access, and set the max-age for better synchronization.
Cache-control: max-age: The HTTP/1.1 replacement for the Expires header. While Expires >specifies the validity period using a full time stamp (valid until X day at Y time), max-age >uses a relative system (valid for Z many seconds) to avoid potential problems with time zones >and unsynchronised clocks.
Reference: http://www.anchor.com.au/hosting/dedicated/CreatingHighPerformanceDynamicWebSitesWithTheVarnishHttpAccelerator
Related
Need some help to dig deeper into why IIS is behaving in a certain way. Edge/Chrome makes an HTTP2.0 request to IIS, using the IPv6 address in the header (https://[ipv6]/) which results in the server generating a 302 response. The ISAPI filter makes some mods to the 302 response and replaces the response buffer. IIS drops the request/response and logs in HTTPERR log:
<date> <time> fe80::a993:90bf:89ff:1a54%2 1106 fe80::bdbf:5254:27d2:33d8%2 443 HTTP/2.0 GET <url> 1 - 1 Connection_Dropped_List_Full <pool>
Suspect related to HTTP2.0, when putting Fiddler in the middle, it isn't HTTP/2.0 anymore, it downgrades to HTTP/1.1 and it works.
When using an IPv4 address, it works. In either case the filter goes through the identical steps. There is no indication in the filter that anything went wrong.
Failed Request Tracing will not write buffers for incomplete/dropped requests that appear in HTTPERR log.
Is there a place where I can find out more detailed information about why IIS is dropping the request?
I did the network capture, and looks like browser is initiating the FIN tear down of session.
Do you use any load balance or reverse proxy before request get in IIS? This error indicates that the log cannot store more lost connections, so the problem is that your connection is lost.
If you use load balance, web application is under heavy load and
because of this no threads are available to currently provide
logging data to HTTP.sys. Check this.
Or before IIS response to client, client has closed the request but
IIS still send response. This is more likely to be a problem with the
application itself not IIS and http.sys. Check this.
One thing I noticed is if you change http2 to 1.1, it can work well. The difference between http1.1 and 2 is performance.
HTTP/1.1 practically allows only one outstanding request per TCP connection (though HTTP pipelining allows more than one outstanding request, it still doesn’t solve the problem completely).
HTTP/2.0 allows using same TCP connection for multiple parallel requests
So it looks like that when you use http2, one connection includes multiple requests and application cannot handle these requests well, especially the request of image.
Aother thing is failed request tracing can catch all request and response, including status code is 200 and 302.
I'm trying to understand what is the difference between HTTPS and http/2?
If I'm going to build a Node.js/express app, what should I use?
Can I use HTTPS with http/2?
Maybe if I use HTTPS, I don't need http/2 because it's the same, or HTTPS use http/2 under the hood?
I'm confused.
Someone is linked to me "difference between HTTP 1.1 and HTTP 2.0 [closed]", but I understand the difference between HTTP and HTTP2. I'm asking about HTTPS and HTTP/2
HTTP - A protocol used by clients (e.g. web browsers) to request resources from servers (e.g. web servers).
HTTPS - A way of encrypting HTTP. It basically wraps HTTP messages up in an encrypted format using SSL/TLS. The web is moving towards HTTPS more and more and web browsers are starting to put more and more warnings when a website is served over unencrypted HTTP. Unless you have a very good reason not to, use HTTPS on any websites you create now.
Digging into HTTP more we have:
HTTP/1.1 - this was the prevalent format of HTTP until recently. It is a text-based protocol and has some inefficiencies in it - especially when requesting lots of resources like a typical web page. HTTP/1.1 messages can be unencrypted (where web site addresses start http://) or encrypted with HTTPS (where web site address start with https://). The client uses the start of the URL to decide which protocol to use, usually defaulting to http:// if not provided.
HTTP/2 - a new version of HTTP released in 2015 which addresses some of the performance issues by moving away from a text based protocol to a binary protocol where each byte is clearly defined. This is easier to parse for clients and servers, leaves less room for errors and also allows multiplexing. HTTP/2, like HTTP/1.1, is available over unencrypted (http://) and encrypted (https://) channels but web browsers only support it over HTTPS, where it is decided whether to use HTTP/1.1 or HTTP/2 as part of the HTTPS negotiation at the start of the connection.
HTTP/2 is used by about a third of all websites at the time of writing (up to 50% of websites as of Jan 2020, and 67% of website requests). However not all clients support HTTP/2 so you should support HTTP/1.1 over HTTPS and HTTP/2 over HTTPS where possible (I believe node automatically does this for you when using the http module). I do not believe HTTP/1.1 will be retired any time soon. You should also consider supporting HTTP/1.1 over unencrypted HTTP and then redirect to HTTPS version (which will then use HTTP/1.1 or HTTP/2 as appropriate). A web server like Apache or Nginx in front of Node makes this easy.
HTTP/3 - the next version of HTTP, currently under development. It is expected to be finalised in 2020 though it will likely be late 2020 or even 2021 before you see this widely available in web servers and languages like node. It will be built on top of a UDP-based transport called QUIC (rather than the TCP-based protocol that HTTP/1.1 and HTTP/2 are based on top of). It will include part of HTTPS in the protocol so HTTP/3 will only be available over HTTPS.
In short you should use HTTP/1.1 over HTTPS, should consider HTTP/2 as well if easy to implement (not always possible as not quite ubiquitous yet - but getting there) and in future you might be using HTTP/3.
I suggest you get a firm understanding of all of these technologies (except maybe HTTP/3 just yet) if you want to do web development. It will stand you in good stead.
In the Server response header returned from my site hosted on Heroku and served by NodeJS/express, I see the value Cowboy.
Who's outputting that value for Server?
Bonus, what's the "etymology"/reason for using Cowboy as the value for Server?
That would probably be heroku/cowboy. "Cowboy" is just an arbitrary project name that ninenines/cowboy chose. It's just as random as "Apache" (a patchy version of its predecessor), "nginx", "unicorn", etc.
Bonus answer "etymology": It doesn't say on it's GitHub page (https://github.com/heroku/cowboy), but here are some guesses:
Because it runs on Ranch (https://github.com/heroku/ranch): Ranch is a socket acceptor pool for TCP protocols?
Possibly also because it sees Apache as a competitor?
I will answer your questions:
Who's outputting that value for Server? It's a HTTP server called Cowboy. Cowboy is a small, fast and modular HTTP server written in Erlang.
Bonus, what's the "etymology"/reason for using Cowboy as the value for Server?
It's a way to identify that your requests are being processed by Cowboy. You could check the following links, with the Cowboy docs.
Cowboy
Cowboy handler docs
I'm developing a tool for checking the integrity of links in a web page.
I've noticed that various Instagram URLs will return a HTTP 500 response in cases where if one were to visit the given URL in a browser one would get a HTTP 200 response accompanied by the expected resource.
This is when requesting regular Instagram URLs as one would do as a browser user and not when using the REST API.
A typical request/response using cURL:
curl -v http://instagram.com/p/YKTkxHBA-P/
* About to connect() to instagram.com port 80 (#0)
* Trying 54.225.159.246... connected
> GET /p/YKTkxHBA-P/ HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: instagram.com
> Accept: */*
>
< HTTP/1.1 500 Server Error
< Cache-Control: no-cache
< Content-Type: text/html
< Date: Tue, 15 Oct 2013 08:31:09 GMT
< Server: nginx
< Content-Length: 87
< Connection: keep-alive
<
<html><body><h1>500 Server Error</h1>
An internal server error occured.
</body></html>
* Connection #0 to host instagram.com left intact
* Closing connection #0
I did for some time get HTTP 200 responses in such cases but am now consistently getting HTTP 500 responses.
This is all specific to a given host; such URLs, even when sending requests with cURL, will return HTTP 200 responses from other machines.
Due to this being specific to the host sending the requests I suspect a form of rate limiting or request filtering by IP is going on, however I can find no documentation to the effect.
Will Instagram officially return a HTTP 500 response as above due to a given IP being denied access?
This is an IP rate limit. If you want to skip the part where you contact Instagram and wait for the monkey crew they have working over there to fix the problem, simply assign 1000 IPs to your server and loop through them randomly for your requests. You won't see anymore 500's.
Cheers.
I've received a mail from Instagram support just yesterday.
...
Hi,
We made some changes to our server configurations, can you please check if you are still seeing the 500 errors?
Thanks,
...
..well I was 100% certain that those 500ers didn't come from IG's IP rate limit because they weren't returned beforehand either.
I've been checking the logfiles and found a couple 502 (Bad Gateway) and "host unreachables", though no more 500 per se, after 2014-04-14 18:28:56 (PST).
Looks like they've finally sorted it out after nearly a month... ^_^
I had the exact same problem - unfortunately there is no way around it. This happens because of two many requests. At first I thought it was my IP or possibly my UDID until I signed into my app from my own phone and from my home IP but using a different Instagram id and it finally worked as expected again. Eventually, I could use my own ID again as time went by but with limited requests. I am not exactly sure how the algorithm works but the more time that went on the more requests I could use again.
Also, this is in real-time on an actual iPhone in an actual app - not on the iOS sim or Instagram API console, FYI.
Main Point: The request limit is based off the user (5000 requests per hour per user)...there is no IP rate limit.
Hope this helps :)
Clayton
I have the same problem. As i've rad, you have to get an extra access to API, I mean the Sandbox mode in your application does not allow you to use all the API. To get extra premissions go to client preferences, tab "premissions".
It seems to be related to curl version, I also experience the same problem with v 7.22.0 on 4 different machines 10 different ip's, while v7.30.0 and v7.19.7 are working like a charm.
Currently investigating further more.
I'm almost 100% sure this happens because the domain/IP address is blocked from the Instagram API.
Reason:
Works to get the JSON in the browser. Doesn't work to get with cURL from webservice.
I copied my exact application from my primary domain (were the application doesn't work) to another domain. Same application worked.
The strange thing is that you get a "500 Internal Error" back and not a message "Your IP is blocked".
My online whiteboard application has been working previously, but for whatever reason, it no longer works. Now it shows a 503 Varnish cache error as seen here: http://grab.by/eFHG
Do you happen to know where I should start to look to try to resolve this issue?
Thanks!
Donny
This error means that Varnish had no response (even not an HTTP error) from the backend within the timeout limit.
You can troubleshoot that in many ways :
On the backend : do you see requests from Varnish in your webserver
log ?
On Varnish server : Run varnishlog and check the request process. You should have events in this order : RxRequest > TxRequest > RxResponse > TxResponse. Your problem is between TxRequest (request sent to backend) and RxResponse (response received from backend).
On your Varnish server try connecting on the backend using telnet (telnet ). Does it connect ? If it does, try sending a request (e.g. "GET / "). Do you receive a response ?
Probable causes could be : firewall/selinux blocking between varnish & backend, bad varnish or backend web server config (are backend address & port sync ?), webserver stopped, ...
You could always check your /etc/varnish/default.vcl (CentOS).
backend default {
.host = "127.0.0.1";
.port = "80";
}
Make sure the .host value is your server IP Address, and change the port to 8080, and adjust your port setting in /etc/httpd/conf/httpd.conf and make sure Apache listen to 8080.
EDIT
It could also mean web server (Apache) have wrong/default settings.
Check the virtual host ports, the values might be 80, they should be
8080.
Check Listen directive and ServerName, they must have 8080
port.
Make sure you do sudo systemctl enable varnish.
And for the final touch, do sudo reboot (cache programs are bugging me in the development state).
When I tried to reach this page today, I got shown the error page instead with following contents:
Error 503 Service Unavailable
Service Unavailable
Guru Mediation:
Details: cache-hel6832-HEL 1623148717 170212118
So it seems StackOverflow is also using a CDN that was failing today, and the failure was impacting huge number of services at the same time. In these kind of situations, we might be at the mercy of external CDN provider. Today showed us that sometimes the only thing that you can do, is to wait for the others to solve it for you.
Luckily these kind of outages are shortlived.
Suitable long term strategy is to use several CDN services instead of single.