I am seeing frequent errors Varnish FetchError "Timed out reusing backend connection". Checked couple of blogs, but not find any resolution. Could you please help?
BereqHeader Accept-Encoding: gzip
BereqHeader X-Varnish: 37849780
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 36 NODEJS_2 xx.xx.xx.xx 9000 yy.yy.yy.yy 43309
Timestamp Bereq: 1605444526.456709 0.000102 0.000102
FetchError Timed out reusing backend connection
BackendClose 36 NODEJS_2
Timestamp Beresp: 1605444571.456893 45.000285 45.000183
Timestamp Error: 1605444571.456900 45.000292 0.000006
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Backend fetch failed
BerespHeader Date: Sun, 15 Nov 2020 12:49:31 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
Length 285
BereqAcct 2940 185 3125 0 0 0
End
The Timestamp Beresp: 1605444571.456893 45.000285 45.000183 tag in your VSL output indicates that your backend took 45.000183 seconds to generate its response, which triggered the first_byte_timeout.
In reality, your backend probably needed more than 45 seconds to generate the output, but Varnish just gave up after it hit the timeout.
Here are your options:
Increase the first_byte_timeout runtime parameter to a better number
Examine why your backend it taking so long
Although option 1 is theoretically viable, you really want to go for option 2, and figure out why it is taking so long for the backend to respond.
We have a vendor who sends us photos that are hosted on Azure Edge. These photos are available and I can download them, but if we do a CURL request we get a 404 roughly 4 out of 5 times. If we do a HEAD request to get the filesize, we get a 404 about 7 out of 10 times. On our production server, we get a 404 100% of the time. Any idea how we might work around this or if there's another way to check these files without the vendor having to fix their issue?
Sample file:
curl -I http://tdrvehicles2.azureedge.net/photos/202008/1419/1850/f253435f-86b1-4cc4-b95c-7756addddad4.jpg
HTTP/1.1 404 Not Found
Pragma: no-cache
Content-Length: 0
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Cache-Control: max-age=31536000
Expires: Thu, 19 Aug 2021 14:12:54 GMT
Date: Wed, 19 Aug 2020 14:12:54 GMT
Connection: keep-alive```
I have a strange problem with static file cache and TYPO3. It works, but not all the time. During tests of the delivered pages I recognized, that not every time the cached html is delivered.
I've then done several tests in differnt browsers which were never logged in to the TYPO3 backend. Everywhere the same. Some pages are cached, some not.
Even when I do multiple requests to the same page, e.g. via curl, sometimes I get the static variant, sometimes TYPO3 delivers the page. But it's not reproduceable.
I've created a small shell script to automate this testing for me:
#!/bin/sh
url="https://domain.tld/foo/bar/"
for i in `seq 10`;
do
echo $i
curl -s $url | grep statically
done
Expected behaviour would be to get the static page on each request, but reality looks like this:
➜ ~ ./test.sh
1
<!-- cached statically on: 14-12-16 08:46 -->
2
3
4
5
<!-- cached statically on: 14-12-16 08:46 -->
6
<!-- cached statically on: 14-12-16 08:46 -->
7
8
9
<!-- cached statically on: 14-12-16 08:46 -->
10
<!-- cached statically on: 14-12-16 08:46 -->
This is what the access log looks like:
xx.xxx.xx.xx - 14/Dec/2016:11:52:44 +0100 GET /foo/bar/ HTTP/1.1 200 26892 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:44 +0100 GET /foo/bar/ HTTP/1.1 200 26892 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:43 +0100 GET /foo/bar/ HTTP/1.1 200 26892 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:43 +0100 GET /foo/bar/ HTTP/1.1 200 26661 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:43 +0100 GET /foo/bar/ HTTP/1.1 200 26661 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:42 +0100 GET /foo/bar/ HTTP/1.1 200 26661 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:42 +0100 GET /foo/bar/ HTTP/1.1 200 26661 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:42 +0100 GET /foo/bar/ HTTP/1.1 200 26661 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:42 +0100 GET /foo/bar/ HTTP/1.1 200 26892 - curl/7.43.0
xx.xxx.xx.xx - 14/Dec/2016:11:52:41 +0100 GET /foo/bar/ HTTP/1.1 200 26892 - curl/7.43.0
The only difference is the size of the delivered content.
This is how the response header looks like. The first is broken, the second is correct:
➜ ~ curl -I https://domain.tld/foo/bar/
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 12:14:17 GMT
Server: Apache/2.4.20
X-Powered-By: PHP/7.0.6
Content-Language: de
Content-Length: 21850
Strict-Transport-Security: max-age=31536000
Connection: close
Content-Type: text/html; charset=utf-8
➜ Downloads curl -I https://domain.tld/foo/bar/
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 12:14:19 GMT
Server: Apache/2.4.20
Strict-Transport-Security: max-age=31536000
Vary: Host,Accept-Encoding
Last-Modified: Wed, 14 Dec 2016 07:46:38 GMT
Accept-Ranges: bytes
Content-Length: 21932
Cache-Control: max-age=38741
Expires: Wed, 14 Dec 2016 23:00:01 GMT
X-UA-Compatible: IE=edge
X-Content-Type-Options: nosniff
Connection: close
Content-Type: text/html; charset=utf-8
My .htaccess file with the mod_rewrite rules can be found here:
http://pastebin.com/5G6f3b4W
It's basically the default .htacess file shipped with TYPO3 7.6 and some custom additions.
Since the hosting is done on a managed server, there is no access to the vhost configuration to enable RewriteLog.
Tl;dr
creation of static file cache files works, the files are present in the file system
it's not reproduceable, why the static files aren't delivered at any request
I'm now searching for hints what I can do to further track down the problem.
I've found that some venues will only return photos if I use a signed in user instead of a client_id / client_secret. Is this intentional?
curl -i https://api.foursquare.com/v2/venues/4c36476d93db0f47f6cc1d92/photos?client_id=xxx\&client_secret=xxx\&group=venue\&v=20120304
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-cache, private, no-store
Content-Type: application/json; charset=utf-8
Date: Mon, 05 Mar 2012 00:28:34 GMT
Expires: Mon, 5 Mar 2012 00:28:34 GMT
Pragma: no-cache
Server: nginx/0.8.52
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4999
Content-Length: 66
Connection: keep-alive
{"meta":{"code":200},"response":{"photos":{"count":0,"items":[]}}}
curl -i https://api.foursquare.com/v2/venues/4c36476d93db0f47f6cc1d92/photos?group=venue\&v=20120304\&oauth_token=xxx\&v=20120304
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-cache, private, no-store
Content-Type: application/json; charset=utf-8
Date: Mon, 05 Mar 2012 00:29:19 GMT
Expires: Mon, 5 Mar 2012 00:29:19 GMT
Pragma: no-cache
Server: nginx/0.8.52
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 1000
Content-Length: 15311
Connection: keep-alive
{"meta":{"code":200},"notifications":[{"type":"notificationTray","item":{"unreadCount":0}}],"response":{"photos":{"count":14,"items":[lots of images here]}}}
I want to fetch a photo to associate with a given place as a background process, not tied to the specific user. Is it intended that this API only functions correctly for signed in users?
Looks like there's a bug in userless access to /venues/photos. The team is investigating. The intended behavior is that userless access of that endpoint returns all public photos attached to that venue.
I'm attempting to embed an HTML5 audio element pointing to MP3 or OGG data served by a PHP file . When I view the page in Safari, the controls appear, but the UI says "Live Broadcast." When I click play, the audio starts as expected. Once it ends, however, I can't start it playing again by clicking play. Even using the JS API on the audio element and setting currentTime to 0 fails with an index error exception.
I suspected the headers from the PHP script were the problem, particularly missing a content length. But that's not the case. The response headers include a proper Content- Length to indicate the audio has finite size. Furthermore, everything works as expected in Firefox 3.5+. I can click play on the audio element multiple times to hear the sound replay.
If I remove the PHP script from the equation and serve up a static copy of the MP3 file, everything works fine in Safari.
Does this mean Safari is treating audio src URLs with query parameters differently than URLs that don't have them? Anyone have any luck getting this to work?
My simple example page is:
<!DOCTYPE html>
<html>
<head></head>
<body>
<audio controls autobuffer>
<source src="say.php?text=this%20is%20a%20test&format=.ogg" />
<source src="say.php?text=this%20is%20a%20test&format=.mp3" />
</audio>
</body>
</html>
HTTP Headers from PHP script:
HTTP/1.x 200 OK
Date: Sun, 03 Jan 2010 15:39:34 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 8993
Keep-Alive: timeout=2, max=98
Connection: Keep-Alive
Content-Type: audio/mpeg
HTTP Headers from direct file access:
HTTP/1.x 200 OK
Date: Sun, 03 Jan 2010 20:06:59 GMT
Server: Apache
Last-Modified: Sun, 03 Jan 2010 03:20:02 GMT
Etag: "a404b-c3f-47c3a14937c80"
Accept-Ranges: bytes
Content-Length: 8993
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
I tried hard-coding the Accept-Ranges header into the script too, but no luck.
Can you post the headers sent by the server both with and without the PHP script? I'm wondering if the PHP script is sending a different Content-Type than serving the files normally.
It would also be a good idea to specify the type attribute on the source elements, so the browser does not have to download both clips to determine if it can play them.
I cannot reproduce your problem. I have tried to recreate the problem in Safari 4.0.4, and the current WebKit nightly, with the following test page. I am simply using mod_rewrite to dispatch to different formats based on a parameter instead of PHP, but I don't think that should make a difference, unless somehow PHP is modifying the file.
<!DOCTYPE html>
<title>Auido test</title>
<audio controls autobuffer>
<source src="gnossienne-no-1?foo=bar&format=.mp4">
<source src="gnossienne-no-1?foo=bar&format=.ogg">
</audio>
Can you try my example out and let me know if it works for you?
edit Ah. After poking around at it a bit more, it appears that the problem is due to an odd way that the <audio> element in Safari behaves in attempting to determine the size of the content.
Here's an excerpt from a packet capture of Safari upon encountering an <audio> element pointing to a file served directly from Apache. As you can see, it first tries to fetch the first two bytes of the media, presumably so it can get a Content-Length back, and possibly other headers. It then tries to fetch the whole thing. Then, inexplicably, it tries to fetch the first two bytes again, but passes appropriate caching headers to get a "304 Not Modified" response. And finally, still inexplicably, it fetches the last 3440 bytes of the file all over again. It does all of these in separate TCP connections, which adds considerable overhead, in addition to the overhead of fetching the data a couple of times.
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:48 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 2
Content-Range: bytes 0-1/4598
Connection: close
Content-Type: audio/mpeg
# 2 bytes of data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-4597
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:48 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 4598
Content-Range: bytes 0-4597/4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
If-None-Match: "b2a80ad-11f6-47c6139aaa800"
If-Modified-Since: Tue, 05 Jan 2010 02:02:08 GMT
HTTP/1.1 304 Not Modified
Date: Tue, 05 Jan 2010 02:12:49 GMT
Server: Apache
Connection: close
ETag: "b2a80ad-11f6-47c6139aaa800"
# no data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=1158-4597
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:49 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 3440
Content-Range: bytes 1158-4597/4598
Connection: close
Content-Type: audio/mpeg
# 3440 bytes of data
Anyhow, on to how it deals with the output of your PHP script. Here, Safari again tries to download the first two bytes, but your script ignores the Range request and returns the whole thing. Apparently, WebKit doesn't like that, and so it tries again, without the Range request. Again, your script sends the full contents. Safari now tries once more, adding an Icy-Metadata header, which indicates it thinks that it's downloading a stream and wants streaming metadata to be sent. It finally accepts the output of that, and the <audio> element can play.
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Accept: */*
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Icy-Metadata: 1
Connection: close
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
In summary, it appears that Safari (or more accurately, QuickTime, which Safari uses to handle all media and media downloading) has a completely braindamaged approach to downloading media. Something in the way you send your data back, probably the fact that you don't respond to Range requests, makes it think that you are sending streaming media, causing it to download the content repeatedly (though even when confronted with a server that does respond to a Range request, it still does several more requests than it really needs to).
My advice would be to try to respond appropriately to Range requests; when serving up media, browsers will likely use them to try to minimize bandwidth, by only buffering as much as they need to be able to play through (although have the autobuffer attribute which indicates that you would like them to buffer the whole thing, browsers may ignore that). I would recommend using X-Sendfile to let Apache deal with serving the file, caching, and range requests, but you appear to be on Dreamhost, which doesn't have mod_xsendfile installed, so you're going to have to roll your own Range handling.
Old topic but still valid in 2019. We finally found a solution... Below PHP script sample will consider Safari's "Range" header request.
$bytes_total = strlen($data);
if (isset($_SERVER['HTTP_RANGE'])) {
$byte_range = explode('-',trim(str_ireplace('bytes=','',$_SERVER['HTTP_RANGE'])));
$byte_from = $byte_range[0];
$byte_to = ($byte_range[1]+1);
$data = substr($data,$byte_from,$byte_to);
header('HTTP/1.1 206 Partial Content');
}
else {
$byte_from = 0;
$byte_to = $bytes_total;
}
$length = strlen($data);
header('Content-type: '.$content_type);
header('Accept-Ranges: bytes');
header('Content-length: ' . $length);
header('Content-Range: bytes '.$byte_from.'-'.$byte_to.'/'.$bytes_total);
header('Content-Transfer-Encoding: binary');
print($data);
For the record, while both Pochang and Chris are correct that you need the Content-Range header to fix this problem in Safari, Chrome requires one extra header that must be included for setting currentTime to work properly:
header( 'Accept-Ranges: bytes');
Note that you don't actually have to respond correctly to the request's Range header, you just have to include this in the response.
I got the same problem.
The key point is the Content-Range header.
Everything works fine after I add it to the mp3-output php.
Pochang is correct. Including a Content-Range header in the PHP response will cause Safari to behave properly. It also allows seeking (media.currentTime = 0;) without the dreaded INDEX_SIZE_ERR in Safari, though not in Chrome.
The PHP code for the header is:
$len = strlen( $data );
$shortlen = $len - 1;
header( 'Content-Range: bytes 0-'.$shortlen.'/'.$len);
And in 2020 it is still actual question.
Simply adding Content-Range header does not work.
Bellow is my implementation(based on some answers here).
$content_length = strlen($media_total);
$total_bytes = $content_length;
$content_length_1 = $content_length - 1;
if (isset($_SERVER['HTTP_RANGE'])) {
$byte_range = explode('-',trim(str_ireplace('bytes=','',$_SERVER['HTTP_RANGE'])));
$byte_from = $byte_range[0];
$byte_to = intval($byte_range[1]);
$byte_to = $byte_to == 0 ? $content_length_1 : $byte_to;
$media_total = substr($media_total,$byte_from,$byte_to);
$content_length = strlen($media_total);
header('HTTP/1.1 206 Partial Content');
}
else {
$byte_from = 0;
$byte_to = $content_length_1;
}
$content_range = 'bytes '.$byte_from.'-' . $byte_to . '/' . $total_bytes;
header('Accept-Ranges: bytes');
header("Content-Range: ".$content_range);
header("Content-type: ".$type);
header("Content-length: ".$content_length);
header('Content-Transfer-Encoding: binary');
echo $media_total;
exit;
Original question here: Timing problem for generated audio in some browsers