We have a vendor who sends us photos that are hosted on Azure Edge. These photos are available and I can download them, but if we do a CURL request we get a 404 roughly 4 out of 5 times. If we do a HEAD request to get the filesize, we get a 404 about 7 out of 10 times. On our production server, we get a 404 100% of the time. Any idea how we might work around this or if there's another way to check these files without the vendor having to fix their issue?
Sample file:
curl -I http://tdrvehicles2.azureedge.net/photos/202008/1419/1850/f253435f-86b1-4cc4-b95c-7756addddad4.jpg
HTTP/1.1 404 Not Found
Pragma: no-cache
Content-Length: 0
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Cache-Control: max-age=31536000
Expires: Thu, 19 Aug 2021 14:12:54 GMT
Date: Wed, 19 Aug 2020 14:12:54 GMT
Connection: keep-alive```
I have excluded the url like this from my varnish cache
if (req.url ~ "^/folder_name/") {
return (pass);
}
but still when i access curl -I http://ip/folder_name i see the below response
HTTP/1.1 301 Moved Permanently
Date: Mon, 04 Jul 2016 08:48:46 GMT
Server: Apache/2.2.15 (CentOS)
Location: http://ip/folder_name/
Content-Length: 319
Content-Type: text/html; charset=iso-8859-1
X-Varnish: 294958
Age: 0
Via: 1.1 varnish-v4
Connection: keep-alive
Can anyone please tell me what am i doing wrong, i need to exclude
"all urls that start with /folder_name" from being cached in varnisg
Your regex ends in a slash and "http://ip/folder_name does" not - could this be it?
I have website which is online. When I'am using it via browser everything is ok and this page is present in browser. When I'm using it as googlebot ( via webmastertools ) i'm getting error
HTTP/1.1 404 Not Found
Date: Mon, 19 Nov 2012 09:57:37 GMT
Server: Apache
X-Powered-By: PHP/5.2.17
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: symfony=55240a0a341202d07fc96cbc1c1bcca5; path=/
Keep-Alive: timeout=2, max=200
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
and rest of the html code...
same thing when I'm trying to validate it via wc3 validator.
Please help :( I tryied everything :(
website address is mojaczestochowa.pl
If more info is needed please let me know.
Try to check the pae with web-sniffer and set user agent to google.bot
Here is the exact query, which will simulate server's response to the GoogleBot crawler:
https://websniffer.cc/?url=http://mojaczestochowa.pl/&uak=9
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 11 years ago.
The community reviewed whether to reopen this question 8 months ago and left it closed:
Original close reason(s) were not resolved
I have a problem with my server redirecting http://www.mylesgray.com:8080/ -> http://www.mylesgray.com/.
Here are my Nginx default and fastcgi_params config files:
https://gist.github.com/1745271
https://gist.github.com/1745313
This is rather a nusance as I am trying to run a benchmark of Nginx w/ caching vs Varnish w/ caching on top of Nginx to see if there is any performance benefit of one over the other.
As such I have straight Nginx w/ caching listening on port 8080 and varnish on port 80 which forwards any non-cached requests to Nginx on localhost:8080, so obviously what I want to do is run an ab benchmark on http://www.mylesgray.com:8080/ and on http://www.mylesgray.com/ to see the difference.
Here are the results of curl -I on various addresses.
# curl -I http://www.mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Date: Sun, 05 Feb 2012 12:07:34 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
# curl -I http://mylesgray.com
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
Content-Length: 0
Date: Sun, 05 Feb 2012 12:15:51 GMT
X-Varnish: 1419774165 1419774163
Age: 15
Via: 1.1 varnish
Connection: keep-alive
# curl -I http://mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Date: Sun, 05 Feb 2012 12:16:08 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
Then running curl -I http://www.mylesgray.com gives:
# curl -I http://www.mylesgray.com
HTTP/1.1 200 OK
Server: nginx/0.7.65
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Content-Length: 5132
Date: Sun, 05 Feb 2012 12:07:29 GMT
X-Varnish: 1419774133 1419774124
Age: 30
Via: 1.1 varnish
Connection: keep-alive
So as you can see 80 is served by Varnish and 8080 by Nginx but I cannot find anywhere anything that does a 301 redirect, not in nginx.conf or in the sites-enabled/default file and I don't believe it is caused by Wordpress itself but an very much open to correction.
Please help, this is driving me nuts!
Myles
You should add a '/' at then end of your URLs. Furthermore if you run ab http://foo.com it will return you a "ab: invalid URL" error. If you do "ab -t 10 http://example.com/" everything will work fine. You should always use '/' in your URLs otherwize your webserver will try to redirect the page to the home page automatically for you which generates an undesirable extra load on the server and some extra bytes on the wire.
You web server told you what it did:
'/' is missing and something is incorrect with the port numer:
# curl -I http://www.mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
[...]
======> Location: http://www.mylesgray.com/
'www' and '/' are missing:
# curl -I http://mylesgray.com
HTTP/1.1 301 Moved Permanently
[...]
=======> Location: http://www.mylesgray.com/
[...]
'/' and 'www' are missing:
# curl -I http://mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
[...]
========> Location: http://www.mylesgray.com/
'hope that helps :)
The presence of an X-Powered-By: PHP header means that wordpress is issuing the 301. It's due to wordpress forcing www.mylesgray.com. When you use a nonstandard port, user agents will generally include the port in the Host: header. Try adding
fastcgi_param HTTP_HOST $host;
with the rest of your fastcgi_param directives (or alog with your "include fastcgi_params;") and it should fix this.
I'm attempting to embed an HTML5 audio element pointing to MP3 or OGG data served by a PHP file . When I view the page in Safari, the controls appear, but the UI says "Live Broadcast." When I click play, the audio starts as expected. Once it ends, however, I can't start it playing again by clicking play. Even using the JS API on the audio element and setting currentTime to 0 fails with an index error exception.
I suspected the headers from the PHP script were the problem, particularly missing a content length. But that's not the case. The response headers include a proper Content- Length to indicate the audio has finite size. Furthermore, everything works as expected in Firefox 3.5+. I can click play on the audio element multiple times to hear the sound replay.
If I remove the PHP script from the equation and serve up a static copy of the MP3 file, everything works fine in Safari.
Does this mean Safari is treating audio src URLs with query parameters differently than URLs that don't have them? Anyone have any luck getting this to work?
My simple example page is:
<!DOCTYPE html>
<html>
<head></head>
<body>
<audio controls autobuffer>
<source src="say.php?text=this%20is%20a%20test&format=.ogg" />
<source src="say.php?text=this%20is%20a%20test&format=.mp3" />
</audio>
</body>
</html>
HTTP Headers from PHP script:
HTTP/1.x 200 OK
Date: Sun, 03 Jan 2010 15:39:34 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 8993
Keep-Alive: timeout=2, max=98
Connection: Keep-Alive
Content-Type: audio/mpeg
HTTP Headers from direct file access:
HTTP/1.x 200 OK
Date: Sun, 03 Jan 2010 20:06:59 GMT
Server: Apache
Last-Modified: Sun, 03 Jan 2010 03:20:02 GMT
Etag: "a404b-c3f-47c3a14937c80"
Accept-Ranges: bytes
Content-Length: 8993
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
I tried hard-coding the Accept-Ranges header into the script too, but no luck.
Can you post the headers sent by the server both with and without the PHP script? I'm wondering if the PHP script is sending a different Content-Type than serving the files normally.
It would also be a good idea to specify the type attribute on the source elements, so the browser does not have to download both clips to determine if it can play them.
I cannot reproduce your problem. I have tried to recreate the problem in Safari 4.0.4, and the current WebKit nightly, with the following test page. I am simply using mod_rewrite to dispatch to different formats based on a parameter instead of PHP, but I don't think that should make a difference, unless somehow PHP is modifying the file.
<!DOCTYPE html>
<title>Auido test</title>
<audio controls autobuffer>
<source src="gnossienne-no-1?foo=bar&format=.mp4">
<source src="gnossienne-no-1?foo=bar&format=.ogg">
</audio>
Can you try my example out and let me know if it works for you?
edit Ah. After poking around at it a bit more, it appears that the problem is due to an odd way that the <audio> element in Safari behaves in attempting to determine the size of the content.
Here's an excerpt from a packet capture of Safari upon encountering an <audio> element pointing to a file served directly from Apache. As you can see, it first tries to fetch the first two bytes of the media, presumably so it can get a Content-Length back, and possibly other headers. It then tries to fetch the whole thing. Then, inexplicably, it tries to fetch the first two bytes again, but passes appropriate caching headers to get a "304 Not Modified" response. And finally, still inexplicably, it fetches the last 3440 bytes of the file all over again. It does all of these in separate TCP connections, which adds considerable overhead, in addition to the overhead of fetching the data a couple of times.
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:48 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 2
Content-Range: bytes 0-1/4598
Connection: close
Content-Type: audio/mpeg
# 2 bytes of data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-4597
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:48 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 4598
Content-Range: bytes 0-4597/4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
If-None-Match: "b2a80ad-11f6-47c6139aaa800"
If-Modified-Since: Tue, 05 Jan 2010 02:02:08 GMT
HTTP/1.1 304 Not Modified
Date: Tue, 05 Jan 2010 02:12:49 GMT
Server: Apache
Connection: close
ETag: "b2a80ad-11f6-47c6139aaa800"
# no data
GET /stackoverflow/audio-test/say-noid3?foo=bar&format=.mp3 HTTP/1.1
Host: ephemera.continuation.org
Range: bytes=1158-4597
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
Cookie: [redacted]
HTTP/1.1 206 Partial Content
Date: Tue, 05 Jan 2010 02:12:49 GMT
Server: Apache
Last-Modified: Tue, 05 Jan 2010 02:02:08 GMT
ETag: "b2a80ad-11f6-47c6139aaa800"
Accept-Ranges: bytes
Content-Length: 3440
Content-Range: bytes 1158-4597/4598
Connection: close
Content-Type: audio/mpeg
# 3440 bytes of data
Anyhow, on to how it deals with the output of your PHP script. Here, Safari again tries to download the first two bytes, but your script ignores the Range request and returns the whole thing. Apparently, WebKit doesn't like that, and so it tries again, without the Range request. Again, your script sends the full contents. Safari now tries once more, adding an Icy-Metadata header, which indicates it thinks that it's downloading a stream and wants streaming metadata to be sent. It finally accepts the output of that, and the <audio> element can play.
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Range: bytes=0-1
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
Accept-Encoding: identity
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Connection: close
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Accept: */*
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
GET /say.php?text=this%20is%20a%20test&format=.mp3 HTTP/1.1
Host: tts.mindtrove.info
Accept: */*
User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
Icy-Metadata: 1
Connection: close
HTTP/1.1 200 OK
Date: Tue, 05 Jan 2010 02:14:28 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Content-Length: 4598
Connection: close
Content-Type: audio/mpeg
# 4598 bytes of data
In summary, it appears that Safari (or more accurately, QuickTime, which Safari uses to handle all media and media downloading) has a completely braindamaged approach to downloading media. Something in the way you send your data back, probably the fact that you don't respond to Range requests, makes it think that you are sending streaming media, causing it to download the content repeatedly (though even when confronted with a server that does respond to a Range request, it still does several more requests than it really needs to).
My advice would be to try to respond appropriately to Range requests; when serving up media, browsers will likely use them to try to minimize bandwidth, by only buffering as much as they need to be able to play through (although have the autobuffer attribute which indicates that you would like them to buffer the whole thing, browsers may ignore that). I would recommend using X-Sendfile to let Apache deal with serving the file, caching, and range requests, but you appear to be on Dreamhost, which doesn't have mod_xsendfile installed, so you're going to have to roll your own Range handling.
Old topic but still valid in 2019. We finally found a solution... Below PHP script sample will consider Safari's "Range" header request.
$bytes_total = strlen($data);
if (isset($_SERVER['HTTP_RANGE'])) {
$byte_range = explode('-',trim(str_ireplace('bytes=','',$_SERVER['HTTP_RANGE'])));
$byte_from = $byte_range[0];
$byte_to = ($byte_range[1]+1);
$data = substr($data,$byte_from,$byte_to);
header('HTTP/1.1 206 Partial Content');
}
else {
$byte_from = 0;
$byte_to = $bytes_total;
}
$length = strlen($data);
header('Content-type: '.$content_type);
header('Accept-Ranges: bytes');
header('Content-length: ' . $length);
header('Content-Range: bytes '.$byte_from.'-'.$byte_to.'/'.$bytes_total);
header('Content-Transfer-Encoding: binary');
print($data);
For the record, while both Pochang and Chris are correct that you need the Content-Range header to fix this problem in Safari, Chrome requires one extra header that must be included for setting currentTime to work properly:
header( 'Accept-Ranges: bytes');
Note that you don't actually have to respond correctly to the request's Range header, you just have to include this in the response.
I got the same problem.
The key point is the Content-Range header.
Everything works fine after I add it to the mp3-output php.
Pochang is correct. Including a Content-Range header in the PHP response will cause Safari to behave properly. It also allows seeking (media.currentTime = 0;) without the dreaded INDEX_SIZE_ERR in Safari, though not in Chrome.
The PHP code for the header is:
$len = strlen( $data );
$shortlen = $len - 1;
header( 'Content-Range: bytes 0-'.$shortlen.'/'.$len);
And in 2020 it is still actual question.
Simply adding Content-Range header does not work.
Bellow is my implementation(based on some answers here).
$content_length = strlen($media_total);
$total_bytes = $content_length;
$content_length_1 = $content_length - 1;
if (isset($_SERVER['HTTP_RANGE'])) {
$byte_range = explode('-',trim(str_ireplace('bytes=','',$_SERVER['HTTP_RANGE'])));
$byte_from = $byte_range[0];
$byte_to = intval($byte_range[1]);
$byte_to = $byte_to == 0 ? $content_length_1 : $byte_to;
$media_total = substr($media_total,$byte_from,$byte_to);
$content_length = strlen($media_total);
header('HTTP/1.1 206 Partial Content');
}
else {
$byte_from = 0;
$byte_to = $content_length_1;
}
$content_range = 'bytes '.$byte_from.'-' . $byte_to . '/' . $total_bytes;
header('Accept-Ranges: bytes');
header("Content-Range: ".$content_range);
header("Content-type: ".$type);
header("Content-length: ".$content_length);
header('Content-Transfer-Encoding: binary');
echo $media_total;
exit;
Original question here: Timing problem for generated audio in some browsers