How do I make Varnish purge all variants of a URL? - varnish

Varnish 2.1: I'm trying to use http PURGE to get Varnish to renew one cached URL.
While testing, I would get results that seemed strange until I read this bit in the documentation: "If there are several variants of the same URL in the cache however, only the matching variant will be purged. To purge a gzip variant of the same page the request would have to look like this:"
So now if I do these two commands:
curl -I http://example.com/my-url
curl -X PURGE http://example.com/my-url
I get back a 200 Purged response.
And if I do these two:
curl -I http://example.com/my-url -H "Accept-Encoding: gzip"
curl -X PURGE http://example.com/my-url -H "Accept-Encoding: gzip"
I also get a 200 Purged.
Now if I try:
curl -I http://example.com/my-url -H "Accept-Encoding: deflate"
I get back headers that indicate that the Content-Encoding: is "text/html;charset=utf-8", and it has age in the cache. That seems to indicate that "deflate" isn't being supported by my app, which is probably correct.
The page also has an Age: header indicating that it's being cached.
Now how do I purge that? (Am I asking a moot question?)
All of these attempts return "404 Not in cache":
curl -X PURGE http://example.com/my-url -H "Accept-Encoding: deflate"
curl -X PURGE http://example.com/my-url -H "Accept-Encoding: text/html;charset=utf-8"
curl -X PURGE http://example.com/my-url -H "Accept-Encoding: text/html"
Is there a magic way to purge all of the variants of a URL?

Aha, by applying some google-fu, I stumbled on a forum post, that suggests the following:
purge("req.url ~ ^" req.url "$");
i.e. using a regexp behind purge, to handle all Accept-Encoding headers.
More info here: http://www.gossamer-threads.com/lists/varnish/misc/15124
.. which means you need to modify your VCL though.

I think this is what u need:
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
# allow PURGE from localhost and 192.168.55...
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}

Related

How to send POST request from commandline?

I'm trying to send an "on" signal to a wemo smart plug. I want to send the signal from the linux command line. I know that the request is supposed to look like what I've included below, but I'm not sure what syntax to use. I tried using cURL, but couldn't quite seem to figure it out. Any help would be super appreciated!
POST /upnp/control/basicevent1
SOAPACTION: "urn:Belkin:service:basicevent:1#SetBinaryState"
Content-Type: text/xml; charset="utf-8"
Accept: */*
User-Agent: PostmanRuntime/7.15.2
Cache-Control: no-cache
Host: 192.168.1.116:49153
Accept-Encoding: gzip, deflate
Content-Length: 306
Connection: keep-alive
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:SetBinaryState xmlns:u="urn:Belkin:service:basicevent:1">
<BinaryState>0</BinaryState>
</u:SetBinaryState>
</s:Body>
</s:Envelope>
I tried putting each of the headers in quotes after a "-H" but then I wasn't sure what to do with the headers that are already quoted. Also, it appeared to be still sending to port 80 even though I included the host header with a different port?
As far as I know, the plug has an IP address but no web address.
Post method have two part (header & body).
You should run cURL command like that:
Send Header:
curl -X POST -H 'SOAPACTION: "urn:Belkin:service:basicevent:1#SetBinaryState"' -H 'Content-Type: text/xml; charset="utf-8"' http://192.168.1.116:49153/upnp/control/basicevent1
With this method, you can head multi header POST.
Send body POST:
And also with that command you can send POST body:
curl -X POST -F '<?xml version="1.0" encoding="utf-8"?>...' http://192.168.1.116:49153/upnp/control/basicevent1
Also, they're available at the following links:
How to send body POST ,
How to send header POST
Command you want at single line:
curl -X POST -H 'SOAPACTION: "urn:Belkin:service:basicevent:1#SetBinaryState"' -H 'Content-Type: text/xml; charset="utf-8"' -H 'Accept: */*' -H 'User-Agent: PostmanRuntime/7.15.2' -H 'Cache-Control: no-cache' -H 'Host: 192.168.1.116:49153' -H 'Accept-Encoding: gzip, deflate' -H 'Content-Length: 306' -H 'Connection: keep-alive' -F '<?xml version="1.0" encoding="utf-8"?>\n<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">\n <s:Body>\n <u:SetBinaryState xmlns:u="urn:Belkin:service:basicevent:1">\n <BinaryState>0</BinaryState>\n </u:SetBinaryState>\n </s:Body>\n</s:Envelope>' http://192.168.1.116:49153/upnp/control/basicevent1
alias
If you think this is a long command, you can set as alias.
For e.g:
alias myPersonalCommandLS='ls -lthra1d'

Eclipse Hono - 401 Unauthorized Error (Even when the credentials are correct)

I am using the following commands to create a tenant in Eclipse Hono
$ curl -X POST -i -H 'Content-Type: application/json' -d '{"tenant-id": "testenant1"}'
http://localhost:28080/tenant
HTTP/1.1 201 Created
location: /tenant/testenant1
content-length: 0
Registering a device in the tenant using the below command
curl -X POST -i -H 'Content-Type: application/json' -d '{"device-id": "1"}'
http://localhost:28080/registration/testenant1
HTTP/1.1 201 Created
location: /registration/testenant1/1
content-length: 0
Authenticating the registered device using the below command
$ curl -i -X POST -H 'Content-Type: application/json' --data-binary '{
"device-id": "1",
"type": "hashed-password",
"auth-id": "newAuth1",
"secrets": [{
"pwd-plain": "mylittle"
}]
}' http://localhost:28080/credentials/testenant1
HTTP/1.1 201 Created
location: /credentials/testenant1/newAuth1/hashed-password
content-length: 0
When I try to send data to this registered and Authenticated device using the below command.
curl -X POST -i -u newAuth1#testenant1:mylittle -H 'Content-Type: application/json' -d '{"temp": 23.07, "hum": 45.85}' http://localhost:8080/telemetry
HTTP/1.1 401 Unauthorized
content-length: 0
I will be getting 401 Unauthorized error (I am expecting 503 - Service Unavailable error).
Note: I was using the similar approach before and it was working perfectly fine. I am not sure if I am missing something.
You are using wrong credentials when POSTing the data. The username always consists of the auth-id and the tenant-id separated by #.
You need to use:
curl -X POST -i -u newAuth1#testenant1:mylittle -H 'Content-Type: application/json' -d '{"temp": 23.07, "hum": 45.85}' http://localhost:8080/telemetry
That said, based on the URIs you are using for registering the tenant and device, you seem to be using quite an old version of Hono. Please consider upgrading to the latest version (1.1.1) in order to take advantage of recent development/bug fixing ...

Linux curl post request and Error 417 but sudo success?

My service is deployed in docker,the exposed nginx port is 18082;service port is 38087,and I tried both.
while I use command
curl -i -X POST -H 'content-type: text/json' -d #post.json \
http://127.0.0.1:18082/youtu/openliveapi/livedetectfour
return 417
HTTP/1.1 417 Expectation failed
Server: squid/2.7.STABLE9
Date: Tue, 15 Aug 2017 11:57:04 GMT
Content-Type: text/html
Content-Length: 1436
X-Squid-Error: ERR_INVALID_REQ 0
X-Cache: MISS from SK-SQUIDDEV-118
X-Cache-Lookup: NONE from SK-SQUIDDEV-118:8080
Connection: close
But when I add sudo in the front,it return success.
HTTP/1.1 100 Continue
HTTP/1.1 200 OK Server: openresty/1.9.15.1
I really do search and knew while curl post over 1024 bytes,it will first send
expect 100-continue.if server don't support that request, it will return 417 error.
then how can sudo succsess,maybe It is related to nginx mechanism,I'm not familiar with that.thanks.
Since you are using proxy in your corporate setup and the proxy might be insert in your user's bash profile. You can do 3 things
Update ~/.bash_profile or ~/.bashrc
Remove the proxy from your profile and you won't need the proxy
Unset the variables
You can unset the variables before the call
unset http_proxy
unset https_proxy
curl -i -X POST -H 'content-type: text/json' -d #post.json \ http://127.0.0.1:18082/youtu/openliveapi/livedetectfour
Blank the variables for the call
You can set the variables just for that one curl call
http_proxy= https_proxy= curl -i -X POST -H 'content-type: text/json' -d #post.json \ http://127.0.0.1:18082/youtu/openliveapi/livedetectfour

How to call cURL without using server-side cache?

Is there a way to tell cURL command not to use server's side cache?
e.g; I have this curl command:
curl -v www.example.com
How can I ask curl to send a fresh request to not use the cache?
Note: I am looking for an executable command in the terminal.
I know this is an older question, but I wanted to post an answer for users with the same question:
curl -H 'Cache-Control: no-cache' http://www.example.com
This curl command servers in its header request to return non-cached data from the web server.
The -H 'Cache-Control: no-cache' argument is not guaranteed to work because the remote server or any proxy layers in between can ignore it. If it doesn't work, you can do it the old-fashioned way, by adding a unique querystring parameter. Usually, the servers/proxies will think it's a unique URL and not use the cache.
curl "http://www.example.com?foo123"
You have to use a different querystring value every time, though. Otherwise, the server/proxies will match the cache again. To automatically generate a different querystring parameter every time, you can use date +%s, which will return the seconds since epoch.
curl "http://www.example.com?$(date +%s)"
Neither -H 'Pragma: no-cache' nor -H 'Cache-Control: no-cache' helped me. In browser with "cmd+shift+r" (full reload) I was seeing a new version than the output of curl in terminal.
How to debug for yourself
To get the exact same result, I went to browser > F12 (Dev Tools) > Network/Requests > Right-click on the request > "Copy as cURL" and got the equivalent cURL command to the browser's call.
Then, I pasted that in the terminal and started removing the params one by one, until I found that surprisingly --compressed was making a difference in my case. (Calling CloudFront AWS)
You could try following ways to force not to keep Cache when curl.
Note: The server may or may not be configured to respect the Cache-Control header. Therefore, whether this method will work is dependent on the server or website we’re sending the HTTP request to.
curl command with the Cache-Control header
$ curl -H 'Cache-Control: no-cache, no-store' http://www.example.com
Adding the Pragma HTTP Header
$ curl -H 'Pragma: no-cache' http://www.example.com
Finally, the most common way: bypass the cache by changing the URL
curl -H 'Cache-Control: no-cache, no-store' http://www.example.com?$(date +%s)
My problem is I should use single quotes in a query string.
My code
if (event.queryStringParameters && event.queryStringParameters['Name']) {
responseMessage = 'Hello, ' + event.queryStringParameters['Name'] + '!';
}
My requests
curl https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform
returns zsh: no matches found
curl 'https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform'
returns {"message":"Hello, Terraform!"}
This didn't work for me:
curl -H 'Cache-Control: no-cache' http://www.example.com
but this ended up doing the trick:
curl -H 'Cache-Control: no-cache' http://www.example.com&someFakeParam=$RANDOM
the &someFakeParam=$RANDOM makes the URL unique each time and bypasses caching.

GET request with curl. No errors, but Content-Length equals 0

Simple GET request with curl returns empty body (Content-Length: 0):
curl -v https://www.flyorientthai.com/booking/en/index.php
On the other hand wget can handle that url just fine:
wget https://www.flyorientthai.com/booking/en/index.php
What's wrong with curl?
Turned out 'Connection: Keep-Alive' header is required. It's added by default to request with wget but not with curl.
For anyone using "Copy as cURL" in the Chrome developer tools network tab to generate a curl command — one of the lines in the generated curl command is a header that looks something like...
-H 'If-None-Match: W/"18f6a6-6p8AL7X/p71IhN/WztZm60Aue4k"'
That causes the server to return an empty 304 response if nothing's changed. Just whip it out.
Let's try "Content-Type: application/x-www-form-urlencodeds"

Resources