Usually I request a file through a Varnish proxy, it returns a response from the cache, or accesses the backend for this.
Is there any way to request file from cache, or get 404 without forcing the Varnish to access the backend?
The following test case shows a possible approach:
varnishtest "..."
server s1 {
rxreq
txresp -hdr "Cache-Control: s-maxage=60"
} -start
varnish v1 -vcl+backend {
sub vcl_miss {
if (!req.http.X-Warm-Cache) {
return (synth(404));
}
}
} -start
client c1 {
txreq -url "/foo" -hdr "X-Warm-Cache: 1"
rxresp
expect resp.status == 200
txreq -url "/foo"
rxresp
expect resp.status == 200
txreq -url "/bar"
rxresp
expect resp.status == 404
} -run
Related
I have tcl expect script to daily check some version on the network devices.
#!/usr/bin/expect --
set env(TERM) vt100
set timeout 5
set username [lindex $argv 0]
set password [lindex $argv 1]
set hostname [lindex $argv 2]
set success 1
#login script
log_user 1
spawn ssh $username#$hostname
expect {
"Connection refused" {
spawn telnet $hostname
expect {
"?sername:" {
if { $success == 0 } {
log_user 1
puts "error user or password incorrect"
exit 1;
} else {
incr success -1
send "$username\r"
}
exp_continue
}
"?assword:" {
send "$password\r"
exp_continue
}
timeout {
log_user 1
puts "error could not ssh or telnet devices"
exit 1;
exp_continue
}
"#" {
send "terminal length 0\r"
}
}
exp_continue
}
timeout {
spawn telnet $hostname
expect {
timeout {
log_user 1
puts "error could not ssh or telnet devices"
exit 1;
exp_continue
}
"?..." {
log_user 1
puts "error could not ssh or telnet devices"
exit 1;
exp_continue
}
"?sername:" {
if { $success == 0 } {
log_user 1
puts "error user or password incorrect"
exit 1;
} else {
incr success -1
send "$username\r"
}
exp_continue
}
"?assword:" {
send "$password\r"
exp_continue
}
"#" {
send "terminal length 0\r"
}
}
exp_continue
}
"continue connecting (yes/no)?" {
send "yes\r"
exp_continue
}
"?assword:" {
if { $success == 0 } {
log_user 1
puts "error user or password incorrect"
exit 1;
} else {
incr success -1
send "$password\r"
}
exp_continue
}
"$ " {
send "ssh keygen -R $hostname\r"
exp_continue
}
"#" {
send "terminal length 0\r"
}
}
#execute script
expect "#"
send "\r"
expect "#"
log_user 1
send "show version\r"
expect "#"
send "exit\r"
My expect script working like this,
ssh to the devices.
If ssh not working ("Connection refused" or "timeout") it will go to telnet condition
If ssh known_hosts key is not present on linux devices, it will send "yes"
If ssh key on the remote devices is changed, it will send "ssh-keygen -R $hostname"
and the problem is, my expect program from could not work if ssh key on remote devices is changed.
[linux]$ ./sshtelnet.tcl mine password 1.1.1.1
spawn ssh mine#1.1.1.1
###########################################################
# WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! #
###########################################################
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
83:24:a5:c4:2c:98:0d:0b:d6:ad:cb:74:12:7e:84:83.
Please contact your system administrator.
Add correct host key in /home/linux/.ssh/known_hosts to get rid of this message.
Offending key in /home/linux/.ssh/known_hosts:152
RSA host key for 1.1.1.1 has changed and you have requested strict checking.
Host key verification failed.
expect: spawn id exp4 not open
while executing
"expect "#""
(file "./sshtelnet.tcl" line 106)
When the remote host identification has changed, the ssh process terminates. Assuming you know that this is not because "someone is doing something nasty", you want to execute the "ssh-keygen -R $hostname" locally, not send it to the spawned ssh process. After cleaning up the offending key, you have to spawn the ssh command again.
The easiest way to be able to repeat the ssh command is to put things in procs:
proc connectssh {username password hostname} {
global spawn_id
set success 1
spawn ssh $username#$hostname
expect {
"Connection refused" {
connecttelnet $username $password $hostname
}
timeout {
connecttelnet $username $password $hostname
}
"continue connecting (yes/no)?" {
send "yes\r"
exp_continue
}
"?assword:" {
if {$success == 0} {
log_user 1
puts "error user or password incorrect"
exit 1;
} else {
incr success -1
send "$password\r"
}
exp_continue
}
"Host key verification failed" {
wait
exec ssh-keygen -R $hostname
connectssh $username $password $hostname
}
"#" {
send "terminal length 0\r"
}
}
}
proc conecttelnet {username password hostname} {
global spawn_id
spawn telnet $hostname
expect {
# ...
}
}
set env(TERM) vt100
set timeout 5
lassign $argv username password hostname
log_user 1
connectssh $username $password $hostname
# The rest of your script
I'm confused why I'm not getting any content back from the following function, which uses reqwest:
fn try_get() {
let wc = reqwest::Client::new();
wc.get("https://httpbin.org/json").send().map(|res| {
println!("{:?}", res);
println!("length {:?}", res.content_length());
});
}
I'm expecting this function to display the response object and then give me the content length. It does the first but not the second:
Response { url: "https://httpbin.org/json", status: 200, headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "connection": "keep-alive", "content-type": "application/json", "date": "Tue, 26 Feb 2019 00:52:47 GMT", "server": "nginx"} }
length None
This is confusing because if I hit the same endpoint using cURL, it gives me a body as expected:
$ curl -i https://httpbin.org/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Tue, 26 Feb 2019 00:54:57 GMT
Server: nginx
Content-Length: 429
Connection: keep-alive
{
"slideshow": {
"author": "Yours Truly",
"date": "date of publication",
"slides": [
{
"title": "Wake up to WonderWidgets!",
"type": "all"
},
{
"items": [
"Why <em>WonderWidgets</em> are great",
"Who <em>buys</em> WonderWidgets"
],
"title": "Overview",
"type": "all"
}
],
"title": "Sample Slide Show"
}
}
What is the problem with my function that it does not provide me with the content length?
The reqwest documentation for content_length() is always a good place to start. It states
Get the content-length of the response, if it is known.
Reasons it may not be known:
The server didn't send a content-length header.
The response is gzipped and automatically decoded (thus changing the actual decoded length).
Looking at your example curl output, it contains Content-Length: 429 so the first case is covered. So now lets try disabling gzip:
let client = reqwest::Client::builder()
.gzip(false)
.build()
.unwrap();
client.get("https://httpbin.org/json").send().map(|res| {
println!("{:?}", res);
println!("length {:?}", res.content_length());
});
which logs
length Some(429)
so the second case is the issue. By default, reqwest appears to be automatically handling gzipped content, whereas curl is not.
The Content-Length HTTP header is entirely optional, so generally relying on its presence would be a mistake. You should read the data from the request using the other reqwest APIs and then calculate the length of the data itself. For instance, you might use .text()
let wc = reqwest::Client::new();
let mut response = wc.get("https://httpbin.org/json").send().unwrap();
let text = response.text().unwrap();
println!("text: {} => {}", text.len(), text);
Similarly, for binary data you can use .copy_to():
let wc = reqwest::Client::new();
let mut response = wc.get("https://httpbin.org/json").send().unwrap();
let mut data = vec![];
response.copy_to(&mut data).unwrap();
println!("data: {}", data.len());
I have this:
if (bereq.http.X-Path ~ "[a-z0-9]+\.(js|css)$") {
set beresp.http.Cache-Control = "max-age=259200";
}
And I need to write a test.vtl to demonstrate that is working. I'm trying with:
client c1 {
txreq -url "/content/css/main.min.aer234vcvb.css"
rxresp
} -run
But it fails.
the problem is that you - match your condition on X-Path incomming http header.
therefore in your client you are not sending it in, and the condition just does not match.
see this sample, based on your config, but sending in X-Path.
varnishtest "Test Cache-Control on X-Path"
server s1 {
rxreq
txresp
} -start
varnish v1 -vcl+backend {
sub vcl_backend_response {
if (bereq.http.X-Path ~ "[a-z0-9]+\.(js|css)$") {
set beresp.http.Cache-Control = "max-age=259200";
}
}
} -start
client c1 {
txreq -url "/content/css/main.min.aer234vcvb.css" -hdr "X-Path: /content/css/main.min.aer234vcvb.css"
rxresp
expect resp.http.Cache-Control == "max-age=259200"
} -run
you might want to check against bereq.url and not bereq.http.X-Path ?!
The problem
I am trying to use boolean fields in a script to score. It seems like doc['boolean_field'].value can't be manipulated as a boolean, but _source.boolean_field.value can (even though the Scripting documentation here says "The native value of the field. For example, if its a short type, it will be short.").
What I've tried
I have a field called 'is_new'. This is the mapping:
PUT /test_index/test/_mapping
{
"test": {
"properties": {
"is_new": {
"type": "boolean"
}
}
}
}
I have some documents:
PUT test_index/test/1
{
"is_new": true
}
PUT test_index/test/2
{
"is_new": false
}
I want to do a function_score query that will have a score of 1 if new, and 0 if not:
GET test_index/test/_search
{
"query": {
"function_score": {
"query": {
"match_all": {}
},
"functions": [
{
"script_score": {
"script": "<<my script>>",
"lang": "groovy"
}
}
],
"boost_mode": "replace"
}
}
}
Scripts work when I use the _source.is_new.value field, but don't if I use doc['is_new'].value.
This works:
"if ( _source.is_new) {1} else {0}"
These don't work:
"if ( doc['is_new'].value) {1} else {0}" (always true)
"if ( doc['is_new'].value instanceof Boolean) {1} else {0}" (value isn't a Boolean)
"if ( doc['is_new'].value.toBoolean()) {1} else {0}" (always false)
"if ( doc['is_new']) {1} else {0}" (always true)
I've checked the value, and it thinks it is a string, but I can't do string comparison:
"if ( doc['is_new'].value instanceof String) {1} else {0}" (always true)
"if ( doc['is_new'].value == 'true') {1} else {0}" (always false)
"if ( doc['is_new'].value.equals('true')) {1} else {0}" (always false)
Is this broken, or am I doing it wrong? Apparently it is faster to use doc['field_name'].value, so if possible, it would be nice if this worked.
I am using Elasticsearch v1.4.4.
Thanks!
Isabel
I'm having the same issue on ElasticSearch v1.5.1: Boolean values in the document show up as characters in my script, T' for true and 'F' for false:
if ( doc['is_new'].value == 'T') {1} else {0}
I've just got it!
First, it works only with _source.myField, not with doc['myField'].value.
I think there's a bug there because the toBoolean() method should return a boolean depending on the actual value, but it doesn't.
But I also needed to declare the mapping of the field explicitly as boolean and not_analyzed.
I hope it helps!
I am trying to use OAuth client_credentials with Windows Microsoft Azure. I can successfully generate an access_token but get when I try and access https://graph.windows.net/mydomain.com/users?api-version=2013-04-05 I get 403 Forbidden.
What am I doing wrong? I think I'm not requestion a sufficiently authorized access_token but I cannot find/guess other resource or params to add.
My current flow
Request
> POST /mydomain.com/oauth2/token HTTP/1.1
> Host: login.windows.net
> Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
&client_id={client_id}
&client_secret={client_secret}
&resource=https://graph.windows.net
Response
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
{
"access_token": "<string>",
"token_type": "Bearer",
"expires_in": "<seconds>",
"expires_on": "<timestamp>",
"resource": "https://graph.windows.net"
}
Request
> GET /mydomain.com/users
> ?api-version=2013-04-05 HTTP/1.1
> Host: graph.windows.net
> Authorization: Bearer {access_token}
Response
< HTTP/1.1 403 Forbidden
{
"odata.error": {
"code": "Authorization_RequestDenied",
"message": {
"lang": "en",
"value": "Insufficient privileges to complete the operation."
}
}
}
Note: I get 403 Unauthorized if I miss out or mess with the {access_token}.
Per my comment, you need to make sure that you've selected the proper permissions from the "permissions to other applications" drop-down in the portal, otherwise access won't be granted. See this page in the documentation for more information.