In some special cases, there would exist http request headers which have duplicate values, such as XFF headers appended as following in header:
x-forwarded-for: *.*.*.*
x-forwarded-for: *.*.*.*
And by use of npm package express and request, could we parse the duplicate headers in following code?
req.headers['x-forwarded-for']
The duplicate HTTP headers will be combined into an comma-separated list, so your example the
'x-forwarded-for': '1.2.3.4'
'x-forwarded-for': '5.6.7.8'
will become
'x-forwarded-for': '1.2.3.4, 5.6.7.8'
This is done per HTTP RFC2616 available here:
Multiple message-header fields with the same field-name MAY be present
in a message if and only if the entire field-value for that header
field is defined as a comma-separated list [i.e., #(values)]. It MUST
be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the interpretation
of the combined field value, and thus a proxy MUST NOT change the
order of these field values when a message is forwarded
Here you can find a related issue in node.
Related
I'm using xKey Vmode plugin to purge objects through tags. I did set up my varnish configuration to work and support xkey but now I didn't find any resource on how to send that data through varnishadm or vcl. currently, I'm using HTTP ban
curl -X BAN -H 'X-Purge-Regex: 1.pbf' varnish
to invalidate with BAN.
Also is that possible to send xkey value with commoa seperated?
like: my cached URL is something like:
www.example.com/foo/xyz?name="t1;t2"
www.example.com/foo/abc?name="t1
www.example.com/foo/xyz?name="t2"
Currently, with BAN URL - i pass t1 value with regex and that is able to invalidate #1 and #2,
but now with Xkey
How to send http with xkey?
Is there a way Xkey supports multiple tags in a single request?
Can I send xkey with (xyz, t2) - With this, I want to invalidate #1 and #2.
Install vmod_xkey
In order to use vmod_xkey, you need to install it by compiling https://github.com/varnish/varnish-modules from source. Please make sure you select the right branch in GitHub, based on the Varnish version you use.
The xkey API
vmod_xkey has 2 functions:
xkey.purge(), which will immediately remove content from cache
xkey.softpurge(), which will mark content as expired, but keeps it around for asynchronous revalidation
The VCL code
Here's the VCL code you can use to invalidate content using tags:
vcl 4.1;
import xkey;
import std;
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.x-xkey-purge) {
return(synth(400,"x-xkey-purge header missing"));
}
set req.http.x-purges = xkey.purge(req.http.x-xkey-purge);
if (std.integer(req.http.x-purges,0) != 0) {
return(synth(200, req.http.x-purges + " objects purged"));
} else {
return(synth(404, "Key not found"));
}
}
}
Please ensure acl purge contains the right IP addresses or IP ranges prior to using this.
By adding import xkey; to the VCL file, secondary keys are automatically registered in Varnish, and can be used later on.
The PURGE request method is used to trigger xkey.purge() and the x-xkey-purge request header is used to specify the tags.
Registering keys
Registering keys happens by specifying them in the Xkey response header. You can register a single key, but you can also add multiple ones.
Multiple keys are separated by space or comma.
Here's an example where 3 keys are registered:
category_sports
id_1265778
type_article
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Cache-Control: public, s-maxage=60
Xkey: category_sports id_1265778 type_article
Removing content based on keys
By performing a PURGE call and by specifying the right X-Xkey-Purge value, content will be removed for these keys.
Here's an example where we remove all objects matching the category_sports tag for all pages on the http://example.com website:
PURGE / HTTP/1.1
Host: example.com
X-Xkey-Purge: category_sports
Here's another example where we invalidate content that matches the foo and bar keys:
PURGE / HTTP/1.1
Host: example.com
X-Xkey-Purge: foo bar
I am trying to pull data from a REST API that uses a "similar standard to JSON RPC". The params I am passing look right according to the documentation here and here.
The error I am receiving is ...message:"Header missing request ID".... I am unsure what I am missing that would properly declare the requestID.
I have looked at the documentation provided via the API I am trying to pull data from but it's not very helpful considering it's all in PHP and cURL. I am trying to complete this task using python-requests.
getParams = {'method': 'getCustomers', 'params':{'where':'', 'limit': 2}, 'id': 'getCustomers'}
Result:
{"result":null,"error":{"code":102,"message":"Header missing request ID","data":[]},"id":null}
The return result should contain a list of All Customers and their attributes in JSON format.
Turns out there is nothing wrong with the code I am using. There is an issue with the API I am attempting to call.
In my situation, I was getting the same error back and was required to send a X-Request-ID header. I fixed it by adding the following to my headers:
headers = {
'X-Request-ID': str(uuid.uuid1()) # generate GUID based on host & time
...
Note that (for me) the GUID needed to be of a specific format (e.g. matching the Regex ^[{]?[0-9a-fA-F]{8}-([0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}[}]?$
taken from https://www.geeksforgeeks.org/how-to-validate-guid-globally-unique-identifier-using-regular-expression/). For example, it still gave the same error if I just sent "test".
I would like to do a DELETE request with unspecified number of parameters a=someValue. There is 2 main ways of supplying parameters to my understanding
Query parameters. ?a=someValue . This approach turn everything into
string and since I allow any number of parameters, I cannot know
which one is String, Boolean or Integer
Parameters in Body.This approach goes against the spec of DELETE operation to not have a body. Some server even strip away the body-content. But as I
send an json object, user can specify which type of value each of
their parameters has.
What would be your approach for this?
I'd use query parameters over body as the DELETE method has an optional body. Some clients may choose to ignore the body totally.
How does the nodejs http.IncomingMessage object handle multiple WWW-Authenticate headers in a response given that the HTTP specification supports this and that the response headers seem to be made available only through the response.headers object which is header-name : value?
Multiple WWW-Authenticate headers are string-concatenated into a single www-authenticate property, separated by a comma + space.
For example,
GET / HTTP/1.1
WWW-Authenticate: foo
WWW-Authenticate: bar
Will result in
req.headers['www-authenticate'] == 'foo, bar'
This is in line with RFC 2616 ยง 4.2, which states:
Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.
Why are there hexadecimal numbers included in my view results from CouchDB? How can I get rid of them?
7f
{"total_rows":108,"offset":0,"rows":[
{"id":"5c718dbd01bc0cde8152e08ed6003405","key":"2013-03-19T22:43:27.2683661Z","value":0}
5b
,
...
{"id":"5c718dbd01bc0cde8152e08ed6037404","key":"2013-03-19T23:07:35.5972058Z","value":0}
5b
,
{"id":"5c718dbd01bc0cde8152e08ed60376e5","key":"2013-03-19T23:07:35.6062063Z","value":0}
4
]}
1
0
TL;DR
I am new to CouchDB, and are investigating its use as a database for an event log. I have created a simple map function to view the event log by date:
function(doc)
{
if (doc.DateTime)
{
emit(doc.DateTime, doc);
}
}
When I use Fiddler to test this view with the following request:
GET http://localhost:5984/stuff/_design/eventlog/_view/datetime
Host: localhost:5984
User-Agent: Fiddler
The results returned included hexadecimal numbers that aren't a part of the JSON structure. Hence the JSON returned is invalid. Why are these hexadecimal numbers included in the results, and how can I get rid of them?
I am using Windows (x86) CouchDB version 1.2.1.
The weird hex numbers are used for so-called chuncked transfer-encoding. This is a way for HTTP responses to become available in a streaming format instead of the client having to wait for entire response to be ready. The hex numbers denote the length for the next chunk.
I think the use of chunking is independent of the request's Accept values, but I'm not sure.
To get a pure JSON result you must include the Accept: application/json header in your HTTP request.
If you omit the the Accept header CouchDB will return results in a manner that is more suitable for displaying nicely in web browsers. The results will be in a JSON format, but with a text/plain content-type.
See Apache CouchDB 1.3 Manual Section 2.2.1. Request Headers.
The hexadecimal numbers are a result of Chunked transfer encoding.