How to work with Vmod Xkey to invalidate object in cache - varnish

I'm using xKey Vmode plugin to purge objects through tags. I did set up my varnish configuration to work and support xkey but now I didn't find any resource on how to send that data through varnishadm or vcl. currently, I'm using HTTP ban
curl -X BAN -H 'X-Purge-Regex: 1.pbf' varnish
to invalidate with BAN.
Also is that possible to send xkey value with commoa seperated?
like: my cached URL is something like:
www.example.com/foo/xyz?name="t1;t2"
www.example.com/foo/abc?name="t1
www.example.com/foo/xyz?name="t2"
Currently, with BAN URL - i pass t1 value with regex and that is able to invalidate #1 and #2,
but now with Xkey
How to send http with xkey?
Is there a way Xkey supports multiple tags in a single request?
Can I send xkey with (xyz, t2) - With this, I want to invalidate #1 and #2.

Install vmod_xkey
In order to use vmod_xkey, you need to install it by compiling https://github.com/varnish/varnish-modules from source. Please make sure you select the right branch in GitHub, based on the Varnish version you use.
The xkey API
vmod_xkey has 2 functions:
xkey.purge(), which will immediately remove content from cache
xkey.softpurge(), which will mark content as expired, but keeps it around for asynchronous revalidation
The VCL code
Here's the VCL code you can use to invalidate content using tags:
vcl 4.1;
import xkey;
import std;
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.x-xkey-purge) {
return(synth(400,"x-xkey-purge header missing"));
}
set req.http.x-purges = xkey.purge(req.http.x-xkey-purge);
if (std.integer(req.http.x-purges,0) != 0) {
return(synth(200, req.http.x-purges + " objects purged"));
} else {
return(synth(404, "Key not found"));
}
}
}
Please ensure acl purge contains the right IP addresses or IP ranges prior to using this.
By adding import xkey; to the VCL file, secondary keys are automatically registered in Varnish, and can be used later on.
The PURGE request method is used to trigger xkey.purge() and the x-xkey-purge request header is used to specify the tags.
Registering keys
Registering keys happens by specifying them in the Xkey response header. You can register a single key, but you can also add multiple ones.
Multiple keys are separated by space or comma.
Here's an example where 3 keys are registered:
category_sports
id_1265778
type_article
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Cache-Control: public, s-maxage=60
Xkey: category_sports id_1265778 type_article
Removing content based on keys
By performing a PURGE call and by specifying the right X-Xkey-Purge value, content will be removed for these keys.
Here's an example where we remove all objects matching the category_sports tag for all pages on the http://example.com website:
PURGE / HTTP/1.1
Host: example.com
X-Xkey-Purge: category_sports
Here's another example where we invalidate content that matches the foo and bar keys:
PURGE / HTTP/1.1
Host: example.com
X-Xkey-Purge: foo bar

Related

Handling of etags in batch request using SAP Cloud SDK

I am trying to carry out a batch request including a create, update and a delete (all are different salesorders). As per this question here which deals with something similar, I have done a get for the items I want to update and delete before I add them to the batch request. I am using the SalesOrder.builder() to prepare the SalesOrder I want to create.
final ErpHttpDestination destination = DestinationAccessor.getDestination(DESTINATION_NAME)
.asHttp().decorate(DefaultErpHttpDestination::new);
final SalesOrderItem salesOrderItem1 = SalesOrderItem.builder().material(material)
.requestedQuantityUnit(requestedQuantityUnit).build();
final SalesOrder salesOrder1 = SalesOrder.builder().distributionChannel(distributionChannel)
.salesOrderType(salesOrderType).salesOrganization(salesOrganization)
.organizationDivision(organizationDivision).soldToParty(soldToParty)
.item(salesOrderItem1).build();
final SalesOrder orderToUpdate = new GetSingleSalesOrderCommand(orderToUpdateID, destination,
new DefaultSalesOrderService()).execute();
orderToUpdate.setSoldToParty(updateSoldToParty);
final SalesOrder orderToDelete = new GetSingleSalesOrderCommand(orderToDeleteID, destination,
new DefaultSalesOrderService()).execute();
SalesOrderServiceBatch service = new DefaultSalesOrderServiceBatch(
new DefaultSalesOrderService());
BatchResponse bRes = service.beginChangeSet().createSalesOrder(salesOrder1).updateSalesOrder(orderToUpdate)
.deleteSalesOrder(orderToDelete).endChangeSet().execute(destination);
I am then logging the BatchResponse and see I am getting a Batch Response Failure:
eTag handling not supported for http method 'POST'
I have searched for this error but can't find any resolution to it. Any ideas?
Thanks.
UPDATE: Increasing the logging to DEBUG I can see the batch request that is being sent and can see that there is an if-match header being added to the create request, which doesn't make sense as it can't match something that doesn't exist yet.
"msg":"--batch_123\r\nContent-Type: multipart/mixed;
boundary=changeset_(changeset number)\r\n\r\n--
changeset_(changeset number)\r\nContent-Type:
application/http\r\nContent-Transfer-Encoding: binary\r\n\r\nPOST
/sap/opu/odata/sap/API_SALES_ORDER_SRV/A_SalesOrder HTTP/1.1\r\nContent-
Length:
193\r\nIf-Match: W/\"datetimeoffset'2020-05-
01T11%3A51%3A16.8631720Z'\"\r\nAccept:
application/json;odata=verbose\r\nContent-Type:......
The I get the error:
Inner Error:
"msg":"batch
responseFailure(com.sap.cloud.sdk.odatav2.connectivity.ODataException:
null: <?xml version=\"1.0\" encoding=\"utf-8\"?><error
xmlns=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\">
<code>/IWFND/CM_MGW/537</code><message xml:lang=\"en\">eTag handling not
supported for http method 'POST'</message><innererror>...
However, what does work is if I wrap each request in its own changeset e.g.
service
.beginChangeSet().createSalesOrder(order).endChangeSet()
.beginChangeSet().updateSalesOrder(orderToUpdate).endChangeSet()
.beginChangeSet().deleteSalesOrder(orderToDelete).endChangeSet()
.execute(destination);
Edit:
This is fixed as of version 3.25.0.
Initial Answer:
This seems to be a bug. I was able to reproduce this with a different service and the behaviour is the same: The if-match header is incorrectly applied to the POST operation as well.
When debugging it seems like the request is build up correctly with the header only being present on update and delete. However, it seems that when the batch request is serialised to JSON it gets added to all requests.
So until this is fixed the workaround is isolating these operations via change sets, as you already pointed out.
Looks like eTag handling is not supported for your endpoint.
Now you can do the following to omit eTag headers:
orderToUpdate.setVersionIdentifier(null);
orderToDelete.setVersionIdentifier(null);
However I'm not sure how 'POST' fits the error description, because update uses PATCH and delete uses DELETE. The only POST that I expect would be coming from create. But we do not add headers for entity version identifiers (eTag) in OData create operation. If the same error still comes up, please try again without running createSalesOrder(salesOrder1).

Custom update actions in RESTful Services

My API requires many different types of updates that can be performed by different types of roles. A ticket can have it's data updated, a ticket can be approved (which includes some information), a ticket can be rejected, a ticket can be archived (state that makes a ticket unable to be updated), etc.
I've recently started working as a backend developer and I really do not know what is the most correct approach to this situation but I've two ideas in mind:
A single update endpoint (e.g. /api/tickets/:id) that accepts an action field with the type of update that wants to be done to that file;
Multiple custom action endpoints (e.g. /api/tickets/:id/validate, /api/tickets/:id/accept, etc.)
Which one of those is the best approach to the situation when it comes to the REST architecture? If both are incorrect when it comes to REST, what would be the most correct approach? I couldn't really find any post on SO that answered my question so I decided to create my own. Thank you!
REST stands for Representational State Transfer, which means that the client and the server affect each other’s state by exchanging representations of resources.
A client might GET a representation of a ticket like this:
GET /api/tickets/123 HTTP/1.1
HTTP/1.1 200 OK
Content-Type: application/json
{
"id": 123,
"state": "new",
"archived": false,
"comments": []
}
Now the client can PUT a new representation (replacing it wholesale) or PATCH specific parts of the existing one:
PATCH /api/tickets/123 HTTP/1.1
Content-Type: application/json-patch+json
[
{"op": "replace", "path": "/state", "value": "approved"},
{"op": "add", "path": "/comments", "value": {
"text": "Looks good to me!"
}}
]
HTTP/1.1 200 OK
Content-Type: application/json
{
"id": 123,
"state": "approved",
"archived": false,
"comments": [
{"author": "Thomas", "text": "Looks good to me!"}
]
}
Note how the update is expressed in terms of what is to be changed in the JSON representation, using the standard JSON Patch format. The client could use the same approach to update any resource represented in JSON. This contributes to a uniform interface, decoupling the client from the specific server.
Of course, the server need not support arbitrary updates:
PATCH /api/tickets/123 HTTP/1.1
Content-Type: application/json-patch+json
[
{"op": "replace", "path": "/id", "value": 456}
]
HTTP/1.1 422 Unprocessable Entity
Content-Type: text/plain
Cannot replace /id.
Still, this might require more complexity on the server than dedicated operations like “approve”, “reject”, etc. You might want to use a library to implement JSON Patch. If you find that you need many custom actions which are hard to express in terms of a representation, an RPC architecture might suit you better than REST.
REST is all about resources. And the state of the resources should be manipulated using representations (such as JSON, XML, you name it) on the top of stateless communication between client and server.
Once URIs are meant to identify the resources, it makes sense to use nouns instead of verbs in the URI. And when designing a REST API over the HTTP protocol, HTTP methods should be used to indicate the action intended to be performed over the resource.
Performing partial updates
You could use the PATCH HTTP verb to perform partial updates to your resource. The PATCH request payload must contain set of changes to applied to the resource.
There are a couple of formats that can be used to describe the set of changes to be applied to the resource:
###JSON Merge Patch
It's defined in the RFC 7396 and can be applied as described below:
If the provided merge patch contains members that do not appear within the target, those members are added. If the target does contain the member, the value is replaced. Null values in the merge patch are given special meaning to indicate the removal of existing values in the target.
So a request to modify the status of a ticket could be like:
PATCH /api/tickets/1 HTTP/1.1
Host: example.org
Content-Type: application/merge-patch+json
{
"status": "valid"
}
###JSON Patch
It's defined in the RFC 6902 and expresses a sequence of operations to be applies to a target JSON document. A request to modify the status of a ticket could be like:
PATCH /api/tickets/1 HTTP/1.1
Host: example.org
Content-Type: application/json-patch+json
[
{
"op": "replace",
"path": "/status",
"value": "valid"
}
]
The path is a JSON Pointer value, as described in the RFC 6901.
Try to either
Deal with a single object -> api/v1/tickets/1
Deal with a list of objects -> api/v1/tickets/.
Then try to capture all actions as CRUD actions.
Create object(s) -> HTTP POST
Retreive object(s) -> HTTP GET
Update object(s) -> HTTP PATCH
Delete object(s) -> HTTP DELETE
And also:
Save object(s) entirely -> HTTP PUT
When you are changing statuses, and these are just attributes on a ticket. I would send a PATCH request, for instance. If I need to change the statues of ticket #1 to "rejected" I would send something like PATCH api/v1/tickets/1 with a payload like:
{
"status": "rejected"
}
REST has a lot of best practices but not everything is set in stone. Maybe this tutorial: https://restfulapi.net can help you?
Really it all comes down to a matter of taste. It is often observed in the industry to have the static parameters in the URL (eg: /tickets/update, /users/add, /admin/accounts) and the variable parameters in the query (eg: IDs, messages, names). It allows to have a fixed number of endpoints.
I see you're using NodeJS so you're probably using Express and in Express getting the url parameters and the body parameters are equally easy:
// supposing you're using a JSON-based API with body-parser for JSON parsing
app.get('/some/path/:someParam', (req, res) => {
console.log(req.params.someParam);
console.log(req.body.someOtherParam);
res.send();
}

Parse duplicate http headers from request in Node?

In some special cases, there would exist http request headers which have duplicate values, such as XFF headers appended as following in header:
x-forwarded-for: *.*.*.*
x-forwarded-for: *.*.*.*
And by use of npm package express and request, could we parse the duplicate headers in following code?
req.headers['x-forwarded-for']
The duplicate HTTP headers will be combined into an comma-separated list, so your example the
'x-forwarded-for': '1.2.3.4'
'x-forwarded-for': '5.6.7.8'
will become
'x-forwarded-for': '1.2.3.4, 5.6.7.8'
This is done per HTTP RFC2616 available here:
Multiple message-header fields with the same field-name MAY be present
in a message if and only if the entire field-value for that header
field is defined as a comma-separated list [i.e., #(values)]. It MUST
be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the interpretation
of the combined field value, and thus a proxy MUST NOT change the
order of these field values when a message is forwarded
Here you can find a related issue in node.

Is a partial representation of document a valid "set of changes" as per HTTP PATCH RFC?

Here is what RFC 5789 says:
The PATCH method requests that a set of changes described in the request entity be applied to the resource identified by the Request-URI. The set of changes is represented in a format called a "patch document" identified by a media type. If the Request-URI does not point to an existing resource, the server MAY create a new resource, depending on the patch document type (whether it can logically modify a null resource) and permissions, etc.
The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version.
Let's say I have { "login": "x", "enabled": true }, and I want to disable it.
According to post "Please. Don't Patch Like An Idiot.", the proper PATCH request would be
[{ "op": "replace", "path": "/enabled", "value": false }]
However, let's take this request:
{ "enabled": false }
It also 'contains a set of instructions describing how a resource currently residing on the origin server should be modified', the only difference is that JSON property is used instead of JSON object.
It seems less powerful, but array changes can have some other special syntax if required (e.g. {"a":{"add":[], "remove":[]}}), and server logic might not be able to handle anything more powerful anyway.
Is it an improper PATCH request as per RFC? And if so, why?
And, on the other hand, would a { "op": "disable" } be a correct PATCH request?
the only difference is that JSON property is used instead of JSON object.
It's actually a bit deeper than that. The reference to RFC 6902 is important. The first request has a Content-Type of application/json-patch+json, but the second is application/json
The important thing is that you use a 'diff media type,' one that's useful for this purpose. You don't have to use JSON-PATCH, (I'm a big fan of json-merge-patch), but you can't just use anything you want. What you're asking about in the second part is basically 'can I make my own media type' and the answer is 'yes,' just please document it and register it with the IANA.

How does nodejs handle multiple www-authenticate headers in a http response?

How does the nodejs http.IncomingMessage object handle multiple WWW-Authenticate headers in a response given that the HTTP specification supports this and that the response headers seem to be made available only through the response.headers object which is header-name : value?
Multiple WWW-Authenticate headers are string-concatenated into a single www-authenticate property, separated by a comma + space.
For example,
GET / HTTP/1.1
WWW-Authenticate: foo
WWW-Authenticate: bar
Will result in
req.headers['www-authenticate'] == 'foo, bar'
This is in line with RFC 2616 § 4.2, which states:
Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.

Resources