I have a scenario where my URL will be either contains a comma delimiter with value or without.
i.e. /api/parameters/XXXXXXXXXX?tables=x0 or tables=x0;x1;x2.
now based on this URL I want to check in the varnish that, if URL contains multiple values as tables then separate that out and pass each table name in seperate URL (/api/parameters/XXXXXXXXXX?tables=x0, /api/parameters/XXXXXXXXXX?tables=x1, /api/parameters/XXXXXXXXXX?tables=x2) either to cache if miss then backend server.
then based on the response of this need to combine the result and return it to the client.
my question here is:
How to segregate the value from the URL and pass a modified URL to varnish cache or backend.
after returning the result I want to return it as a combined JSON object in a sequence of which it was originally requested with a comma delimiter(i.e. x0 result;x1 result;x2 result).
It is possible to turn a single request into multiple subrequests in Varnish. Unfortunately this cannot be done with the open source version, only with the Enterprise version.
vmod_http
https://docs.varnish-software.com/varnish-cache-plus/vmods/http/ describes how you can perform HTTP calls from within Varnish using vmod_http.
By sending HTTP requests to other URLs through Varnish, you can get multiple objects out of the cache and aggregate them into a single response
No looping
The fact that Varnish doesn't have loops makes matters a bit more complicated. You'll have so set an upper limit to the amount of values the tables querystring parameter has and you'll have to check the values using individual if-statements.
Returning the combined JSON output
Once you have fetched the results from the various URLs, you can create a JSON string and return it via return(synth(200,req.http.json)). Where req.http.json contains the JSON string.
This will create a synthetic response.
In Varnish Enterprise it is also possible to cache synthetic output. See https://docs.varnish-software.com/varnish-cache-plus/vmods/synthbackend/ to learn more about vmod_synthbackend.
Varnish Enterprise disclaimer
The solution I suggested in my answer uses Varnish Enterprise, the commercial version of Varnish. It extends Varnish capabilities with additional VMODs and features, which you can read about here. One easy way to try it out without upfront licensing payments, if you’re interested, is to spin up an instance on cloud infrastructure:
Varnish Enterprise on AWS
Varnish Enterprise on Azure
Varnish Enterprise on GCP
Related
I have an Eve instance running and pagination enabled. In some cases I want to request all items under a resource. This is done together with a projection to get a full list of ids.
This question is very similar to another question, but this question concerns external requests rather than internal calls.
I have tried setting max_results to 0 and -1 but both yield a single result. Is there a way to request all items without disabling pagination globally?
Edit My current solution to circumvent this is a custom flask endpoint which just access the database directly. The issue with this approach is that I would like to add various projects and make use of Eve's database optimizations. All of which I need to manually reimplement.
newbie here
I want to try the Netlify split testing feature which basically split the traffic randomly on multiple GitHub branches (but keeps the same URL).
But what I need to do is instead of splitting the traffic randomly on different versions, I want to split the traffic based on conditions, especially by using the document.referrer request.
For example, a user from Facebook will see the site from branch A, and others will see branch B.
Is there any way to do this?
Thank you.
It doesn't look like Netlify has built-in referrer targeting options built into their split testing product. At least, not according to their docs. Tools like Google Optimize and Optimizely provide options to split test against the HTTP Referer header, which is the URL of the site the user was on before they hit your page.
Netlify does, however, mention the following on the above page:
We set a cookie called nf_ab to ensure that the same visitor always gets the same branch. By default, the value of the cookie is a random number between zero and one and is configured out of the box to ensure that your site visitors have a consistent experience. If you'd like your visitors to manually opt in to a split test, you can also use client-side JavaScript to manually set the value of the nf_ab cookie to a branch name, which Netlify's CDN will read and serve accordingly.
So I believe your only option would be to write custom client-side JS that checks the HTTP Referer header value and sets the value of the nf_ab cookie used by Netlify to the branch you want that user to be served a version of your site from.
I know what is the difference between Azure CDN query string modes and I have read a helpfull example of query string modes but...
I don't understand what is the purpose of "Ignore query strings" or how this can be useful in a real dynamic web.
For example, suppose we have a product purchase website with a URL similar to www.myweb.com/products?id=3
If we use "Ignore query strings"... Does this mean that if an user later requests product 4 (www.myweb.com/products?id=4), he will receive the page for product 3?
I think I'm not understanding correctly Azure CDN, I'm seeing Azure CDN as a dynamic content CDN, however Azure CDN is only used for static content as this article explains:
Standard content delivery network (CDN) capability includes the ability to cache files closer to end users to speed up delivery of static files.
This is correct? Any help or example on the subject is welcome
Yes, if you are selected Ignore query strings Query string caching behavior (this is the default), in your case subsequent requests after the initial request www.myweb.com/products?id=3, no matter the query string value, that POP server will serve the same content until it's cache period expires.
And for the second question, CDN is all about serving static files. To my understanding i believe what the article says is about dynamic site accelaration. It's about bunch of techniques to optimize dynamic web sites content serving performance. Because unlike static web sites, dynamic web sites assets (static files. ex: images, js, css, html) are loading dynamically based on the user behavior.
Now that I have it clearer, I will answer my question:
Azure CDN - Used to cache static content, even on dynamic web pages.
For the example in the question, all products must download the same javascript and css content, for those types of files Azure CDN is used. Real example using "Ignore query strings":
User A www.myweb.com/products?id=3, jquery-versionX.js and mystyles.css are not cached, the server is requested and the user receives it.
User B www.myweb.com/products?id=4, since we are using "Ignore query strings" the jquery-versionX.js and mystyles.css files are cached, they are served to the user without requesting it from the server again.
User C www.myweb.com/products?id=3, since we are using "Ignore query strings" the jquery-versionX.js and mystyles.css files are cached, they are served to the user without requesting it from the server again.
Reddis or other similar - Used to cache dynamic content (queries to databases for example).
For the example in the question, all the products have different information, which is obtained by doing a database query. We can store those queries or JSON objects in a Reddis cache. Real example:
User A www.myweb.com/products?id=3, product 3 is not cached, it is requested from the server and received by the user.
User B www.myweb.com/products?id=4, product 4 is not cached, it is requested from the server and received by the user.
User C www.myweb.com/products?id=3, product 3 is cached, the server is not requested and the user receives it from the cache.
Summary:
Both methods can be used simultaneously, Azure CDN is for static content and Reddis or similar for dynamic content.
Is it possible to use cache based on a key in the request payload?
Eg. let's say we got a json or xml request payload where one of the elements is CustomerId.
Would it then be possible to cache based on CustomerId?
Thanks
I hope I understood your query properly and am not too late. I think you want to cache only when 'CustomerId' is present in the input OR it contains a certain value.
You can refer to the samples given in the foll link
https://azure.microsoft.com/en-us/blog/policy-expressions-in-azure-api-management/
It will help you to write policy expressions to check the presence or value of a particular field. Then you can cache or ignore based on that.
On a side note, Custom Caching is also something cool to check
https://learn.microsoft.com/en-us/azure/api-management/api-management-sample-cache-by-key
I am developing an ASP.NET Web API application which responds to the clients with a custom datetime http header. While I have read several articles describing how to remove response headers from ASP.NET/IIS, this one always seem to be resilient, i can't get rid of it. It seems to be placed in the response pipeline somewhere out of the programmer's/administrator's control, at the very end.
I know it may be a bad practice not to include the "Date" header in the response but, as i mentioned, the custom datetime header (which is in ticks instead of a string representation) makes the default one redundant; furthermore, this is a private API, so i know exactly who and how uses it.
Is it thus possible in any way to remove this header in IIS (v7+) for a specific site (or directly from the Web API application)?
Edit:
I tried (without success) the following techniques:
Creating a custom handler to remove the header right from the Web API project
Registering a custom IHttpModule
Explicit removal of headers in web.config in <httpProtocol><customHeaders> section
Remove HTTP response headers in IIS Manager
Header removal code in protected void Application_PreSendRequestHeaders(object sender, EventArgs e) method in Global.asax.cs
According to HTTP Spec, Date header is mandatory, except for these conditions which I dont think apply to your case:
Origin servers MUST include a Date header field in all responses, except in these cases:
1. If the response status code is 100 (Continue) or 101 (Switching
Protocols), the response MAY include a Date header field, at
the server's option.
2. If the response status code conveys a server error, e.g. 500
(Internal Server Error) or 503 (Service Unavailable), and it is
inconvenient or impossible to generate a valid Date.
3. If the server does not have a clock that can provide a
reasonable approximation of the current time, its responses
MUST NOT include a Date header field. In this case, the rules
in section 14.18.1 MUST be followed.
This is not going to be possible from within the actual WebApi/Mvc pipeline, so options like action filters and delegating handlers are out.
Instead you will probably need to implement a custom IHttpModule and register it inside IIS. There is an article here you should read and follow. The approach is very simple and easy to adapt.
Just replace the set in that example with:
HttpContext.Current.Response.Headers.Remove("Date");