Varnish how to return same cache for multiple domains - varnish

MI have two domains pointing to the same nginx server. this is my setup and how I want to run my domains.
My problem is that When a user for example requests site1.com/page/ its cache is diffrent than site2.com/page.
I want both site1.com/page/ & site2.com/page/ to return the same cache so the server does not store cache twice.
my vcl setup is set to 127.0.0.1
backend default {
.host = "127.0.0.1";
.port = "8081";
}
Nginx
server {
listen 8081;
server_name site1.com site2.com;
What rule or config can I add to make make varnish treat site1.com & site2.com the same?
Basically I want varnish to ignore the hostname(domain) & cache base on the URL and other hash data.
my varnish version is 4.0

The vcl_hash subroutine in VCL is responsible for creating the hash that is used to lookup objects in the cache. The Varnish built-in VCL for vcl_hash goes as follows:
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
As you can see, it takes both the URL (through req.url) and the Host header (through req.http.host and server.ip if there is no host) to create the lookup hash.
The Host header is important here because it identifies the site that needs to be cached. If you're accelerating multiple websites on a single Varnish server, the Host header ensures site1.com/page and site2.com/page don't share the same cache.
You seem to want the exact opposite. This means that instead of relying on the built-in VCL, you have to specify a custom vcl_hash definition that will look like this:
sub vcl_hash {
hash_data(req.url);
return (lookup);
}
This will share the cache between all websites and will only use the URL to identify objects in the cache.
If you want to have more security in place, you can also share the cache between sites explicitly, rather than sharing the cache for every single site.
Here's some example VCL code that only allows the cache to be shared between site1.com and site2.com:
sub vcl_hash {
if(req.http.host == "site1.com" || req.http.host == "site2.com") {
hash_data(req.url);
return (lookup);
}
}
If the condition is not matched, no return statement is executed, which means the built-in VCL is still used.
WARNING: I noticed that you use Varnish version 4. Please do not use this version as it is end-of-life. There are no more updates happening for this version and no security fixes. There are known security vulnerabilities for Varnish 4. Please upgrade to Varnish 6.0 LTS or one of the fresh releases.

Related

Generic domain part with fixed subdomain using Caddy and auto SSL?

I'd like to setup a Caddy server where the subdomain is static but the domain part is "wildcard", such as "api.*"
From my understanding of Caddy, the wildcard is possible for one part of the full domain (*.domain.com matches bar.domain.com but not foo.bar.domain.com).
Moreover, this configuration would automatically create a SSL certificates (which Caddy does in general, but I'm not sure here) for any new DNS entry that points to my server with a domain starting with "api.*".
The "*" here would be the domain directly, not any subdomain (it would work for api.domain.com, but not for api.foo.domain.com).
Is this something possible using a simple Caddy command (such as api.* { ... }, which I tried without luck), or does it need a more complex implementation?
Thank you for your help!
I found a working solution with the help of the Caddy Community.
Here's the code :
{
on_demand_tls {
ask https://static.site.com/domain/verify
interval 2m
burst 5
}
}
static.site.com {
...
}
:443 {
tls {
on_demand
}
// Your custom config, for instance:
reverse_proxy * ...
}
The nifty part is the tls { on_demand } part for your generic HTTPS, which will create a certificate automatically. But, this can be abused by anyone that points one of their DNS entry to your server.
So to avoid that, the Caddy community highly recommends you to set a on_demand_tls that will query an endpoint, and allow the SSL certificate to be created only if that endpoint returns true.
NOTE: The ask is a GET request that DO NOT FOLLOW redirects! Anything but a 200 status code will be considered a failure, even a 3xx!
The ask url will have the ?domain appended and will allow you to verify that domain against your logic, such as custom value in the domain like "starting by static.*", and verify that the domain exists in your database (for example).
If your URL already contains some query parameter, don't worry, Caddy is clever enough to add them. (https://static.site.com/domain/verify?some=query will become https://static.site.com/domain/verify?some=query&domain={domain}.
Caddy support https for the ask parameter, and that URL can also be external with no problems at all (no need for localhost or local server configuration).
I met the same problem, and after 1 day's stucking, here is my solution:
Assuming the site name is: site.com, and I want caddy handle these domains for me:
a.dot.site.com
b.dot.site.com
c.dot.site.com
a.eth.site.com
b.eth.site.com
c.eth.site.com
1.make sure you set SSL access available. e.g. via cloudflare:
2.set the A address pointing to your Caddy server's IP.
2.Caddy file should looks like:
# the key is: you have to list all the patterns for your multiple subdomains
*.site.com *.eth.site.com *.dot.site.com {
reverse_proxy 127.0.0.1:4567
log {
output file /var/log/access-wildcard-site.com.log
}
tls {
dns cloudflare <your cloud flare api key>
}
}

Azure Application Gateway Redirection from empty hostname

I have created an Application Gateway that needs to fulfill the working of my previous Resource (F5).
As a listener I use a hostname: hostname.stackoverflow.com that listens on 443
As a Http Setting I am using a specific port being 4443
As a BackEnd pool I use the URL/FQDN of my dev VM.
This totally works If i create a VM in the VNET and add "hostname.stackoverflow.com" to the hosts file with the ip of the application gateway.
Now I want to get a little further and add paths to my Application Gateway.
The goal is that if I use "hostname.stackoverflow.com" I need to redirect this to "Hostname.stackoverflow.com/login.aspx?guestlogin".
As far I have tried the following.
Add the "/login.aspx?guestLogin" to the HTTPS settings like this.
When I try this inside my VM. The URL changes but the path that I added there was not added in the right way, This is what I got:
So That made me think override backend path is maybe not the right way to do this.
Wanted To create a Redirection Rule That will redirect my "hostname.stackoverflow.com" to the "hostname.stackoverflow.com/login.aspx?guestLogin" But in the settings of the Application Gateway I need to provide a source path (meaning: I can not redirect from an empty hostname to a new url I think)
I am very new to Azure and even more new to the Application Gateway. Is there something that I did wrong. Is there a better way to do this ?
The iRule that I need to get in Application Gateway is as followed.
if { [string tolower [HTTP::host]] equals "hostname.stackoverflow.com" } {
if {[HTTP::path] eq "/"} {
HTTP::redirect "login.aspx?guestLogin"
}
elseif {[string tolower [HTTP::uri]] starts_with "/login.aspx?id="} {
set tail [string range [HTTP::uri] 12 end]
HTTP::redirect "login.aspx?guestLogin&$tail"
}
pool default.pool
}

How does Tumblr approach custom domain mapping?

I searched all over but can't find a clear answer or even an engineering blog post to describe how companies map custom domains to their application.
For example, let's say I have a Tumblr page with a URL of www.ashley.tumblr.com. The site allows you to add a custom domain so that visiting www.Ashley.com will render www.ashley.tumblr.com with full support for additional pages and directories.
What is the technical name for developing this?
There's no single name for what they're doing - which is engineering their HTTP/web-server code to handle requests from arbitrary HTTP request Host: header and mapping them to their existing Tumblr accounts. It has nothing to do with DNS other than requiring the owner of a custom domain-name to change their A, AAAA, or CNAME records to point to the same host as the non-custom domain (to guarantee this happens it's usual to make the custom domain-name a CNAME for the non-custom domain, in case the non-custom domain's IP address is subject to change).
Exposition time! - Most conventional web-servers (Apache, IIS) are built around the concept of a "website": a physical directory mapped to requests corresponding to a predefined list of HTTP Host: header values (or some wildcard matching pattern) and protocol and port bindings. For example, you'd add an entry called "MyWebsite.com" (the Website Name) that accepts requests to mywebsite.com and www.mywebsite.com (as these are two distinct Host: header values) and maybe some more, like secure.mywebsite.com using HTTPS on port 443).
More modern lightweight webservers and reverse-proxies (like nginx and Node.js' Express) dispense with physical directory mapping and let the application code entirely decide how to route requests within the application's logic (this is what a "router" and/or "demultiplexer" (demux) does in web-application terminology) - this comes at the expense of needing to handle all that logic yourself (to be fair, these webservers come with the necessary tools to easily configure them like the older conventional web-servers, it just isn't the default).
...but the advantage is that you can make it work exactly like you want.
In pseudocode their program probably looks something like this:
void handleRequest(Request request) {
String hostHeader = request.getHeader("Host")
RegexMatch nonCustomDomainMatch = hostHeader.match( "([^\.]+).tumblr.com" )
if nonCustomDomainMatch.success {
String accountName = nonCustomDomainMatch.groups[0]
showAccount( accountName )
}
else {
// Look up the custom domain name in a database or other mutable data store:
String accountName = db.execQuery( "SELECT accountName FROM accounts WHERE accounts.customDomainName = #cdn", new { cdn: hostHeader } )
if accountName == null {
showHttp404Error()
}
else {
showAccount( accountName )
}
}
}
In reality, given their size and scale, it would likely be some custom logic inside hardware load-balancers or some other lightweight frontend service - and always with aggressive caching (database lookups are expensive!).

How to use Restrict attribute in service stack

Is there any documentation on use of [Restrict] attribute with service stack?
Not finding any documentation, I started trying to figure this out. I discovered you have to enable restrictions in AppHost.cs Configure event with
var endpointHostConfig = new EndpointHostConfig
{
EnableAccessRestrictions = true,
};
Then I added attributes to my request DTO:
[Route("Hello/World", "GET")]
[Restrict(EndpointAttributes.InternalNetworkAccess)]
This does not work...looks like that removes all 'default' restrictions and replaces it with just that one restriction? Using this instead seems to work:
[Restrict(InternalOnly = true)]
When I do a GET from the local lan it works, but from remote it does not. Interesting, the 'detailed stack error' it gives from remote is:
The following restrictions were not met: '\n -[InternalNetworkAccess, Secure, HttpHead, HttpPost, HttpPut, HttpDelete,
HttpOther, OneWay, Soap11, Soap12, Xml, Jsv, ProtoBuf, Csv, Html, Yaml, MsgPack, FormatOther, AnyEndpoint]'
Note, it does not even list HttpGet as one of the possiblities - which does work. Also mentions Secure and not InSecure...neither of which I am specifically requiring.
Can we get some clarification on exactly how this is supposed to work? What if I wanted to require SSL - how would I specify that?
What if I wanted to require SSL in production, but not staging on all services for this endpoint? (Realizing this may be a completely different way to configure).
The [Restrict] attribute feature is in the latest version of ServiceStack. Currently the only documentation for this exists in the Security wiki page.
Here are some EndpointAttributes restrictions tests that test the validation of the restriction attributes, and some different service configurations you can use.
The way it works is that it's restricted to anything that's specified, so if you want to enable SSL and leave everything else as unrestricted, you would only add:
[Restrict(EndpointAttributes.Secure)]
public class SslOnly { }
It also supports specifying multiple combinations of environments that are allowed, e.g. You can enforce HTTP internally, but HTTPS externally with:
[Restrict(EndpointAttributes.Secure | EndpointAttributes.External,
EndpointAttributes.InSecure | EndpointAttributes.InternalNetworkAccess)]
public class SslExternalAndInsecureInternal { }
Note: each environment is combined with Enum flags and delimited with a ,.
But it doesn't let you distinguish between debug and release builds, to enable this you would need to use C# conditional compilation symbols.
E.g only allow HTTP for Debug builds and HTTPS for production Release builds:
#if DEBUG
[Restrict(EndpointAttributes.InSecure)]
#else
[Restrict(EndpointAttributes.Secure)]
#endif
public class MyRequestDto { ... }

grails 2.1.1 does not reload static resources

I have a fresh grails 2.1.1 installation, windows 7.
Problem: Static resources (CSS) don't reload in development mode.
According to http://grails.1312388.n4.nabble.com/Re-Grails-2-0-CSS-Changes-not-Picked-Up-td3680479.html I've tried in Config.groovy:
environments {
development {
grails.resources.processing.enabled = false;
}
}
Result: no CSS is loaded at all. Chrome needs 30 seconds to load a page, and does not load any CSS.
I reverted that change.
Then, according to http://grails.1312388.n4.nabble.com/disabling-static-resource-stuff-td3827384.html
I've tried to uncomment the plugins runtime ":resources:1.1.6"
Now I'm back to how it was initially, CSS files are served but not reloaded.
What works temporarily is the hack mentioned here:
http://grails.1312388.n4.nabble.com/The-resources-plugin-is-not-reloading-my-CSS-and-JS-of-a-bundle-td3857137.html
Top append to the url: ?_debugResources=y
How do I make static resources being reloaded in development mode? Shouldn't that be the default anyway? It doesn't make any sense in my opinion to reload gsp's (which it does) but not the static resources.
I set the grails.resources.debug flag in Config.groovy on debug to get the URL to be appended like this
css/main.css?_debugResources=y&n=1386063248351
environments {
development {
grails.resources.debug = true
}
production {
}
}
Any URL in your application can have the query parameter _debugResources=y added to it, and the request will perform no processing. So for example if you are browsing http://localhost:8080/myapp/admin and need to bypass resources, just change the URL in your browser to http://localhost:8080/myapp/admin?_debugResources=y
Reference Link : https://grails-plugins.github.io/grails-resources/guide/8.%20Debugging.html

Resources