I followed the tutorial here https://varnish-cache.org/docs/5.1/users-guide/vcl-separate.html
and I need the script
vcl.load il_1 /etc/varnish/il.vcl
vcl.load co_1 /etc/varnish/cor.vcl
vcl.label l_il il_1
vcl.label l_cor cor_1
vcl.load top_1 /etc/varnish/top.vcl
vcl.use top_1
to load at startup, I replaced the default.vcl in varnish param. with top.vcl but did not work
The approach with VCL labels to load website specific configuration is not so elegant. It requires you to use a multitude of if statements in your main VCL file.
Thus, you might better look at configuring Varnish virtual hosts. That seems to be a more clean and elegant way to define website specific VCL code.
Related
I am using two application load balancers that are routing requests to 4 backend varnish servers. I got answers to configure the PHP file to purge the cache but I have no idea where to put it and how to execute.
For which type of PHP application are you trying to configure cache purging?
A custom application?
WordPress?
Drupal?
Magento?
Some other CMS or framework?
If you're using an existing platform, CMS, or framework, the documentation will probably state how to configure purging.
Varnish Configuration
Of course, the Varnish VCL code should also be tuned to process purges.
You can find more information about purging (and banning) in Varnish on http://varnish-cache.org/docs/6.0/users-guide/purging.html
Here are the questions you should ask yourself regarding purging. Maybe the documentation of your CMS or framework will answer these as well?
Are you trying to purge individual URLs?
Does your code have pattern matching in play to invalidate multiple URLs at once (uses bans in VCL)
If pattern matching is used, are you sending the invalidation pattern via an HTTP request header?
Does your invalidation code use the URL to identify objects in cache, or does it rely on tagged content?
Are you restricting access to the purging mechanism based on IP address or subnet. If so, please configure an ACL in VCL.
Many WordPress plugins rely on individual URL purging. Other WordPress plugins use bans through request header patterns.
Drupal uses bans, but has a system in place that tags content. The ban patterns don't match URLs, but tags.
Magento uses bans.
Conclusion
If you use a CMS or framework, the purging strategy is set in advance. It's just a matter of configuring your app and making sure the VCL can handle it.
If you have custom code, you have a choice, and you can implement purging or banning.
Please have a look at the user guide section about purging I mentioned above. It should help you understand the underlying mechanism.
I want to follow Google's directive in terms of cache headers for images, scripts and styles.
After reading symfony's documentation about http cache, I decided to install FOSHttpCacheBundle. The I set up rules for path like ^/Resources/ or ^/css/. I then fail to see it the proper headers for my images using Chrome's console.
Alternatively, I have read that, since my server is handling the resource, this is not Symfony that deals with this matter (yet I read in the doc that Symfony Proxy was good for shared-hosting servers, which is what I have).
So should I just add lines to my .htaccess as explained in here, or am I simply misusing FOSHttpCacheBundle? (Or both.)
Static files (including javascript files, CSS stylesheets, images, fonts...) are served directly by the web server. As the PHP module is not even loaded for such files, you must configure the server to set proper HTTP headers. You can do it using a .htaccess file if you use Apache but doing it directly in httpd.conf/apache2.conf/vhost conf (depending of your configuration) will be better from a performance point of view.
If you also want want to set HTTP cache headers for dynamic content (HTML generated by Symfony...), then you must use FosHttpCache or any other method provided by Symfonny such as the #Cache annotation.
Is there any way for varnish to read a list of backend urls from a text file, and then proxy cache misses to a random url taken from the text file?
What I imagine is something like this pseudocode...
/var/services/backend-urls.conf
http://backend-host-1/path/to/application
http://backend-host-2/path/to/application
http://backend-host-3/path/to/application
# etc
varnish config
sub vcl_miss {
// read a list of urls from a text file
backendHosts = readFile("/var/services/backend-urls.conf");
//choose a random url from the file
randomHost = chooseLineAtRandom(backendHosts);
//proxy the request to the random host
set req.backend = randomHost;
}
To provide some background, I work on a server system that comprises a number of backend applications that currently sit behind a front-end running apache. We are evaluating replacing the apache layer with varnish so we can benefit from the caching capabilities of varnish. We also have a service discovery framework that knows the endpoint locations for each backend application (the endpoint urls change periodically as new hosts emerge or are taken out of service).
Currently we use the RewriteMap functionality in mod_rewrite to route requests to the backend services. Then we have a process to maintain the lists of backend services based upon the contents of the service discovery framework.
All this works well for us in apache, except that apache is like using a sledgehammer to crack a nut. All we really want is the reverse proxy loigc, and the caching in varnish would be helpful too.
Is there any way to have varnish read the list of backend urls from an external resource?
Without resorting to custom vmod/c modules, the quick answer is no.
The VCL instruction are being compiled within varnish, and that rules out run-time inclusions.
But why not include within the VCL a separate backend vcl which includes the current backends.
that vcl file could be written out on demand. Then using varnishadm CLI command you could request a new compile of the VCL, therefore bringing the config live.
I can see two potential solutions.
The first is to have something generate your VCL and backends such as Chef or some custom scripting. You can then process the text file into backend definitions and the necessary VCL to invoke them. To handle the requirement for the random backend you could use a director. I've not dealt with directors myself but it looks like they are meant to solve that requirement. When changes to the backends occur you could rerun the generation script/Chef and tell Varnish to reload its configuration either using varnishadm or service varnish reload to avoid a full restart.
The second would be to implement it in C, either via a VMOD as Marcel Dumont suggests or possibly using inline C in your VCL.
With vmod_dynamic you can just use any DNS name as a backend or even service records.
For your use case, one option would be to set up an SRV record in DNS pointing to all your servers and then just use that as for example in the basic-stub.vtc test case.
I would like to add the Varnish-Cache version/signature to my incoming HTTP requests so I can log the Varnish version with requests on my webserver. I understand this information is available in obj.http.Server, but this doesn't work inside vcl_recv or vcl_miss:
set req.http.X-VARNISH-VERSION = obj.http.Server;
Apparently those vcl subs only have access to req and not obj. Is there any other way to get the version number into an HTTP request header?
I am using Varnish 3.0.2.
[Edit]
I am using a Varnish module as an integral component in my system and as part of my automated testing I am running functional tests through the load balancer. I want my web servers (hhvm i this case) to know what version of Varnish is proxying the requests. Currently I am using a hardcoded string for this purpose, but I would like to automate this so I can distribute a non-hardcoded configuration to my varnish servers.
Varnish only sets the Server header when performing a synthetic response (like in vcl_error) and the header in that case doesn't contains Varnish' version.
Please extend your question, I can't envision what you want to achieve with that (and why a fixed string header substitution won't feet your needs).
I'm currently learning node.js and loving it. I noticing, however, that it seems that's it's really only fit for one site. So it's great for hosting mydomain.com, but what if I want to build an actual full web server with it. In other words, I would like to host mydomain.com, example.com, yourdomain.com and so on. What solutions (modules) are available for this? I was thinking of simply parsing the url from the request object and simply reading from the appropriate directory. For example if I get a request for example.com then read from the example_com directory or if I get a request from mydomain.com read from the mydomain_com directory. The issue here is I don't know how this will affect performance and scalability.
I've looked into Multi-node but I don't fully follow the idea of processes yet (I'm a node beginner).
Any suggestions are welcome.
You can do this a few different ways. One way is to write it directly into your web application by checking what domain the request was made to and then route within your application but unless your application is very basic this can make it fairly bloated and can get messy. A good time to do something like this might be if you're writing a blogging platform where everything is pretty much the same across all your domains. The key difference might be how you query your data to display the right data.
In this case you'd probably use the request to see which blog is being accessed.
If you want to just host a few different domains on the same server all using port 80 (like most websites do) you will want to proxy each request off to a different process. You can do this with nginx or even with node itself. It all comes down to what best fits your needs. bouncy is a quick way to get setup doing this as its a nodejs module and has some pretty impressive benchmarks. nginx (proxy with nginx) is probably the most wildly used method though, as a lot of nodejs servers use nginx to serve static content anyways.
http://blog.noort.be/2011/03/07/node-js-on-nginx.html
https://github.com/substack/bouncy/
You can use connect's vhost middleware (which is also available in express) to dispatch requests to separate request handlers based on the Host: header. This assumes that everything is being handled by the same node process on the same port; if you really need separate processes, then the suggestion about using nginx as a reverse proxy is probably the way to go.