I have a requirement to use wget command from within the hosting server to download a bunch of html files to a particular folder. I am placing the urls to download in a text file and using -i (input file) flag of wget. The urls are of the form https://.com/page1.php,https://.com/page2.php and so on.
In such case, I believe there will be dns resolution for each and every request. Is there a way to optimize the dns lookup?
You don't need anything else. Wget internally caches the DNS responses within a single run. So after the first request, it will not send any more DNS queries but will instead directly use the IP addresses found in the response.
In general, if you'd like to do this, you should install a DNS caching service like pdnsd on your server
Related
because of some background issues we have to change webhosting. unfortunately after the change I have found that we did not backup one file. We have no access to ftp server or any access to previous hosting. Only available way was via http and public access.
The domain is spedimex.cz, now running on new hosting with new dns etc. I thought that if I search for previous records and try to access the site via IP address I will be able to gain desired file.
History was gained from here
https://dnshistory.org/dns-records/spedimex.cz
but when I try to access A record 46.28.50.165 it says that page does not exits. Any tips how to access previous data?
thanks for tips!
Use hosts file. On linux is usually at /etc/hosts.
So you write on the last line
46.28.50.165 spedimex.cz
(if I understood correctly), then go get the files, then restore the hosts file.
I have a server with Plesk and CentOS.
http://cdn.oyunlar1.com/images/8836.jpg file is located in server A.
http://www.oyunlar1.com/modaokulu.html file is located in server B.
I basically want the same thing except I want to locate media files in a different domain instead of different server. Like this:
user visits cdn.mydomain.com/file.swf
but they actually read files from mydomain2.com/file.swf
I tried changing IP of A record of the subdomain to mydomain2.com's IP but it didn't work.
How can I do it?
Edit: I want to do this because it will help me to change my hosting provider very quickly and easily becuse media files are +100GB and the rest is like 10MB.
Solution was simply adding CNAME record to mydomain.com of mydomain2.com and using htaccess at mydomain2.
I need to know how to prevent repetitive file downloads using .htaccess, or if not via .htaccess than some other method.
A site I maintain had over 9,000 hits on a single PDF file, accounting for over 80% of the site's total bandwidth usage, and I believe that most of the hits were from the same IP address. I've banned the IP, but that's obviously not an effective solution because there are always proxies and besides, I can only do that after the fact.
So what I want to do is cap the number of times a single IP can attempt to download a designated file or file type over a given period of time. Can this be done with .htaccess? If not, what other options do I have?
EDIT: The suggestion that I redirect requests to a server-side script that would track requests by IP via database sounds like a good option. Can anyone recommend an existing script or library?
If you have some server side code stream out your files, you have the opportunity to control what's being sent. I'm not aware of a .htaccess solution (which doesn't mean there isn't one).
My background is in Microsoft products, so I'd write a bit of ASP.NET code that would accept the filename as a parameter & stream back the result. Since it would be my code, I could easily access a database to track which IP I was serving, how often the file was sent, etc.
This could easily be done using any server side tech - PHP, etc.
is there any way how to retrieve a list of domains hosted within a list of servers, f.e.
trying to find out how many other domains are hosted on the same server, and sharing common resources.
Preferable if this could be used under bash.
/Saulius
You could query this online service with wget or curl... but the form post's procedure looks a bit strange (look at the HTML source code before you use it).
Basically, this is looking-up an IP address in a database of domain names.
Is there any way in which I can download all the first visits to a webpage to my local box and all the subsequent visits will retrieve the data from the local box rather than the internet? That is, like a service is running on a port and if i access that port and not the HTTP port, i get the data from local box?
I need to use this service for parsing webpages whose contents might change every time, so that i get the same content to work with.
You can use a caching proxy such as squid.
The squid service stores the webpages locally and the next requests return the stored file.
Sounds like you're talking about a proxy server
I need to use this service for parsing webpages whose contents might change
Have a look for a spidering engine, e.g. pavuk.