I am trying to download the contents of a website, but since this website is in more than one language, I faced a problem. the structure of the site is as follows:
Mysite.com
fr.Mysite.com
ar.Mysite.com
...
I just want to download files in the Arabic domain (ar.Mysite.com), but using -r option it also downloads the contents of other languages. How can I say wget to recursively download the files, but just the files within a specific domain?
The --no-parent option does what you want, if I understand the question. It limits queries to those that lie underneath the "directory" specified by the original URL.
Related
I have no experience using linux to make a script, and I need to make a script to download files (multiple files on the same page) whose download links are dynamic. What path should I take or what should I know to be able to do it?
If you are having page with links to resources you want to download I suggest trying Recursive Retrieval Options of GNU Wget, to limit download only to resource to which links are present in site you might set level to 1, i.e. do something like
wget -r -l 1 http://www.example.com
I need to download a single folder from a github repository in a Python 3 script.
Listing all raw file URLs to download is tedious and eventual new files would need to be added manually
Downloading the whole repository as zip takes rather long; there are lots of unneeded files. [This how I do at the moment]
I have read about web services that do what I would need, such as downgit.github.io, but the problem is that generating the relevant URL and fetching it via urllib.request.urlretrive() downloads the website rather than the actual file.
What can I do? Is there a web service that provides raw file links that I can download as described above?
I just found a solution: The ffspec library may be used to download a single folder from a GitHub repository using Python. See https://sebastianwallkoetter.wordpress.com/2022/01/30/copy-github-folders-using-python/
So, I decided to try to break my website...I googled my site by typing in site:mysite.com/whatever and behold, all of the users uploaded files were available for view under a specific directory.
What kind of script/ counter measure should I use to block these files from being viewed? I already have a script that checks the path and the logged in status, however this doesn't seem to be working. I've looked all over for solutions...but I can't quite find one. I'm using ColdFusion 8.
This isn't a ColdFusion issue so much as a web server configuration issue.
You should either:
configure your web server not to show a directory of files when using a URL without a filename (e.g., http://www.example.com/files/)
drop a blank default web document (index.html, index.htm, default.htm, index.cfm, whatever) into that directory so that it displays that document rather than the list of files. If you use index.cfm, it'll fire your Application.cfm/cfc in your file path and use whatever other security you've built.
(or, better, do both)
The best way to secure your file listings and the files themselves is to store them in another folder outside of the Web site root folder. You can then serve them up using CFDIRECTORY and CFCONTENT. The pages that display the files can check your access controls and only serve the files to those allowed to see them.
I have created a new website for a client, (joomla) which is currently running within a folder in the root called /new/ (original I know)
I am now trying to work out what would be the best way to switch over the sites, there are a lot of files (pdf's, doc's etc) so moving everything would take some time, plus the old site is very messy, and would also take some time to remove in one go.
What about htaccess, and having a redirect? I could then remove the old site and files from the root over time?
One approach could be the use of symbolic links(ln -s basically), but you won't be able to do that for files that have the same name in both root and your new/ subdirectory. You will encounter this same issue with apache redirects, thus basically, you need to decide before hand what do you want to serve from either root or new/ and resolve the possible conflicts.
I need to archive complete pages including any linked images etc. on my linux server. Looking for the best solution. Is there a way to save all assets and then relink them all to work in the same directory?
I've thought about using curl, but I'm unsure of how to do all of this. Also, will I maybe need PHP-DOM?
Is there a way to use firefox on the server and copy the temp files after the address has been loaded or similar?
Any and all input welcome.
Edit:
It seems as though wget is 'not' going to work as the files need to be rendered. I have firefox installed on the server, is there a way to load the url in firefox and then grab the temp files and clear the temp files after?
wget can do that, for example:
wget -r http://example.com/
This will mirror the whole example.com site.
Some interesting options are:
-Dexample.com: do not follow links of other domains
--html-extension: renames pages with text/html content-type to .html
Manual: http://www.gnu.org/software/wget/manual/
Use following command:
wget -E -k -p http://yoursite.com
Use -E to adjust extensions. Use -k to convert links to load the page from your storage. Use -p to download all objects inside the page.
Please note that this command does not download other pages hyperlinked in the specified page. It means that this command only download objects required to load the specified page properly.
If all the content in the web page was static, you could get around this issue with something like wget:
$ wget -r -l 10 -p http://my.web.page.com/
or some variation thereof.
Since you also have dynamic pages, you cannot in general archive such a web page using wget or any simple HTTP client. A proper archive needs to incorporate the contents of the backend database and any server-side scripts. That means that the only way to do this properly is to copy the backing server-side files. That includes at least the HTTP server document root and any database files.
EDIT:
As a work-around, you could modify your webpage so that a suitably priviledged user could download all the server-side files, as well as a text-mode dump of the backing database (e.g. an SQL dump). You should take extreme care to avoid opening any security holes through this archiving system.
If you are using a virtual hosting provider, most of them provide some kind of Web interface that allows backing-up the whole site. If you use an actual server, there is a large number of back-up solutions that you could install, including a few Web-based ones for hosted sites.
What's the best way to save a complete webpage on a linux server?
I tried couple of tools curl, wget included but nothing works up to my expectations.
Finally I found a tool to save a complete webpage (images, scripts, linked pages.... everything included). Its written in rust named monolith. Take a look.
It do not save images and other scripts/ stylesheets as separate files but pack them in 1 html file.
For example
If I had to save https://nodejs.org/en/docs/es6 to a es6.html with all page requisites packed in one file then I had to run:
monolith https://nodejs.org/en/docs/es6 -o es6.html
wget -r http://yoursite.com
Should be sufficient and grab images/media. There are plenty of options you can feed it.
Note: I believe wget nor any other program supports downloading images specified through CSS - so you may need to do that yourself manually.
Here may be some useful arguments: http://www.linuxjournal.com/content/downloading-entire-web-site-wget