I am using Apache Nutch for crawling websites. Nutch is not crawling links for external websites.
I have gone through this link How do you crawl external links on a found page? but it did not produce intended result.
Related
I'm trying to understand how web crawling works. There are 3 questions:
Do we have to have an initial directory of URLs to build a larger
directory of URLs? How does this work?
Are there any open source
web crawlers written in python?
Where is the best place to learn more about web
crawlers?
Answering your second question first; Scrapy is a great tool to do web scraping in python.
When using it there are a number of ways to start the spiders. The CrawlSpider can be given a list of initial URLs to start from. It then scrapes these pages looking for new links which are added to the queue of pages to search.
Another way to use it is with the sitemap spider. For this spider you give the crawler a list of the URLs of websites sitemaps. The spider then looks up the list of pages from the sitemap and crawls those.
I want to crawl this site: https://511.org/alerts/traffic/incidents using Apache Nutch. The webpage has dynamically loaded ajax content. If I crawl it with the default configurations, Nutch just brings headers and footers and dynamically loaded content is lost. I am using Nutch 1.14.
With Nutch 1.14, you can use either the Nutch Selenium or the Nutch Interactive Selenium plugins to crawl pages with dynamically loaded elements.
I am trying to understand why my website does not get visited my google bots.
http://www.nateiss.com/
I used Site-Analyser to analyse my site - you can see the website report.
http://www.site-analyzer.com/en/audit/http://www.nateiss.com#report-page-6
What Should I do to make majors bots: google, yahoo, bing to visit it?
Thanks
Same problem I was facing before 1 year ago. Because my website http://www.silkyquote.com/ was not listed in google. But now googlebot daily visiting my website.
If your website or blog is new than googlebot takes time to crawl your website.
If your website is not update regularly than googlebot not visit your site regularly.
So, update your website daily. In shot time googlebot will crawling your site.
I want to add a robot.txt so my web page can be found...
So I have heard that putting a robot.txt with meta tags in the root of my site can do this.
Is this true?, if so,
What would be the steps to add or generate this robot.txt?
I have found this
Robots.txt is more for telling the crawlers where to and where to not go once they've already reached your site.
A better way to get crawlers onto your site is to build a sitemap for your site, then use Google Webmaster to submit this sitemap to google. You'll also want to include the sitemap on your site's root url and tell google where it is (all of this can be done in Google's Webmaster Tool linked to above).
No, it won't make your webpage suddenly visible. It just instructs web crawlers on how to index your site.
http://www.robotstxt.org/
I have a website that contains links to other sites only.
They link to an image gallery or a video.
Does google accepts this or will it penalize my site because I don't have any real content?
Thanks
Google does look for human generated content. They also look for the links to your site. I would add stuff to your site and also guest blog etc to get traffic to your site.