While building a Webiste, i created a robot.txt on the server, to prevent the yet unfinished application from getting into Google's Index...
Now that i am done with the site, i removed the robot.txt and i expected that my site would show up on Google, since the robot.txt is gone!
But this is not happening! I have removed the robot.txt now for about 3 to 4 weeks, and yet the site is still not showing up :(
Is there something that one needs to do after the removal of robot.txt to get into the indexes of search engines again? Or isn't this suppose to happen naturally?
Or is this my case that of not being patient enough?
You can add your site for crawling in here.
Create a sitemap file and submit it to google, bing, and others. For google you can use their webmaster tools for this.
I would just setup a new default robots.txt file:
User-agent: *
Disallow:
Also sign up for Google Webmaster Tools and setup sitemap files. This might help Google to recognize things have changed.
As the first step step get site verified in Webmaster Tools. So you can see google crawler visits and reasons for denies.
Read more # http://www.google.com/support/forum/p/Webmasters/thread?tid=671635798b0e75ba&hl=en
For an optimal position in the Google search results you should definitely check this document:
Search Engine Optimization Starter Guide
Related
I'd like to know why google do not index my posts on my blog writes in NodeJS.
Link of a post : http://icecom.fr/articles-icecom/9
Anthony
There are several reasons why Google isn't indexing your website.
There are no links to your website. Google follows links on the internet to other pages. If there are no links to your website it won't find it.
You are denying access to Google through the robots meta-tag or robots.txt.
You haven't waited long enough yet, Google may take some time before it has indexed your website.
Of course you can supply Google with the proper URL's with a [sitemap]{https://support.google.com/webmasters/answer/156184?hl=en}. A good place to create this if you're new to it could be [here]{http://www.xml-sitemaps.com/}
#szenbalu already mentioned you can upload this sitemap.xml to Google Webmaster Tools and this way Google can index your site without the need of links. It is also faster most of the time.
Another way to get your website indexed through Google Webmaster Tools is the 'Fetch as Google' tool. In here you can tell Google to fetch and index your website. This is especially useful if you change content and want it reindexed.
About your specific case:
* You do not block Google with the meta robots tag
* I can not find a robots.txt file
* I can not find any links to your articles from [OpenSiteExplorer]{http://www.opensiteexplorer.org/}
I think that uploading a sitemap to Google Websmaster Tools + Using the Fetch as Google tool will get your site indexed within no time.
If you have any questions left, feel free to ask. :)
Do you have the robots.txt file and webmaster tools account joined to your page?
With webmaster tools you can upload sitemap that google will use to index pages.
I would password-protect/hide the website using htaccess to disallow robots. Then If I bring it live after some weeks, will the indexing work as usual or should be notified using Google webmaster tools?
Thanks :)
robots crawl your page more than once so I think it will be reindexed after some weeks
OK, I have a website which has a series of buttons which call an api to return informtion relating to products. The problem i have is that google is accessing and following the api every 10 minutes approximatley which means i am doing loads of lookups a day.
Can i stop google bots from doing this??
I have a robots.txt file and no follow meta but they do not seem to be doing much.
Any advice would be appreciated.
User-Agent: *
Disallow:/index/
Disallow /manufacturer/
I just noticed the pages didn't have extensions i.e manufacturer should be manufacturer.php
Thanks
Google Webmaster tools will analyze your robots.txt for you and let you know what it is making out of it. I have found it very useful in the past. You can just paste your robots.txt and it will point out any flaws.
Failing that you can visit the forums.
There is a duplicate development website that exists for legacy reasons and is pending a complete removal, it always had a rule in it's robots.txt file to deny all search engines, but at one point the robots.txt got deleted by accident, and for a point in time there were two cross-domain duplicates and Google indexed the entire duplicate website, and caused thousands of incoming links to the production website to show up in Google webmaster tools (Your site on the web > Links to your site).
The robots.txt got restored, and the entire development site is protected by a password, but the incoming links from the duplicate site remain in the production website webmaster tools, even though the development site robots.txt was downloaded by Google 19 hours ago.
I have spent hours reading about this, and see a lot of contradiction on the web, so would like to get an updated consensus from stackoverflow on how to perform a complete site removal and remove the links that point from the development site to the production site from Google.
Nobody will be able to tell you exactly how much time will it take for Google to remove the "bad" links from index, but it's likely going to take a few days not hours. Another thing to keep in mind is that only "good" crawlers will be actually honoring your robots.txt file, so if you don't want these links to show up elsewhere, just using disallow in your robots.txt file certainly won't be enough.
I want to add a robot.txt so my web page can be found...
So I have heard that putting a robot.txt with meta tags in the root of my site can do this.
Is this true?, if so,
What would be the steps to add or generate this robot.txt?
I have found this
Robots.txt is more for telling the crawlers where to and where to not go once they've already reached your site.
A better way to get crawlers onto your site is to build a sitemap for your site, then use Google Webmaster to submit this sitemap to google. You'll also want to include the sitemap on your site's root url and tell google where it is (all of this can be done in Google's Webmaster Tool linked to above).
No, it won't make your webpage suddenly visible. It just instructs web crawlers on how to index your site.
http://www.robotstxt.org/