I want to add a robot.txt so my web page can be found...
So I have heard that putting a robot.txt with meta tags in the root of my site can do this.
Is this true?, if so,
What would be the steps to add or generate this robot.txt?
I have found this
Robots.txt is more for telling the crawlers where to and where to not go once they've already reached your site.
A better way to get crawlers onto your site is to build a sitemap for your site, then use Google Webmaster to submit this sitemap to google. You'll also want to include the sitemap on your site's root url and tell google where it is (all of this can be done in Google's Webmaster Tool linked to above).
No, it won't make your webpage suddenly visible. It just instructs web crawlers on how to index your site.
http://www.robotstxt.org/
Related
I made an update of my non-responsive to responsive template and there is no need of my old fashion static mobile website and redirection to it (www.mysite.com/mobile/index.html). I want to completely remove directory with mobile site so that my old mobile site is not available anymore.
I'm concern with numerous 404 errors afterwords and their effect on my current Google search appearance. Maybe somebody could help me with advice what would be the best practice in this case.
I'm using CSM Joomla, Apache server and I have configured .htaccess file.
I would like the most to 301 redirect whole mobile directory to my home website link (www.mysite.com) but I'm aware that would be really bad from the Google's point of view. Any help would be greatly appreciated.
Redirecting the whole site to the home page would be seen as a soft 404 by Google. Either redirect each page to the new equivalent, or return a 404/410 response.
I'd like to know why google do not index my posts on my blog writes in NodeJS.
Link of a post : http://icecom.fr/articles-icecom/9
Anthony
There are several reasons why Google isn't indexing your website.
There are no links to your website. Google follows links on the internet to other pages. If there are no links to your website it won't find it.
You are denying access to Google through the robots meta-tag or robots.txt.
You haven't waited long enough yet, Google may take some time before it has indexed your website.
Of course you can supply Google with the proper URL's with a [sitemap]{https://support.google.com/webmasters/answer/156184?hl=en}. A good place to create this if you're new to it could be [here]{http://www.xml-sitemaps.com/}
#szenbalu already mentioned you can upload this sitemap.xml to Google Webmaster Tools and this way Google can index your site without the need of links. It is also faster most of the time.
Another way to get your website indexed through Google Webmaster Tools is the 'Fetch as Google' tool. In here you can tell Google to fetch and index your website. This is especially useful if you change content and want it reindexed.
About your specific case:
* You do not block Google with the meta robots tag
* I can not find a robots.txt file
* I can not find any links to your articles from [OpenSiteExplorer]{http://www.opensiteexplorer.org/}
I think that uploading a sitemap to Google Websmaster Tools + Using the Fetch as Google tool will get your site indexed within no time.
If you have any questions left, feel free to ask. :)
Do you have the robots.txt file and webmaster tools account joined to your page?
With webmaster tools you can upload sitemap that google will use to index pages.
I want to submit a Google Webmaster Tools sitemap for a Blogger "blog" that only consists of static pages (using it as a website instead of a blog).
Since there's no way for me to host a file on the Blogger server, and therefore no way to submit a sitemap directly through Google Webmaster Tools, is it possible to host a sitemap.txt file on another server and reference it in robots.txt? Or would I run into some sort of cross-domain issue?
If you can managed to map robots.txt file to your root directory, Which is www.domain.com/robots.txt thats okay, but i don't think you have to do that. Just add your site to Google Webmasters Tools and use regular seo strategy. Google is smart enough to Index your static pages.
--Hope this helps!!
I have a website that contains links to other sites only.
They link to an image gallery or a video.
Does google accepts this or will it penalize my site because I don't have any real content?
Thanks
Google does look for human generated content. They also look for the links to your site. I would add stuff to your site and also guest blog etc to get traffic to your site.
While building a Webiste, i created a robot.txt on the server, to prevent the yet unfinished application from getting into Google's Index...
Now that i am done with the site, i removed the robot.txt and i expected that my site would show up on Google, since the robot.txt is gone!
But this is not happening! I have removed the robot.txt now for about 3 to 4 weeks, and yet the site is still not showing up :(
Is there something that one needs to do after the removal of robot.txt to get into the indexes of search engines again? Or isn't this suppose to happen naturally?
Or is this my case that of not being patient enough?
You can add your site for crawling in here.
Create a sitemap file and submit it to google, bing, and others. For google you can use their webmaster tools for this.
I would just setup a new default robots.txt file:
User-agent: *
Disallow:
Also sign up for Google Webmaster Tools and setup sitemap files. This might help Google to recognize things have changed.
As the first step step get site verified in Webmaster Tools. So you can see google crawler visits and reasons for denies.
Read more # http://www.google.com/support/forum/p/Webmasters/thread?tid=671635798b0e75ba&hl=en
For an optimal position in the Google search results you should definitely check this document:
Search Engine Optimization Starter Guide