OK, I have a website which has a series of buttons which call an api to return informtion relating to products. The problem i have is that google is accessing and following the api every 10 minutes approximatley which means i am doing loads of lookups a day.
Can i stop google bots from doing this??
I have a robots.txt file and no follow meta but they do not seem to be doing much.
Any advice would be appreciated.
User-Agent: *
Disallow:/index/
Disallow /manufacturer/
I just noticed the pages didn't have extensions i.e manufacturer should be manufacturer.php
Thanks
Google Webmaster tools will analyze your robots.txt for you and let you know what it is making out of it. I have found it very useful in the past. You can just paste your robots.txt and it will point out any flaws.
Failing that you can visit the forums.
Related
I have created a new website www.bucketshowers.com and I tried to index it using google webmaster tools. Fetch as Google for the desktop worked just fine, but doing the same for mobile shows an error "Temporarily unreachbale". It's been a few days and the website REALLY is not avaible on mobile. It's driving me nuts. Here're is some information and things I have already tried:
Website is made with WP
I have disabled all SEO/meta tags plugins and I added a very basic robots.txt http://bucketshowers.com/robots.txt
I tried waiting 15min between fetching the root page on mobile
I have checked source code for the homepage to make sure there are no meta tags with nofollow or noindex attributes
I baffled by this issue and I would gladly take any advise/pointers what else can be done. Thank you.
The crazy thing was, that it was caused by WP Statistics plugin, which is probably the most popular from its kind - 500k downloads. When I deactivated it, everything is fine, google fetches of the mobile and the website is available. Incredible! I'm still searching for the actual problem within that plugin.
I'd like to know why google do not index my posts on my blog writes in NodeJS.
Link of a post : http://icecom.fr/articles-icecom/9
Anthony
There are several reasons why Google isn't indexing your website.
There are no links to your website. Google follows links on the internet to other pages. If there are no links to your website it won't find it.
You are denying access to Google through the robots meta-tag or robots.txt.
You haven't waited long enough yet, Google may take some time before it has indexed your website.
Of course you can supply Google with the proper URL's with a [sitemap]{https://support.google.com/webmasters/answer/156184?hl=en}. A good place to create this if you're new to it could be [here]{http://www.xml-sitemaps.com/}
#szenbalu already mentioned you can upload this sitemap.xml to Google Webmaster Tools and this way Google can index your site without the need of links. It is also faster most of the time.
Another way to get your website indexed through Google Webmaster Tools is the 'Fetch as Google' tool. In here you can tell Google to fetch and index your website. This is especially useful if you change content and want it reindexed.
About your specific case:
* You do not block Google with the meta robots tag
* I can not find a robots.txt file
* I can not find any links to your articles from [OpenSiteExplorer]{http://www.opensiteexplorer.org/}
I think that uploading a sitemap to Google Websmaster Tools + Using the Fetch as Google tool will get your site indexed within no time.
If you have any questions left, feel free to ask. :)
Do you have the robots.txt file and webmaster tools account joined to your page?
With webmaster tools you can upload sitemap that google will use to index pages.
A client would like to have his website implemented like the following one, on google:
https://www.google.ch/search?q=mlzd&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:fr:official&client=firefox-a
With:
a list of links to the differents sections of the website.
the google function with the arrow (on rollover of the links) to get previews of the website pages.
So, here are my questions:
Do you know to implement the several links? Is it tags?
Do I then have to do something to get the google images previews, or is it automatic?
Will the google images previews work, with a Flash website? If not, the preview will be a screenshot the website with no flash enabled?
Thank you a lot for your answers!!
David
The links are called Sitelinks, they are automatic. You cant do anything to sepefically enable them.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=47334
The preview is automatic, but you do need to make sure that 1) allow google to cache the url (ie avoid "nocache" and 2) not block in anyway the Previews https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews
See link in the previous answer
My question is about Search Engine Result pages, if your site is the first search engine result, many site's search results show the page listing like this as is this screenshot.
So, is there a procedure to follow so I can achieve the same effect for my site.
Unfortunately not, those are automatically generated by Google. You can read more details on the Google Webmaster page about Sitelinks.
Those are completely generated automatically by GOOGLE
Google Doesn't Say How to Get Sitelinks
The workings of many Google algorithms, including Sitelinks, are kept secret to discourage people from manipulating the rankings, but we can still look at examples and try to understand where Sitelinks come from. I've worked on a number of sites with Sitelinks, and these sites are similar in the following ways:
* Site ranks first for the keyword(s) that generate the Sitelinks listing
* Easily spiderable, structured navigation
* Fairly high natural search traffic
* High click through rates from the search results page
* Useful outbound links
* Inbound links from high quality sites
* Site age is several years or older
Sitelinks generally appear when a site is considered to be the main authority on a particular keyword, in short an old site with lots of links with the keyword anchor text is likely to get sitelinks.
I've found that, as stated above, having clear navigation helps. Also, it appears that the Google bot has particular support for media wiki, automatically pulling the TOC out into the results.
You can get more information about your own site links by logging into google webmaster tools.
While building a Webiste, i created a robot.txt on the server, to prevent the yet unfinished application from getting into Google's Index...
Now that i am done with the site, i removed the robot.txt and i expected that my site would show up on Google, since the robot.txt is gone!
But this is not happening! I have removed the robot.txt now for about 3 to 4 weeks, and yet the site is still not showing up :(
Is there something that one needs to do after the removal of robot.txt to get into the indexes of search engines again? Or isn't this suppose to happen naturally?
Or is this my case that of not being patient enough?
You can add your site for crawling in here.
Create a sitemap file and submit it to google, bing, and others. For google you can use their webmaster tools for this.
I would just setup a new default robots.txt file:
User-agent: *
Disallow:
Also sign up for Google Webmaster Tools and setup sitemap files. This might help Google to recognize things have changed.
As the first step step get site verified in Webmaster Tools. So you can see google crawler visits and reasons for denies.
Read more # http://www.google.com/support/forum/p/Webmasters/thread?tid=671635798b0e75ba&hl=en
For an optimal position in the Google search results you should definitely check this document:
Search Engine Optimization Starter Guide