I am working on a website at the moment which is multi-platform. Old tech is one CMS, new tech is headless CMS. In my Google Search Console I see that Google is crawling this URL:
https://www.xxxxxx/bank/dynatrace/rb_5a7f65a5-eb55-46f1-baff-f05ff70d4683?type=js3&sn=v_4_srv_10_sn_50BAF12DF4ABCC81ADC642A9746EFE11_perc_100000_ol_0_mul_1_app-3A53cd4acc1b85e003_0_rcs-3Acss_0&svrid=10&flavor=post&vi=QKMEEAHOKJHKRMIHUEDQVWLAQRGPRRPA-0&modifiedSince=1657720563908&rf=https%3A%2F%2Fwww.xxxxxxx.com%2Ffilialen%2Fde%2FViersen&bp=3&app=53cd4acc1b85e003&crc=2355823995&en=vsme09c4&end=1
Clicking the link produces this string, front end:
FL(OA,Percentage)|name=ruxitagentjs|featureHash=ICA27Vfgjoqrux|buildNumber=10241220422021336|lastModification=1657720563908|enabled=false
I do not want this to be crawled but the second URL is a vital SEO landing page.
My question is: Do I add this page to robots.txt and if I do what is the outcome - positive, negative or neutral?
Thanks
Related
I'm using Google Programmable Search Engine on my site and the actual url is something like /search. That page has the Google code embedded into it. When that page is accessed with the querystring params for the search, Google's javascript changes the URL to / with the search parameters appended.
If anyone tries to share that URL, they get the home page, not the search page. Any ideas on how to prevent Google from changing the URL path?
I don't want to migrate the whole site at same time, only some pages for now.
I have done:
install valid certificate
securized all resources into the html code of these pages.
add rel canonical tag
redirect 301 of some pages from http to https version through .htaccess
Google Chrome shows green secured status when access to these pages. Everything it's ok.
I haven't added new site property for https version in Google Search Console because if I do it Google will go to index my whole https site, and i don't want that to happen.
I made these changes 4 days ago and Google still is showing in search results the http version of my migrated pages.
I don't know if we have to add a new property in Google Search Console mandatory or if I just have to wait to google bot recognize the 301 redirect an reindexe pages.
Regards
Try to add links with https in sitemap.xml and add this sitemap in robots.txt and wait googlebot
SOLVED. After 5 days from changes Google is showing https version of my pages. Regards
i'm using robots.txt to exclude some pages from spiders.
User-agent: *
Disallow: /track.php
When i search something refeered to this page, google says: "A description for this result is not available because of this site's robots.txt – learn more."
It means that the robots.txt is working.. but why the link to the page is still found by the spider? I'd like to have no link to the 'track.php' page... how i should setup the robots.txt? (or something like .htaccess and so on..?)
Here's what happened:
Googlebot saw, on some other page, a link to track.php. Let's call that page "source.html".
Googlebot tried to visit your track.php file.
Your robots.txt told Googlebot not to read the file.
So Google knows that source.html links to track.php, but it doesn't know what track.php contains. You didn't tell Google not to index track.php; you told Googlebot not to read and index the data inside track.php.
As Google's documentation says:
While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.
There's not a lot you can do about this. For your own pages, you can use the x-robots-tag or noindex meta tag as described in that documentation. That will prevent Googlebot from indexing the URL if it finds a link in your pages. But if some page that you don't control links to that track.php file, then Google is quite likely to index it.
Is it possible in any way to treat
link1
as
prodcuct/mobile/android/xy in address bar. I mean when click on the 'link1' will show the SEO friendly url in address bar.
Thanks in advance
It's impossible because your product create the extension from a PHP database and if list it as "prodcuct/mobile/android/xy" Google Bot find crawl error page does not found in your google webmaster tool.
So my suggestion is write php code that when ever new product page is created using your backend; make product name to create automatically as page name. Then your site can links can be index really fast.
In my website if a browser has not javascript enabled then I redirect it to a javascripterror.html page.When i search my website on yahoo search then it always shows that javascript error page in its search result.I think that's hapenning because yahoo slurp has its javascript disabled,Is there any way by which Yahoo search result can show my index page in its search results.