Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
How to mark folder hidden for all search engines?
How many alternatives are there?
Which one is the most reliable?
To prevent search engines from visiting certain directories/urls it is common practice to use robots.txt. This is a file that search engines take a look at before spidering your website.
robots.txt
User-agent: *
Disallow: /secret/
This file should be placed in your website root. For example http://www.example.com/robots.txt
There are two important considerations when using /robots.txt:
robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I need to list all the website links (domains) in India under one website based on categories.
You just cannot list 100% of Indian websites. If I understand what you want, you want something like a Web crawler for Indian domains. Another "google". But even big spiders like Google's, Yahoo's, Bing's spiders can't build a database of all the websites. They do this with algorithms (with partly published algorithms), but I think you need even more, because you need a 100% accurate database, so you would need to have to ask them from all domain registrars, but you obviously can't do that.
Even if it was possible I definitely would not do this for a client, but created a company for that. But I would say that's practically impossible.
What you can do is that you can search for a few thousands of sites and categorize them manually or build a small "spider".
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am new in this kind of situation. I must implement a site in php that will have a URL like "mycompany.com". I would like to offer the possibility the users to create his own url inside my site. So user David could have 2 options: 1-Using sub-domains "david.mycompany.com" or 2- Using sub-folders "mycompany.com/david". I see people prefer to use sub-domain for this kind of situation. But what is wrong whit the second one? What is the best practice and why?
foo.bar.com is a subdomain
bar.com/foo is a directory under the domain.
If you are offering subdomain,
then there is absolutely nothing wrong with offering foo.bar.com
There can be permission issues inside your VM.
If you set the permission correctly and jail them and give them
correct privileges, there should be an issue.
for jailing, refer to :
http://how-to.linuxcareer.com/how-to-automatically-chroot-jail-selected-ssh-user-logins
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
I have 2 domains:
www.first.com
www.second.com
Lets assume that In the first one I have an online store, at the second one I have only products of this store (seperate applications that running on the server).
The products link is
www.second.com/firstProduct
www.second.com/secondProduct
www.second.com/thirdProduct
and etc...
I want to redirect users to the first website when someone hit www.second.com, ie not the full product path.
What redirect should I use? 301? In terms of SEO what is the best approach?
Thanks.
Yes, the 301 Moved Permanently, is the code you want to return for this redirect. Search engines typically will queue up 301's for updates to their results, as this indicates that the resource will now be found at the new url, and that the old one is soon to be obsolete.
In your case, since you never want www.second.com/ to be accessed directly, the 301 is exactly what you want.
You might also consider adding a robots.txt file with allow + disallow statements in there, as most of the bots you actually care about for SEO will honor it.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I am building a question and answering site by myself.
I want to make this site indexed as a Q&A site or Forums by Google, which can be retrieved when using the "Discussions" in Google. In my personal experience, Google Discussion Search is a pretty useful function when I want to get others' real opinions or experience.
However, I have no any idea on that how Google determine one site as Q&A/Forum or one page as Q&A/Forum page. I searched a lot on Google, but there is little related information discussing this issue. Do you have any idea or reference on that?
Thanks!
Best,
Jing
Use richsnippets and make Google recognizing your traffic by using Webmaster tools or Analytics . Use a sitemap.xml to invite for revisit and fast indexing, disable archiving (f.e. Google Cache) with meta-robots noarchive. If you have high traffic and fast content building, search-engines will then recognize by themselves.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Well, the title pretty much states the question...
SSL secures the communications, it does not provide content access mechanisms.
As long as there is no password/authentication restricting access to the pages, there's no reason a search engine would be unable to index them.
Yes.
They may choose not to spider over HTTPS, or they may choose to rate lower those sites that are available only over HTTPS, or they may choose to do any number of things. But they can certainly spider the Web over HTTPS just as easily as your browser can view a single Web page over HTTPS.