Google recently published an article on the robots exclusion protocol so read up webmasters!

Interesting points to the article:

The Controlling Caching and Snippets section - as Google states a “snippet” is just a piece of text found in the site itself that is used for the search result. This makes sense because we all know and love the little “snippets” that Google provides us when searching for anything, but how do you control them and keep them off of certain pages in your site? Per Google, use this code:

<META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”>

In case you are not familiar with creating a robots.txt file - it is just a simple text file that you should place in the root directory of your site labeled “robots.txt” - and you can use it to block out search engine spiders from going to certain places of your site.

Here’s an example of a robots.txt file that will allow all search engines:

User-agent: *
Disallow:

Here’s an example of a robots.txt file that will NOT allow ANY search engines:

User-agent: *
Disallow: /

Here’s an example of telling the search engines to not go to a specific folders in a site:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

I hope this is helpful for you. The “snippets” info was news to me.

Posted on February 27th, 2007 by Firelead Affiliate Network
Filed under: Google, Webmasters