If you enjoy WebSite Primer please recommend this site to your friends!
Meta elements That Can Stop A Speeding Search Engine Indexing Spider In Its Tracks?
Originally Published: 2001-10-06
Revised: 2006
Well, not really - but maybe slow them down a little. In this article I share two meta element markup created expressly for search engine robots.
In a previous article I shared various meta elements for use within your Web site. These are elements, within the head area, that some of the search engines gather information from; information that you provide!
Meta elements help the spiders collect information of what your page or Web site is about. That information, along with the page's URL, is then indexed.
Pages that you may want to exclude from being indexed could be incomplete pages or other areas in your site. So what if you have pages on your site that you don't want the search engines' robots indexing?
For those you can insert the following:
<meta name="robots" content="noindex, nofollow">
noindex informs the robots not to index the page. nofollow further instructs not to follow the links on that particular page.
<meta name="robots" content="noindex, follow">
As with the previous meta element, noindex informs to not index the page. follow however tells the robots to follow any links within the page's contents.
I should tell you now that the above markup shared is not 100% guaranteed to work. Please recall that Not all search engine spiders will refer to information shared in meta elements.
Due to past abuse of meta elements some search engines have programmed their spiders and robots to ignore that particular markup. Like it or not, they felt to have good reasons to take such a step. This is why many people recommend to create a robot.txt file for your site.
I will also inform you now that there is not a meta element to help command a search engine spider when to come back or where to go in your site. Although some sites will share a meta element where you can "inform'' a robot to return in a week or a month, it is a waste of your time to include it on your pages.
If your site is in the search engine's index or database - the spider will go by your site. Want the spiders to come around more often? Here's a couple of tips:
Routinely add new content to the site; new pages - new elements - editing of older content. It is noticed - by your readers and by search engine spiders.
Get other sites, preferablely ones relevant or similar to your site's topic/theme, linking to your site. Spiders follow links and this means links to inner pages as well as links to other sites.
The search engines send their spiders out routinely by their schedule, not yours or your meta element's suggestions of when that should occur. The most you can do is suggest where you do not wish them to gather information from. The best way to suggest what information to not index off your site is to use a ROBOTS.TXT where you list elements [e.g.. images] or pages for the spider to ignore.
If wishing to use the revisit meta element, be reasonable in how you decide to use it. If you update the site every other week, then why put in you want the spider to revisit every 5 days? Also use it primarily on pages that are frequently updated or changing in content. If your cat's page hasn't changed in content in 3 years - there is no purpose nor benefit, to your site, in trying to slip in a meta element for a spider to revisit every 7 days now is there?