Robot Spiders NOFOLLOW, NOINDEX Certain pages with Meta tag. Stop the publication of certain pages on search engines, etc.

"Y our suggestion of the message discouraging source code viewing ... will any of that text ever appear anywhere (search engine summaries, etc.?) Thank you!"
[ Kirk Jordan ] [ Block Source Code Viewing ]

T o answer your question in a word, "Yes". But...... there are other measures you can take to stop these pages from appearing in search engine summaries, etc.

B esides the blocking the viewing of the source code, (which is not foolproof), but basically just keeps everyone in the most part, honest, you must modify your "Meta" tag on the page(s) you do not want this information beening seen by search engines etc. If the page does not have a complete "Meta" tag, then you need to add one on those pages you do not want spidered. Below is a sample of two meta tags, one has the spiders indexing the page and the other does not have the spiders indexing the page(s) you want confidential. Notice what we changed to avoid the indexing (spidering) of the page(s):

Spiders Will Index

Spiders Will Not Index

T he above "Meta" tag, (or a basic facsimile thereof) once completely filled out will be inserted in the <HEAD> section of your document. See the below example of where and howto:

</head>
<BODY bgcolor="#000000" text="#ba55dc" link="#fff8dc" alink="#fff8dc" vlink="#fff8dc">

I f you are not too sure or need a little refresher on howto with a "Meta" tag, grab the below link and it will transport you to a page where you can generate your own "Meta" tag.
[ Advanced Meta Tag Generator ] [ Meta Tag Generator ]

B elow is an excerpt from [ Search Engine Optimization Techniques ]

".....B y default, all search engine spiders will use "INDEX, FOLLOW" even if there isn't a tag there. So usually, a search engine spider will try and index your page and follow the links on your page. If you want the spider to not index anything on your page, you can then use the following tag:

I n theory, this will stop robots from indexing your page (some search engine spiders have been known to use browser user agents to index pages), and also stop it from following the links on your page if any.

T hese methods are usually used if you don't want people off of the internet finding things on your page. Usually for security purposes. For example you wouldn't want the general public to be able to see the directory for all your customer information from a "search engine". This could be bad for your company, and maybe worse for your customers......"

W e hope that this tutorial was of help to you. If you need any more assistance, fell free to contact us again.

Answer Updated 10/29/05

Further Information

"......Thanks much for the help. I was mainly referring to the message received when one opens the "view source".

What I'm wondering is, would the above message/text (or excerpts of it) appear indexed in search engines? I really do want bots/spiders to index my site, in order to achieve higher ranking- so the "no index, no follow" tact is not an option.

I was merely considering adding that little "discouragement" message, provided it doesn't show up anywhere outside of the "view source code" window......"

T o answer this, we had to delve deeper into our resources and we came up with the following......

How does an indexing robot decide what to index?

I f an indexing robot knows about a document, it may decide to parse (Analyse syntactically by assigning a constituent structure to [a sentence]) it, and insert it into its database. How this is done depends on the robot: Some robots index the HTML Titles, or the first few paragraphs, or parse the entire HTML and index all words, with weightings depending on HTML constructs, etc. Some parse the META tag, or other special hidden tags.

How does a robot decide where to visit?

T his depends on the robot, each one uses different strategies. In general they start from a historical list of URLs, especially of documents with many links elsewhere, such as server lists, "What's New" pages, and the most popular sites on the Web.

M ost indexing services also allow you to submit URLs manually, which will then be queued and visited by the robot.

S ometimes other sources for URLs are used, such as scanners through USENET postings, published mailing list achives etc.

G iven those starting points a robot can select URLs to visit and index, and to parse and use as a source for new URLs.

W hat the above is saying in essence is that it all depends on the robot(s) that are indexing and spidering your site. It is possible that it would show up and then again it may not. Unless you know exactly what robots are doing this and what process they use, it is hard to say yay or nay.,/font>

I have added a link to a page that might clarify it even further.
[ The Web Robots Pages ]

W e hope our tutorial was easy to follow and we covered everything in detail. If you have any problems with this or anything else, feel free to consult our [ FAQ ] and if you can't find the answer there, [ contact us ].
[ Rate This Page ]

Use Of Meta Tag For Noindex, Nofollow

Further Information