Peenie Wallie

« 3D Rendering Software | Main | Dandelion Farmers »

April 15, 2006

Spiders and the Robots.txt File

I recently made several changes to my robots.txt file. I was tired of the stuipd search engine spiders linking to my main web page, or to some esoteric category-specific page. The spiders need to be linking to the individual archives only. Bad spiders! So, I updated my robots.txt file to steer them into the individual archives. Hopefully, it's working. Here's a website that lets you simulate how a generic spider sees your website.

Posted by Peenie Wallie on April 15, 2006 at 1:04 PM

Comments

Post a comment

Name:

Email Address:(this will not be displayed)

URL:
Remember Me? YesNo

Comments: (you may use HTML tags for style)

Type the word Alabraska to verify you're a person and not a program. This is not case sensitive. Note: It will take 1-2 minutes to post your comments. After that time, you must refresh your browser. Please do not double-post comments or I will kill you. (required):

NOTICE: IT WILL TAKE APPROX 1-2 MINS FOR YOUR COMMENT TO POST SUCCESSFULLY. YOU WILL HAVE TO REFRESH YOUR BROWSER. PLEASE DO NOT DOUBLE POST COMMENTS OR I WILL KILL YOU.