How to do SEO for Bloggers Series 5.b : Robot.txt

Image via Wikipedia

Section 5.b of The 12 Most Important SEO Tips for #blogchat Bloggers

Each section of the 12 Most Important SEO Tips for #blogchat Bloggers will be expanded upon in this blog.  Subscribe for updates!

Section 5 of the 12 Most Important SEO Tips for #blogchat Bloggers discussed XML Site maps, <–click to see.  What was not mentioned is an invaluable addition called the Robots.txt file.  By default, WordPress has a virtual file set to nofollow.  Generically this is NOT good for your blog for SEO purposes.   The simplest thing to do is to create a Robots.txt file and upload it to your site.  An alternate approach is to install a plugin such as  KB Robots.txt by netroby to edit Robosts.txt from within WordPress (for the FTP impaired). I recommend this Plugin only because it is the most current and works in WP 3.1.3.

Let’s talk WHY before how.  When Spiders come a calling to your domain, the nice ones read the robots.txt file.  Think of this as the no shirt no service notice.  If your Robots.txt says nofollow, then the spiders head off to index your competition instead.

You want the spiders to index your domain and your posts IF you have any intention of your site being ranked for SEO.  Do you want to be found online for what you are writing about?  If so install a modified Robots.txt file.  If not, leave it as nofollow and stop reading right now.  I meant it!  Shoo!  As with everything there are some quirks with how this will working depending on how you’ve installed you site.  If you have any issues give me a Tweet.

How to create a Robots.txt file for WordPress

You need to create a simple .txt file and can use Notepad or an HTML editor like CoffeCup, avoid  MS Word.  Or, use the plug-in mentioned above.  You can FTP the file to the root of your site.  The URL address for the file is YourDomain.com/Robots.txt and you can view http://SeoCMO.com/Robots.txt as an example.  Or use the plug-in to achieve the same purpose.  Now, what do you include or exclude?  There are many options, what I can tell you is that bad Bots will ignore your file, so I personally do not try to exclude them.  The main objective of the Robots.txt file for WordPress is to get the Search Engines to find your content without duplicating it! You can and should exclude known admin and personal folders.  You also DO NOT WANT your category or tags or author pages, etc to be read a second time!  This can make it look like you have duplicate content, a big No/No!

Do not stress out over this! If you want a deeper dive geek-fest checkout http://www.robotstxt.org/orig.html for more than I’ll ever want to know about Robots.txt files! Check out my example above as a good starting point and Tweet away if you get lost! Now go get found by Google!