Comments on: Preventing Duplicate Content On WordPress Blogs

By: Kham Tran

Kham Tran — Wed, 17 Dec 2008 12:01:09 +0000

Thanks for pointing that out digitalizes. I have now updated the post and removed the * from the end of Disallow: /wp- as this isn't required.

By: digitalizes

digitalizes — Tue, 02 Dec 2008 17:14:21 +0000

I have been doing a bit more digging and used a robots.txt validator tool to check your script. It did not like this part –

Disallow: /wp-*

The problem was caused by the *

I have decided to use the following instead (which excludes folders individually)

User-agent: *
Disallow: /wp-admin
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: /wp-includes
Disallow: /index.php?
Disallow: /page/
Disallow: /category/
Disallow: /tag/

I am going to wait for the robots.txt file to be downloaded by Googlebot and check the validation again from with my Google Webmaster Tools account.

By: digitalizes

digitalizes — Tue, 02 Dec 2008 16:54:51 +0000

I have been looking into this subject today and have come across a lot of posts from 2007. This is the most up to date post I have been able to find (so far, the quest continues) but what you are saying makes a lot of sense.

I actually started out looking at using robots.txt to help with security issues (I guess blocking the wp- folders will help with that) and as usual I have ended up steering off course but still finding some useful information!

Thanks!