Also, robots.txt completely stops crawling whereas adding the proper meta robot will allow for crawling and link following and still prevent the duplicate entries.

I wrote a post on my blog at http://www.wmtoolbox.com/code-snippets/blogging-and-duplicate-content-rank-better/ on how I do my robots

Still in french , V 0.1 , just release this day

Give your page the 301 +url , 302 + url , 404 error code from the “edit post page”

Traduction needed ! ( mail can be found on our site if you can help )

http://www.wordpress-seo.com/seo-http-error-manager.php

How it works :
Download, copy to plugin directory, Activate.

When editing you can choose Leave, 301, 302, 404 ( and url for 301, 302 )

Licence , “can be stolen” and we still dont care of man who still want to give a licence for “nothing” like this !

Comment for make it better welcome

Sorry for my english
++

I think I will change things back. Thanks for the responses.

“…any incoming links to excluded pages on your site will be effectively lost, as these excluded pages can’t pass the benefit on.”

Right, but not immediately. There is a very high probability he could change it back with no penalty if he did it very soon.

I would just add one caveat though: if you exclude content with robots.txt, and then use Google’s URL removal tool, you will have to wait at least six months before re-inclusion. So, don’t do that

I’d also say there is a concrete risk to changing the robots.txt file - any incoming links to excluded pages on your site will be effectively lost, as these excluded pages can’t pass the benefit on. You may want to examine your incoming links in any engine you can find, as well as landing pages and referral logs, to assess the risk here.

Greg - personally, I’d say the safest option is to carry on as you are, if things are working; as the saying goes, if it ain’t broke, don’t fix it! You could even try this one out for a while and see how well it works. SEO’s a continual cycle of evaluation and implementation.

Don’t sweat it. Using the robots.txt isn’t a mistake. It also isn’t the only way to accomplish the goal of avoiding duplicate content. And, yes, none of this is a cure for bad writing. SEO Rant had an excellent additional suggestion which would also work. I think it might work better in the long run, but that’s my opinion. SEO Rant’s statements about the effectiveness of modifying .htaccess vs. robots.txt are also opinion. Michael Gray’s statements in the video are his opinions. In all these cases, the opinions are backed up by facts and/or experience.

Your situation, Greg, might be one of “if it ain’t broke, don’t fix it”. You could remove or revert your robots.txt file back to its original code without consequence.

Search engine bots don’t come around every single day, but they come around often enough so that if you change it back it would be like it never happened. Again, nothing to worry about, really.

The reason I posted the video is because its advice is something a beginner could do. There’s better, and then there’s better, and at some point you’re splitting hairs.

Generally speaking, you’re much better off spending your time on producing quality content. SEO efforts are meant to give that quality content an edge in search results. Sometimes, it’s not much of an edge, so people obsess over every possible detail.

Don’t feel bad about changing the robots.txt file. It will not hurt you. You will not know for sure if it helped until after some time has passed (probably a couple months). But if you were doing well to begin with, perhaps it wasn’t necessary.

Now I do not know if using the robots.text file was a mistake. This is what is frustrating about reading any blog relating to internet marketing etc.

You all talk over our heads and carrying on conversations about a technical topics that most of us know diddly squat about.

A little information about a topic that requires a deeper understanding of the issues is dangerous in the hands of a noob like me.

Damn, now I feel stupid.

Sorry Michael, I am feeling frustrated.

Why don’t you two (Michael & SEO Ranter) get together and write a manifesto on this topic?

Something anyone could read, be more well informed, and feel like they got some great value.

It would help you both and help the average blogger that wants to blog without becoming an technical whiz and SEO expert.

Help!?