Robots.txt is great for stopping search engine spiders from indexing duplicate content, which essentially confuses search engines about the location of your page. This guide shows you how to edit the robots.txt file for wordpress’s many duplicate content issues. Link