Today, I got an email from google adsense,which saying
” While reviewing your ad implementation, we noticed that your robots.txt file is currently preventing our AdSense crawler from reaching a significant number of pages with ads in your account.” I have searched for this problem, and found a lot people got this mail.Then , I check my sites’ robots.txt, and add the
Disallow: ” which the adsense suggested. Most of my sites are wordpress and blogger, my robots.txt isn’t blocking any bots on pages where ads are displayed. just like this site: dobeweb.com, you can visit http://dobeweb.com/robots.txt, and there is nothing. For blogger, and I have checked the blogger robots.txt, just browse http://yoursitename.blogspot.com/robots.txt,what you see??
So, there is nothing wrong with these robots.txt. I think this mail have send to all of us.
you can use edit the blogger robots.txt if you have added sitemap to google in google webmater center.
more about this, here: