Comments on: Best Robots.txt For WordPress

By: ates

ates — Sun, 04 Jul 2010 19:50:00 +0000

why disallow page ? is this not mean that robot.txt disallow pages on my web site?

By: justblog

justblog — Tue, 06 Apr 2010 09:41:38 +0000

how about it?
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /tag
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

By: Cayman Web Design

Cayman Web Design — Fri, 26 Mar 2010 01:30:03 +0000

Thanks for the robots.txt :) very helpful indeed.

Just one question – I am curious as to why you would disallow the sitemap though. Its not or it shouldn’t duplicate the content and instead be a unique page that has all your website’s menu items enabling the bots to crawl your website more efficiently.

I am interested in one of your banner spots but can you please send me more info – site stats – unique visits, location of clicks, bounce rates, no. of links leaving the homepage.

By: TK

TK — Sat, 17 Oct 2009 00:29:42 +0000

Why on earth would anyone disallow sitemap.xml the whole point of a sitemap is for search engines to read it.

By: Dan O'Neil

Dan O'Neil — Tue, 08 Jan 2008 16:29:24 +0000

Thanks for this useful info – just one quick question…

If your blog is in a subdirectory of the main site, e.g. http://www.example.com/blog what do you change the lines which start with a */… rather than just / e.g.
Disallow: */feed/

Thanks,

Dan O'Neil
http://www.aquariuscoaching.co.uk

By: Kamil Wojcicki

Kamil Wojcicki — Thu, 20 Dec 2007 12:43:21 +0000

Aha, so the web server doesn't redirect to the index file when asked for a root of a sub-domain, it just returns a contents of the index file. Cheers

By: Turk Hit Box

Turk Hit Box — Thu, 20 Dec 2007 11:12:52 +0000

Sure,
1- http://www.turkhitbox.com/ and http://www.turkhitbox.com/index.php are same pages, we don't want both of them listed

2- It removes all urls with a question mark.
3- It removes all urls that has .php in the middle.

By: Kamil Wojcicki

Kamil Wojcicki — Thu, 20 Dec 2007 06:40:07 +0000

Could you explain the reasons for disabling the following?
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$

By: Kamil Wojcicki

Kamil Wojcicki — Wed, 19 Dec 2007 12:46:10 +0000

Cheers for that, I have updated my robots file :D

By: Silkenhut’s World » Blog Archive » Add a robots.txt file to your blog

Mon, 16 Jul 2007 01:17:17 +0000

[...] Best Robots.txt for wordpress [...]