Comments on: Best Robots.txt For Wordpress

By: TK

TK — Sat, 17 Oct 2009 00:29:42 +0000

Why on earth would anyone disallow sitemap.xml the whole point of a sitemap is for search engines to read it.

By: Dan O'Neil

Dan O'Neil — Tue, 08 Jan 2008 16:29:24 +0000

Thanks for this useful info – just one quick question…

If your blog is in a subdirectory of the main site, e.g. http://www.example.com/blog what do you change the lines which start with a */… rather than just / e.g.
Disallow: */feed/

Thanks,

Dan O'Neil
http://www.aquariuscoaching.co.uk

By: Kamil Wojcicki

Kamil Wojcicki — Thu, 20 Dec 2007 12:43:21 +0000

Aha, so the web server doesn't redirect to the index file when asked for a root of a sub-domain, it just returns a contents of the index file. Cheers

By: Turk Hit Box

Turk Hit Box — Thu, 20 Dec 2007 11:12:52 +0000

Sure,
1- http://www.turkhitbox.com/ and http://www.turkhitbox.com/index.php are same pages, we don't want both of them listed

2- It removes all urls with a question mark.
3- It removes all urls that has .php in the middle.

By: Kamil Wojcicki

Kamil Wojcicki — Thu, 20 Dec 2007 06:40:07 +0000

Could you explain the reasons for disabling the following?
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$

By: Kamil Wojcicki

Kamil Wojcicki — Wed, 19 Dec 2007 12:46:10 +0000

Cheers for that, I have updated my robots file

By: Silkenhut’s World » Blog Archive » Add a robots.txt file to your blog

Mon, 16 Jul 2007 01:17:17 +0000

[...] Best Robots.txt for wordpress [...]

By: eylultoprak

eylultoprak — Fri, 06 Jul 2007 15:37:29 +0000

türkçeleştire bilir misin eğer türkçe olursa.. sitemizde yayınlamayı düşünüyorum ..tam anlamadımda

By: Bill Petro

Bill Petro — Thu, 05 Jul 2007 21:10:13 +0000

What's the final verdict on allowing/dis-allowing sitemap.xml?

By: Wayne Price

Wayne Price — Tue, 03 Jul 2007 20:53:12 +0000

Where do you put the robots.txt file?