Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	AdamCubed3	Date:	September 01, 2011 03:58:29pm
Forum:	web	Subject:	Re: Ezboard content suddenly not available in the new system - why?

The same has happened with http://www.cubed-3.co.uk - it worked before the BETA version of this site was released, but now I just get the 'robots.txt' comment and can't access ANY part of the archived site.

Someone here really needs to sort this out.

Reply to this post

Reply [edit]

Poster:	mrob27	Date:	September 02, 2011 10:01:56pm
Forum:	web	Subject:	Re: Ezboard content suddenly not available in the new system - why?

Ironically, the Wayback archive can be used to view the history of an affected site's robots.txt file, even when the rest of the site is victim to this bug. For example, look at http://web.archive.org/web/*/http://www.cubed-3.co.uk/robots.txt

As you can see, in the 2003-2004 period (when the site was presumably active) the robots.txt the type of content that most active sites have:

# robots.txt
User-agent: *
Disallow: /cache/
Disallow: /admin/
(various other directories...)

This changed over the years but only in 2011 did it become the current:

User-Agent: *
Disallow: /
Noindex: /

which is causing archive.org to forget 2003-2004 ever happened.