The new whitehouse is so open it wants Robots to come in too
Received:: January 24, 2009 | Category:: ygolonhcet
via Ajaxian ยป The new whitehouse is so open it wants Robots to come in too.
Congrats to the new President, and to the great development team that created change.gov and the like, and now get to run whitehouse.gov.
Jason Kottke poked around and compared the robots.txt files from both administration.
The Bush version was ~2400 lines long:
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
….
Obama?
User-agent: *
Disallow: /includes/
Nice to see the cleanup begin.