robots.txt

Print Friendly, PDF & Email

A few months ago, we talked about adding a robots.txt file so that webcrawlers won’t index archived courses, such as the course material for cs110 back in fall of 2012.

The robotstxt.org page is, unsurprisingly, the authority on creating a robots.txt file. You put the robots.txt file in the top level directory of the webserver, which on tempest is /var/www/html. When I looked, there was actually already a robots.txt page, so I just added a few lines like  “Disallow:/~cs110f12/”. Hopefully this will prevent robots from indexing those archived courses.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *