A few months ago, we talked about adding a robots.txt file so that webcrawlers won’t index archived courses, such as the course material for cs110 back in fall of 2012.
The robotstxt.org page is, unsurprisingly, the authority on creating a robots.txt file. You put the robots.txt file in the top level directory of the webserver, which on tempest is /var/www/html. When I looked, there was actually already a robots.txt page, so I just added a few lines like “Disallow:/~cs110f12/”. Hopefully this will prevent robots from indexing those archived courses.