robots.txt
A few months ago, we talked about adding a robots.txt file so that webcrawlers won’t index archived courses, such as the course material for cs110 back in fall of 2012. The robotstxt.org page is, unsurprisingly, the authority on creating a … Continue reading