websense

Often in conversations with clients, potential clients, the mention of a robots.txt file brings a glazed over look. When you work in this field every day, the jargon can become second nature. However, for for those who are responsible for it and wonder if Google ever violates your wishes, here’s a great video Matt Cutts put togeter explaining how Google handles the robots.txt.

It’s pretty cool because it describes how/why they’ll return URLS in SERPs even though a webmaster may have accidentally, or intentionally “blocked” it with the robots.txt.