robots.txt

Hide your dev and stage environments with robots.txt

Published March 22, 2011

Have you ever had a client call you up saying their new website, which has not launched yet, is showing up in Google?

No matter how much you think, "we won't get crawled", those pesky search engines always seem to find your development site. Thankfully there is an agreed-upon standard for removing a website (or just certain pages) from search engines called the robots.txt file.