User-agent: * Disallow: / To remove your site from Search Engine only and prevent just Googlebot from crawling your site in the future, place the following robots.txt file in your server root: User-agent: Googlebot Disallow: / Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below. For your http protocol (http://yourserver.com/robots.txt): User-agent: * Allow: / For the https protocol (https://yourserver.com/robots.txt): User-agent: * Disallow: /