HTTP headers archives

Recent HTTP headers articles



Preventing your site from being indexed, the right way

We’ve said it way back when, but we’ll repeat it: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. Do you know why it keeps amazing us? Because …

Read: "Preventing your site from being indexed, the right way"
Preventing your site from being indexed, the right way


What’s the X-Robots-Tag HTTP header? And how to use it?

3 January 2017 Joost de Valk

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains, or other content search engines will be allowed to crawl. But did you know there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss the possibilities and how this might be a better option for your …

Read: "What’s the X-Robots-Tag HTTP header? And how to use it?"
What’s the X-Robots-Tag HTTP header? And how to use it?