HTTP headers archives

Recent HTTP headers articles


Preventing your site from being indexed, the right way

19 March 2021 | 18 Comments | Joost de Valk

We’ve said it way back when, but we’ll repeat it: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. Do you know why it keeps amazing us? Because …

Read: "Preventing your site from being indexed, the right way"
Preventing your site from being indexed, the right way


HTTP status codes and what they mean for SEO

20 January 2017 | 8 Comments | Edwin Toonen

HTTP status codes, like 404, 301 and 500, might not mean much to a regular visitor, but for SEO’s they are incredibly important. Not only that, search engine spiders, like Googlebot, use these to determine the health of a site. These status codes offer a way of seeing what happens between the browser and the server. …

Read: "HTTP status codes and what they mean for SEO"
HTTP status codes and what they mean for SEO

Playing with the X-Robots-Tag HTTP header

3 January 2017 | 2 Comments | Joost de Valk

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss what the possibilities are and how this might be a better …

Read: "Playing with the X-Robots-Tag HTTP header"
Playing with the X-Robots-Tag HTTP header