HTTP headers

The entire web is based on HTTP, knowing the basics about the different types of HTTP headers you can send is a base requirement for anyone doing web development and website optimization.



Preventing your site from being indexed, the right way

5 June 2017 by Joost de Valk - 9 Comments

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt …

Read: "Preventing your site from being indexed, the right way"
googlebot prevents your site being indexed

HTTP status codes and what they mean for SEO

20 January 2017 by Edwin Toonen - 8 Comments

HTTP status codes, like 404, 301 and 500, might not mean much to a regular visitor, but for SEOs they are incredibly important. Not only that, search engine spiders, like Googlebot, use these to determine the health of a site. These status codes offer a way of seeing what happens between the browser and the server. …

Read: "HTTP status codes and what they mean for SEO"
Category: Technical SEO
Tags: ,
HTTP status codes

Playing with the X-Robots-Tag HTTP header

3 January 2017 by Joost de Valk - 2 Comments

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss what the possibilities are and how this might be a better …

Read: "Playing with the X-Robots-Tag HTTP header"



Browse other categories