HTTP headers

The entire web is based on HTTP, knowing the basics about the different types of HTTP headers you can send is a base requirement for anyone doing web development and website optimization.


Preventing your site from being indexed, the right way

5 June 2017 by Joost de Valk » - 9 Comments

prevent your site being indexed

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt »

Category: Technical SEO
Tags: ,

HTTP status codes and what they mean for SEO

20 January 2017 by Edwin Toonen » - 8 Comments

HTTP status codes

HTTP status codes, like 404, 301 and 500, might not mean much to a regular visitor, but for SEOs they are incredibly important. Not only that, search engine spiders, like Googlebot, use these to determine the health of a site. These status codes offer a way of seeing what happens between the browser and the server. »

Category: Technical SEO
Tags: ,

Playing with the X-Robots-Tag HTTP header

3 January 2017 by Joost de Valk » - 2 Comments

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss what the possibilities are and how this might be a better »

Category: Technical SEO
Tags: , ,

HTTP 451: content unavailable for legal reasons

21 January 2016 by Joost de Valk »

At the end of last year, a new HTTP status code saw the light. This status code, HTTP 451, is intended to be shown specifically when content has been blocked for legal reasons. If you’ve received a take down request, or are ordered by a judge to delete content, this is the status code that »

Category: WordPress
Tag:

Should we move to an all HTTPS web?

13 January 2014 by Joost de Valk » - 45 Comments

There was a bit of tweeting in the SEO community today because Bing introduced an HTTPS version of their site and people thought that would mean they’d lose their keyword data. That’s not true, if you take the right precautions. I thought I’d write a bit of an intro in how all this works so »

Category: Technical SEO
Tags: , ,

HTTP 503: Handling site maintenance correctly for SEO

18 March 2013 by Joost de Valk »

Sometimes, your site will need some down-time so you can fix things or update plugins. Most of the time, this tends to be a relatively short period in which Google will most likely not attempt to crawl your website. However, in the case that you need more time to get things fixed, chances are much »

Category: Technical SEO
Tag: