8 June 2021
|
Joost de Valk
Sometimes, your site will need some downtime, so you can fix things or update plugins. Most of the time, this tends to be a relatively short period in which Google will most likely not attempt to crawl your website. However, in the case that you need more time to get things fixed, chances are much …
Read: "HTTP 503: Handling site maintenance correctly for SEO"
19 March 2021
|
Joost de Valk
We’ve said it way back when, but we’ll repeat it: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. Do you know why it keeps amazing us? Because …
Read: "Preventing your site from being indexed, the right way"
3 January 2017
Joost de Valk
Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains, or other content search engines will be allowed to crawl. But did you know there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss the possibilities and how this might be a better option for your …
Read: "What’s the X-Robots-Tag HTTP header? And how to use it?"
21 January 2016
Joost de Valk
In 2016, a new HTTP status code saw the light. This status code, HTTP 451, is intended to be shown specifically when content has been blocked for legal reasons. If you’ve received a take down request, or are ordered by a judge to delete content, this is the status code that allows you to indicate …
Read: "HTTP 451: content unavailable for legal reasons"