what is crawlability

SEO basics
What is crawlability?

SEO basics: What is crawlability?

February 20th, 2017 – 6 Comments

Ranking in the search engines requires a website with flawless technical SEO. Luckily, the Yoast SEO plugin takes care (of almost) everything on your WordPress site. Still, if you really want to get most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, I’ll explain one of the most important concepts of technical SEO: crawlability.

What is the crawler again?

A search engine like Google consists of a crawler, an index and an algorithm. The crawler follows the links. When Google’s crawler finds your website, it’ll read it and its content is saved in the index.

A crawler follows the links on the web. A crawler is also called a robot, a bot, or a spider. It goes around the internet 24/7. Once it comes to a website, it saves the HTML version of a page in a gigantic database, called the index. This index is updated every time the crawler comes around your website and finds a new or revised version of it. Depending on how important Google deems your site and the amount of changes you make on your website, the crawler comes around more or less often.

Read more: ‘SEO basics: what does Google do’ »

And what is crawlability?

Crawlability has to do with the possibilities Google has to crawl your website. Crawlers can be blocked from your site. There are a few ways to block a crawler from your website. If your website or a page on your website is blocked, you’re saying to Google’s crawler: “do not come here”. Your site or the respective page won’t turn up in the search results in most of these cases.

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info


There are a few things that could prevent Google from crawling (or indexing) your website:

  • If your robots.txt file blocks the crawler, Google will not come to your website or specific web page.
  • Before crawling your website, the crawler will take a look at the HTTP header of your page. This HTTP header contains a status code. If this status code says that a page doesn’t exist, Google won’t crawl your website. In the module about HTTP headers of our (soon to be launched!) Technical SEO training we’ll tell you all about that.
  • If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page, but won’t add it to its index.

This flow chart might help you understand the process bots follow when attempting to index a page:

Want to learn all about crawlability?

Although crawlability is just the very basics of technical SEO (it has to do with all the things that enable Google to index your site), for most people it’s already pretty advanced stuff. Nevertheless, if you’re blocking – perhaps even without knowing! – crawlers from your site, you’ll never rank high in Google. So, if you’re serious about SEO, this should matter to you.

If you really want to understand all the technical aspects concerning crawlability, you should definitely check out our Technical SEO 1 training. In this SEO course, we’ll teach you how to detect technical SEO issues and how to solve them (with our Yoast SEO plugin).

Keep reading: ‘How to get Google to crawl your site faster’ »

 


6 Responses to SEO basics: What is crawlability?

  1. Karanja M
    By Karanja M on 28 February, 2017

    Does the crawl bots go through content on table press table.

  2. Emmerey Rose
    By Emmerey Rose on 24 February, 2017

    Great post Marieke. We always have to learn the basics. Thanks for sharing. But I wanna know, how often do you think Google crawls every website?

  3. diseño web Tijuana
    By diseño web Tijuana on 23 February, 2017

    Thanks for the information this will help me to improve the structure of my site

    An apology for my bad english
    regards

  4. Jordan Clark
    By Jordan Clark on 21 February, 2017

    Thank you for sharing! Waiting for the course!

  5. Bear Mark
    By Bear Mark on 21 February, 2017

    Thats nyc one..

  6. Peter Zeilstra
    By Peter Zeilstra on 21 February, 2017

    Very nice article! Thank you Marieke.


Check out our must read articles about SEO basics