SEO basics: What is crawlability?

Do you want to outrank your competition? Then basic knowledge of technical SEO is a must. Of course, you also need to create great and relevant content for your site. Luckily, the Yoast SEO plugin takes care of (almost) everything on your WordPress site. Still, it’s good to understand one of the most important concepts of technical SEO: crawlability.

What is the crawler again?

A search engine like Google consists of three things: a crawler, an index, and an algorithm. A crawler follows the links on the web. It does this 24/7! Once a crawler comes to a website, it saves the HTML version in a gigantic database called the index. This index is updated every time the crawler comes around your website, and finds a new or revised version of it. Depending on how important Google deems your site and the number of changes you make on your website, the crawler comes around more or less often.

Fun fact: A crawler is also called a robot, a bot, or a spider! And Google’s crawler is sometimes referred to as Googlebot.

Read more: SEO basics: what does Google do »

And what is crawlability?

Crawlability has to do with the possibilities Google has to crawl your website. Luckily, you can block crawlers on your site. If your website or a page on your website is blocked, you’re saying to Google’s crawler: “Do not come here.” As a result, your site or the respective page won’t turn up in the search results. At least, in most cases.

So how do you block crawlers? There are a few things that could prevent Google from crawling (or indexing) your website:

• If your robots.txt file blocks the crawler, Google will not come to your website or specific web page.
• Before crawling your website, the crawler will take a look at the HTTP header of your page. This HTTP header contains a status code. If this status code says that a page doesn’t exist, Google won’t crawl your website. Want to know more? We’ll explain all about this HTTP header tip in the module of our Technical SEO training!
• If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page, but won’t add it to its index.

How crawlers impact the environment

Yes, you read that right. Crawlers have a substantial impact on the environment. Here’s how: Crawlers can come to your site multiple times a day. Why? They want to discover new content, or check if there are any new updates. And every time they visit our site, they will crawl everything that looks like a URL to them. This means a URL is often crawled multiple times per day.

This is unnecessary, because you’re unlikely to make multiple changes on a URL on any given day. Not to mention, almost every CMS output URLs that don’t make sense that crawlers can safely skip. But instead of skipping these URLs, crawlers will crawl them, again and again, every time they come across one. All this unnecessary crawling takes up tons of energy resources which is harmful for our planet.

Improve your site’s crawlability with Yoast SEO Premium

To ensure you’re not wasting energy, it’s important to stay on top of your site’s crawlability settings. Luckily, you don’t have to do all the work yourself. Using tools such as Yoast SEO Premium will make it easier for you!

So how does it work? We have a crawl settings feature that removes unnecessary URLs, feeds, and assets from your website. This will make crawlers crawl your website more efficiently. Don’t worry, you’re still in control! Because the feature also allows you to decide per type of asset whether you want to actually remove the URL or not. If you want to know more, we’ll explain all about the crawl settings here.

Want to learn more about crawlability?

Although crawlability is a basic part of technical SEO (it has to do with all the things that enable Google to index your site), it’s already pretty advanced stuff for most people. Still, it’s important that you understand what crawlability is. You might be blocking – perhaps even without knowing! – crawlers from your site, which means you’ll never rank high in Google. So, if you’re serious about SEO, crawlability should matter to you.

An easy way to learn is by doing our technical SEO trainings. These SEO courses will teach you how to detect technical SEO issues and solve them (with our Yoast SEO plugin). We also have a training dedicated to crawlability and indexability! Good to know for Premium users: Yoast SEO Academy is already included at no extra cost in your Premium subscription!

Keep reading: Bot traffic: What it is and why you should care about it »