YoastCon 2023!
11 May, Nijmegen, The Netherlands
Yoast’s famous SEO conference is back!
Crawl directives
There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:
- tell a search engine to not crawl a page at all;
- not to use a page in its index after it has crawled it;
- whether to follow or not to follow links on that page;
- a lot of “minor” directives.
We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.
Beginners level
SEO basics: What is crawlability? »
What is crawlability? In what ways could you block Google from crawling (parts of your) site? Why is crawlability important for SEO?
Expert level
The ultimate guide to robots.txt »
The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!
Must read articles about Crawl directives
-
Trying to prevent indexing of your site by using robots.txt is a no-go, use X-Robots-Tag or a meta robots tag instead! Here's why.
-
Canonical URLs tell search engines that similar pages are actually the same. Learn when and how to set a URL as rel=canonical in this guide.
-
Want to keep a page out of the search results? Ask yourself if it should be on your site anyways. If it should, use a robots meta tag to prevent it from being indexed.
-
Search engines need a bit of help to qualify links; use the nofollow, sponsored and UGC attribute to help them out. With Yoast SEO it's easy!
Recent Crawl directives articles
Yoast SEO 20.4 makes our crawl optimization settings available to all users -- helping you reduce unnecessary crawling by bots.
Is technical SEO a mystery to you? Find out which elements you should focus on to optimize your website for both SEO and user experience.
Over the last year or so, we've seen large-scale, widespread SEO spam 'attacks' on WordPress sites; all targeting their internal site search.
