There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:
We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.
The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!
This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.
The robots.txt file is a file that follows a strict standard. With the robots.txt you can tell search engines where they can and cannot go on your site. Learn how to use the robots.txt to your advantage to block the search engine from spidering parts of your site it doesn't need to visit!
Crawl errors occur when a search engine tries to reach a page on your website but fails at it. Let’s shed some more light on crawling first. Crawling is the process where a search engine tries to visit every page of your website via a bot. A search engine bot finds a link to your website and starts …
Read: "Basic SEO: What are crawl errors?"
Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …
Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Paginated archives have long been a topic of discussion in the SEO community. Over time, best practices for optimization have evolved, and we now have pretty clear definitions. This post explains what these best practices are. It’s good to know that Yoast SEO applies all these rules to every archive with pagination. Indicate that an …
Read: "Pagination & SEO: best practices"
Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there as glue for other pages, or simply because whatever regulations require them to be accessible on your website. As a regular visitor to our website, …
Read: "Which pages should I noindex or nofollow?"
This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.
Read: "hreflang: the ultimate guide"
Whenever you make some big changes to your website, for instance to your brand name, you’re probably eager for these changes to show in the search results. Unfortunately, it can take a while for Google to crawl your site again and until then, it will show the indexed version of your site in the results, …
Read: "Ask Yoast: Changes to your site and the search results"
We recently made some changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix …
Read: "Closing a spider trap: fix crawl inefficiencies"
How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. …
Read: "SEO for a new website: the very first things to do"
We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt …
Read: "Preventing your site from being indexed, the right way"
Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal …
Read: "Block your site’s search result pages"
