Crawl directives archives - Page 2 of 3

Recent Crawl directives articles



Block your site’s search result pages

16 August 2018 | 7 Comments | Michiel Heijmans

Why should you block your internal search result pages for Google? Well, how would you feel if you are in need of an answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search …

Read: "Block your site’s search result pages"
block internal search pages


Media / attachment URL: what to do with them?

In our major Yoast SEO 7.0 update, there was a bug concerning attachment URLs. We quickly resolved the bug, but some people have suffered anyhow (because they updated before our patch). This post serves both as a warning and an apology. We want to ask all of you to check whether your settings for the …

Read: "Media / attachment URL: what to do with them?"
attachment pages fix



Changes to your site and the search results

12 January 2018 | 17 Comments | Joost de Valk

Whenever you make some big changes to your website, for instance to your brand name, you’re probably eager for these changes to show in the search results. Unfortunately, it can take a while for Google to crawl your site again and until then, it will show the indexed version of your site in the results, …

Read: "Changes to your site and the search results"

Closing a spider trap: fix crawl inefficiencies

12 October 2017 | 4 Comments | Joost de Valk

Quite some time ago, we made a few changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you …

Read: "Closing a spider trap: fix crawl inefficiencies"
spider traps

SEO for a new website: the very first things to do

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. …

Read: "SEO for a new website: the very first things to do"
seo for a new website - first things to do