Crawl directives archives

Google doesn’t always spider every page on a site instantly. Sometimes, it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it’s time to optimize your crawl budget. In this article, we’ll discuss what a ‘crawl budget’ is and what …

Bots have become an integral part of the digital space today. They help us order groceries, play music on our Slack channel, and pay our colleagues back for the delicious smoothies they bought us. Bots also populate the internet to carry out the functions they’re designed for. But what does this mean for website owners? …

The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!

Do you want to outrank your competition? Then basic knowledge of technical SEO is a must. Of course, you also need to create great and relevant content for your site. Luckily, the Yoast SEO plugin takes care of (almost) everything on your WordPress site. Still, it’s good to understand one of the most important concepts …

If you own a website or are thinking of creating one, you should be mindful about your ecological footprint. Because the tech industry plays a big role in the current climate crisis. With its data centers, large manufacturing operations, and huge amounts of e-waste, it’s accountable for nearly 4% of global CO2 emissions. Which, shockingly …

We’re very excited to be releasing Yoast SEO 20.4. With this release, we’re bringing our crawl optimization feature to Yoast SEO Free. With this feature, you can improve your SEO and reduce your carbon footprint with just a few clicks. This blog post will tell you about this feature and why we’ve brought it to …

In this post, we’ll go into the basics of technical SEO. Now, discussing the basics of technical SEO might seem like a contradiction in terms. Nevertheless, some basic knowledge about the more technical side of SEO can mean the difference between a high-ranking site and a site that doesn’t rank at all. Technical SEO isn’t …

Over the last year or so, we’ve seen large-scale, widespread SEO spam ‘attacks’ on WordPress sites, all targeting their internal site search functionality. In most cases, these attacks aren’t harmful from an SEO perspective, but they do come with time and resource costs – for both the attacker and the victim. Most sites won’t need …

Today we are launching the first two in a series of technical SEO courses: Hosting and server configuration, and Crawlability and indexability! SEO is about more than using the right keywords and writing great content. Without a solid technical foundation, you risk losing the benefits of all your other SEO efforts. We know that technical …

Over the last couple of months, there’s been a lot of talk about a new protocol called IndexNow. This new protocol automatically notifies search engines that support it about all the updates on your site. While initially skeptical about this development, we’ve decided to add an IndexNow integration to Yoast SEO Premium 18.8. What is …