How to block unwanted bots from your site with Yoast SEO

The internet is bustling with traffic, but not all of it comes from humans. Bots – automated scripts and programs – roam the digital landscape as well. Some of these bots can benefit your site or business, but others? Not so much.

In this article, we’ll explore the most important reasons why some people opt to block unwanted bots from their site. Following that, we’ll show you how to block unwanted bots from your site using Yoast SEO.

Should you block bots from your site?

Blocking Google AdsBot

One of these bots crawling the web is called Google AdsBot. Google AdsBot is designed to assess the quality of landing pages to improve ad delivery to end-users. It evaluates the content on pages linked to ads, to make sure the content on those pages is relevant and of good quality.

So why would people want to block Google AdsBot? Well, for example, if they want to prevent the bot from consuming resources on their server, especially if they are not running Google Ads. Another reason could be to avoid potential security risks associated with bots accessing the site.

However, blocking Google AdsBot can have serious negative consequences if you are running Google Ads. It can negatively impact how Google assesses the quality of the landing pages for ads, which can, in turn, affect the performance of running Google Ads campaigns.

Blocking AI bots

Many of the other bots traveling the web are AI bots. These bots are pivotal in training content for AI models. They crawl sites, harvest data, and play a big role in training Large Language Models (LLMs) and the broader field of artificial intelligence (AI).

When deciding whether or not to block AI bots from your site, you need to consider a few things. On the one hand, blocking AI bots safeguards your unique content and enhances your data security. It can also save server resources, because AI bots can generate considerable server load when crawling and indexing pages.

But on the other hand, keeping these bots at bay might hide your site from some online searches and might hold back the growth of AI technologies. And, you should be careful: if you try to block certain unwanted bots, you might accidentally block the good ones.

Do you want to learn more about all this? Dive into our detailed article about the implications of (not) blocking AI bots.

How to block unwanted bots from your site

Yoast SEO has introduced a feature to make it simpler for users to block unwanted AI bots. Here’s how you can use Yoast SEO’s settings to prevent these AI bots from crawling your site.

If you’re using Yoast SEO free, you’ll only be able to block Google AdsBot from crawling your site. Yoast SEO Premium users also have the possibility to block Google Bard and Vertex AI bots, OpenAI GPTBot, an Common Crawl CCBot from your site.

Related articles