How to block unwanted bots from your site with Yoast SEO

The internet is bustling with traffic, but not all of it comes from humans. Bots – automated scripts and programs – roam the digital landscape as well. Some of these bots can benefit your site or business, but others? Not so much.

In this article, we’ll explore the most important reasons why some people opt to block unwanted bots from their site. Following that, we’ll show you how to block unwanted bots from your site using Yoast SEO.

Should you block bots from your site?

Blocking Google AdsBot

One of the bots crawling the web is called Google AdsBot. Google AdsBot is designed to assess the quality of landing pages to improve ad delivery to end-users. It evaluates the content on pages linked to ads, to make sure the content on those pages is relevant and of good quality.

So why would people want to block Google AdsBot? If they want to prevent the bot from consuming resources on their server, especially if they are not running Google Ads. Another reason could be to avoid potential security risks associated with bots accessing the site.

However, blocking Google AdsBot can have serious negative consequences if you are running Google Ads. It can negatively impact how Google assesses the quality of the landing pages for ads, which, in turn, can affect the performance of running Google Ads campaigns.

Blocking AI bots

Many of the other bots traveling the web are AI bots. These bots are pivotal in training content for AI models. They crawl sites, harvest data, and play a big role in training Large Language Models (LLMs) and the broader field of artificial intelligence (AI).

When deciding whether or not to block AI bots from your site, you need to consider the following. On the one hand, blocking AI bots safeguards your unique content and enhances your data security. It can also save server resources, because AI bots can generate considerable server load when crawling and indexing pages.

On the other hand, blocking AI bots may hide your site from some online searches and might hold back the growth of AI technologies. And, you should be careful: if you try to block certain unwanted bots, you might accidentally block the good ones.

Do you want to learn more about all this? Dive into our detailed article about the implications of (not) blocking AI bots.

How to block unwanted bots from your site

Related articles