Yoast SEO settings: Crawl optimization
Crawlability is essential in SEO. If you want search engines to find your site and show it in the search results, your site must be crawlable. Not only that, but you must ensure that search engines can crawl the pages you want to rank with. With Yoast SEO, you can omit all the URLs that don’t have any SEO value. This makes crawling more efficient and alos reduces your site’s carbon footprint. In this article, we’ll explain all of the crawl optimization settings one by one.
Table of contents
Where to find the crawl optimization settings
You can find the crawl optimization settings by following these steps
- In your WordPress Dashboard, go to Yoast SEO
- Go to Settings
- Under Advanced, go to Crawl optimization
Remove unwanted metadata
The first section is Remove unwanted metadata. WordPress adds a lot of URLs and tags to your website’s header and <head> section, which are unnecessary and don’t have any SEO value. We explain what the toggles mean below.
Remove shortlinks
In the <head> section of a single post, WordPress creates the following shortlink output.
<link rel='shortlink' href='http://testsite.com/?p=1' />
The shortlink is basically a shortened version of the URL of the same page. With this toggle, you can remove that output.
Remove REST API links
The WordPress REST API is a developer-oriented feature that lets applications interact with your WordPress site. Automatically, WordPress adds a REST API link to the <head> section of your site for discoverability.
<link rel="https://api.w.org/" href="http://testsite.com/wp-json/" />
However, most sites don’t use the WordPress REST API. If your site doesn’t use the WordPress REST API, you can safely use the toggle to remove this link.
Remove RSD/WLW links
The RSD (Really Simple Discovery) link in the <head> section of your site is for when you use these types of services. If you do not, it is safe to remove the link with this toggle.
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="http://testsite.com/xmlrpc.php?rsd" />
The WLW link is intended for users of the discontinued Windows Live Writer. If you do not use it, you can safely remove this link as well.
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="http://testsite.com/wp-includes/wlwmanifest.xml" />
Remove oEmbed links
With this toggle, you can remove the oEmbed links from the <head> section of all your single posts.
<link rel="alternate" type="application/json+oembed" href="http://testsite.com/wp-json/oembed/1.0/embed?url=http%3A%2F%2Ftestsite.com%2F2022%2F05%2Fhello-world%2F" /><link rel="alternate" type="text/xml+oembed" href="http://testsite.com/wp-json/oembed/1.0/embed?url=http%3A%2F%2Ftestsite.com%2F2022%2F05%2Fhello-world%2F&format=xml" />
These links help other sites consume your content. You won’t harm any of your content by removing them.
Remove generator tag
The generator tag displays the WordPress version your site is using.
<meta name="generator" content="WordPress 6.0" />
This tag has no SEO value, and can potentially be a security threat. You can easily remove it with this toggle.
Pingback HTTP header
Pingbacks are used to notify you when someone has added a link to your site. However, this standard is very old, and you are most likely not using it anymore. If you switch the toggle to remove, it will remove the X-Pingback: http://testsite.com/xmlrpc.php from the response header.
Powered by HTTP header
With this toggle, you remove the information about the PHP version your site is using from the response header. This information is not required for your site to function properly, so you can safely remove it.
Disable unwanted content formats
The next section is called Disable unwanted content formats. Your site probably has more URLs than you realize. For example, WordPress creates feeds for a lot of content on your site, which can be a problem for crawlers. A crawler will start crawling the URLs and may run out of crawl budget before reaching your important posts and pages. Removing those URLs lets search engines crawl your site more efficiently.
In the crawl optimization settings in Yoast SEO, you can toggle multiple switches that let you keep or remove the various feeds. We don’t automatically remove them for each site because we can’t predict the needs of all Yoast SEO users. If you don’t get any value from them, we recommend you switch the toggles off.
Remove global feed
Remove global feed lets you remove the global feed, which is an overview of your recent posts.
- Type of page: any page
- Example feed: https://www.example.com/feed/
Remove global comments feed
Remove global comments feed lets you remove the global comments feed, an overview of recent comments on your site. Disabling this feed will also disable the post comments feeds.
- Type of page: any page
- Example feed: https://www.example.com/comments/feed/
Remove post comments feed
Remove post comments feed is for removing the feed for recent comments on each post. If you enable or disable Remove global comments, the Remove post comments feed will automatically be enabled or disabled too.
- Example feed: https://www.example.com/example-post/feed/
Remove post author feeds
Remove post authors feed is for removing the feeds for recent posts by specific authors.
- Type of page: author archive, e.g., https://www.example.com/author/admin/
- Example feed: https://www.example.com/author/admin/feed/
Remove post type feeds
Remove post type feeds lets you remove post type feeds, which provide information about your recent posts, for each post type.
- Type of page: post type archive, e.g., https://www.example.com/my-books/
- Example feed: https://wwww.example.com/my-books/feed/
Remove category feeds
Remove category feeds lets you remove category feeds, which provide information about your recent posts, for each category.
- Type of page: category archive, e.g., https://www.example.com/fiction/
- Example feed: https://www.example.com/category/fiction/feed/
Remove tag feeds
Remove tag feeds lets you remove tag feeds, which provide information about your recent posts, for each tag.
- Type of page: tag archive, e.g., https://www.example.com/tag/fantasy/
- Example feed: https://www.example.com/tag/fantasy/feed/
Remove custom taxonomy feeds
Remove custom taxonomy feeds lets you remove custom taxonomy feeds, which provide information about your recent posts, for each custom taxonomy.
- Type of page: custom taxonomy archive, e.g., https://www.example.com/book-genre/crime/
- Example feed: https://www.example.com/book-genre/crime/feed/
Remove search results feeds
Remove search results feeds lets you remove search results feeds, which provide information about your search reults
- Type of page: search results, e.g., https://www.example.com/?s=world
- Example feed: https://basic.wordpress.test/search/world/feed/rss2/
Remove Atom/RDF feeds
The final toggle allows you to remove Atom/RDF feeds, which are specific formats for feeds.
- Type of page: any page
- Example feed: any feed listed above, adding
/atomor/rdfin the end, e.g.:- https://www.example.com/feed/atom
- https://www.example.com/feed/rdf
- https://www.example.com/comments/feed/atom
- https://www.example.com/comments/feed/rdf
- https://www.example.com/hello-world/feed/atom
- https://www.example.com/hello-world/feed/rdf
Remove unused resources
In the Remove unused resources section, you can remove the resources that WordPress usually loads but that your site doesn’t always need.
Remove emoji scripts
If you don’t use emojis in your content, you can safely remove the JavaScript used for converting emoji characters in older browsers. You can do that by switching the toggle behind Remove emoji scripts.
Remove WP-JSON API
Remove WP-JSON API allows you to prevent robots from crawling the WordPress JSON API endpoints. Unless you’re using the WordPress REST API to output important content, you can switch this toggle off to improve your crawl efficiency. This adds a disallow rule to your robots.txt file to prevent the crawling of the WordPress JSON API endpoints., e.g. https://www.example.com/wp-json/ and https://www.example.com/?rest_route=/.
Internal site search cleanup
The next section is Internal site search cleanup. Spammers sometimes target internal site search URLs on your site for their own purposes. Those URLs might get crawled by search engines, be seen by users, and can harm your SEO. This feature identifies some common spam patterns and stops them in their tracks.
Filter search terms
First of all, you can choose to filter search terms by switching the toggle behind Filter search terms. If you enable this option, you can access more specific options, which we will discuss below.
Max number of characters to allow in searches
If you choose to filter search terms, then you can set a maximum number of characters to allow in searches. This reduces the impact of spam attacks and confusing URLs.
Filter searches with emojis and other special characters
You can also decide whether you want to block searches with emojis and other special characters, as these searches may be part of a spam attack.
Filter searches with common spam patterns
Finally, you can choose to filter searches with common spam patterns. The common spam patterns our plugin cleans up are: TALK: QQ: [：（）【】［］].
Redirect pretty URLs to ‘raw’ formats
Next, you can choose to redirect pretty URLs for search pages to the raw format. WordPress supports two endpoint formats for site search queries:
- A raw format:
example.com/?s=example - A pretty format:
example.com/search/example
The pretty format will only be supported when pretty permalinks are enabled. When both formats exist, this can lead to problems, because this doubles the number of URLs that search engines can crawl. In addition, it can increase the number of ways in which your site can be attacked by spammers.
Therefore, we provide an option to turn off one of these formats in Yoast SEO. The plugin then redirects requests from the pretty format to the raw format, while maintaining any query parameters and/or pagination. We disable the pretty format because the raw format is relatively universal and language- and territory-agnostic and more natively interoperable with most analytics and tracking systems.
Prevent crawling of internal site search URLs
The final option in this section is to prevent crawling of internal site search URLs. This adds a disallow rule to your robots.txt file so your internal site search URLs won’t be crawled.
In general, blocking your internal search pages via your robots.txt would not be our advice. It’s better to allow search engines to crawl these pages, but to prevent them from indexing them by using a noindex tag, which Yoast SEO automatically does for your site. However, if your search results pages are being crawled excessively and your crawl budget is impacted, or if your search results pages are under attack, you should enable this option.
Advanced: URL cleanup
These are advanced settings that you should only use if you know what you are doing! To learn more read the Advanced crawl settings article.
Will using the crawl settings affect my site’s rankings?
Using the crawl settings in Yoast SEO will not harm your website’s crawlability or rankings. The crawl settings are there to help you clean up unnecessary URLs and can help search engines crawl your site more efficiently.
In addition, it’s important to note that the crawl settings in Yoast SEO do not have any effect on the crawl rate of a website. This means that the speed at which search engines crawl and index a website is not impacted by the crawl settings in Yoast SEO
Did this article answer your question?
Related articles
Get Yoast SEO Premium
Drive more traffic to your site. Use AI to save time doing SEO tasks. Make SEO easier. And get 24/7 support.
Includes Local SEO, Video SEO & News SEO.