As we’ve said many times, crawlability is an important aspect of SEO. Your site has to do away with technical hurdles that can hinder crawling while keeping the number of crawlable URLs in check. Crawlers are eager to crawl everything they can find on your site — even if you have a small site with content that hardly changes. The overhead is enormous because there are loads of URLs that make no sense for them to crawl. We’re doing something about that. In Yoast SEO Premium 18.7, we’re expanding our brand-new crawl settings with a long list of additional options.
More crawl settings in Yoast SEO Premium
In Yoast SEO Premium 18.6, we introduced the first addition to a new tab called Crawl settings. Within this section, you can find many toggles that let you turn off various things that WordPress automatically adds to your site and that most sites won’t miss. These WordPress additions generate URLs that search engines can and will crawl. At Yoast, we’re on a mission to drastically reduce the number of URLs that any given site outputs, as crawling all of these are a tremendous waste.
Only available in Yoast SEO Premium, these crawling features give you more control over what you want Google to crawl. The crawl settings are in beta, and we welcome your feedback. Together, we can help take your site to another level!
Go Premium and get access to all our features!
Premium comes with lots of features and free access to our SEO courses!
Here’s the basic crawling list with stuff added by WordPress that you can turn off to prevent search engines from crawling it. In our help documentation on the crawl settings, you’ll find all these options explained.
Short links
REST API links
RSD / WLW links
oEmbed links
Generator tags
Emoji scripts
Pingback HTTP header
Powered by HTTP header
You can also disable a huge amount of feeds generated by WordPress:
Global feed
Global comment feed
Post comment feed
Post authors feed
Post type feed
Category feeds
Tag feeds
Custom taxonomy feeds
Search results feeds
Atom/RDF feeds
These settings give you much more flexibility in deciding what you want Google to crawl. We don’t activate these settings for you automatically because it might be that your site benefits from or uses one of these things. We don’t want to break stuff!
Yoast SEO Premium comes with new options to manage crawling
Yoast SEO 19.1 enhancements
Yoast SEO 19.1 comes with several fixes and enhancements. For example, our content analyses improved, with support for em dashes as punctuation marks in the focus keyphrase field and correctly detecting caps in focus keyphrases with a period.
Among many other things, Yoast SEO recognizes keyphrases with hyphens in the slug. Our word forms feature and the filtering out of function words now works for hyphenated keyphrases. For example, the keyphrase ex-husband will be recognized in the slug ex-husband, but also the keyphrase ex-husbands.
With the word forms feature, we recognize different grammatical forms of your focus keyphrase in various places in your content. You don’t have to write the same form of your keyphrase each time but can diversify your language by changing forms. This makes you write much more naturally!
WooCommerce SEO 14.9: Global identifiers for product variations + Schema
Working with product variations has always been a hassle. WooCommerce, for instance, doesn’t allow you to add global identifiers for product variations — making it hard to add valuable data for Google to discover. Moreover, there’s no simple way to add structured data to this correctly! In WooCommerce SEO 14.9, we’re fixing this.
WooCommerce SEO now outputs the necessary Schema markup for product variations on individual products. Of course, the plugin already outputs all the product variations Schema markup through the offers property. Now, it will also output the global identifiers for product variations under the offers property. In addition, we’ve also added the variations’ SKU and URL to a product’s Schema.
Open the product variations tab in WooCommerce and fill in the global identifier details for each variation in the new Yoast SEO section. The details will be automatically added to the product structured data generated by WooCommerce SEO.
You can now add global identifiers to each product variant in WooCommerce SEO
Google sometimes shows dates in your search results, which can help users find the most relevant content. But on your WooCommerce product pages, this doesn’t make much sense — so we remove some metadata from your page to discourage them from showing dates here. So, we now remove the datePublished and dateModified attributes on the ItemPage Schema markup for a product.
Update now!
Yoast SEO 19.1, Yoast SEO Premium 18.7, and WooCommerce SEO 14.9 are out today. In Premium 18.7, we’ve vastly expanded the possibilities of the new crawl settings we introduced in 18.6. You now have many options to manage crawling on your WordPress site.
Edwin is an experienced strategic content specialist. Before joining Yoast, he worked for a top-tier web design magazine, where he developed a keen understanding of how to create great content.
February 20 - 22, 2025
Team Yoast is at Attending, Sponsoring, Yoast Booth Wordcamp Asia 2025! Click through to see who will be there, what we will do, and more!
See where you can find us next »
16 January 2025
Learn how to start your SEO journey the right way with our free webinar. Get practical tips and answers to all your questions in the live Q&A!
All Yoast SEO webinars »
3 Responses to Yoast SEO 19.1 & Premium 18.7: More crawl settings
Paul • 3 years ago
Hi nice article and very useful features, thanks for sharing!
There is one question that instantly comes to mind:
If i disable a feed a redirect is created, so google will run into that redirect and will fetch let’s say the blogpost instead of the comments of that blogpost. My question is now, should i also disallow the disabled feeds in the robots.txt to avoid the crawler always running into the redirect? Is there a benefit from doing that?
All the best,
Paul
Tyler Nguyen • 3 years ago
Hi Paul, thank you for your question!
I’ve consulted with our product team and this is our advice. In general, we would say no, but with a heavy dose of ‘it depends’. Some considerations:
– Your feeds might be getting linked to / have some degree of ‘equity’; redirecting them consolidates that to the page, but blocking them just ‘removes’ it.
– If it’s blocked, then Google (and other search engines) doesn’t know that it no longer exists, which feels inelegant.
– Redirects are (generally) ‘cheaper’ than full page loads (and 301s are heavily cached), so the efficiency/environment angle is lessened already.
We hope this answers your question, and do let us know if you have more questions!
Paul • 3 years ago
Hi Tyler,
Ok that clarifies it a bit, so lets keep the redirects for now!
Hi nice article and very useful features, thanks for sharing!
There is one question that instantly comes to mind:
If i disable a feed a redirect is created, so google will run into that redirect and will fetch let’s say the blogpost instead of the comments of that blogpost. My question is now, should i also disallow the disabled feeds in the robots.txt to avoid the crawler always running into the redirect? Is there a benefit from doing that?
All the best,
Paul
Hi Paul, thank you for your question!
I’ve consulted with our product team and this is our advice. In general, we would say no, but with a heavy dose of ‘it depends’. Some considerations:
– Your feeds might be getting linked to / have some degree of ‘equity’; redirecting them consolidates that to the page, but blocking them just ‘removes’ it.
– If it’s blocked, then Google (and other search engines) doesn’t know that it no longer exists, which feels inelegant.
– Redirects are (generally) ‘cheaper’ than full page loads (and 301s are heavily cached), so the efficiency/environment angle is lessened already.
We hope this answers your question, and do let us know if you have more questions!
Hi Tyler,
Ok that clarifies it a bit, so lets keep the redirects for now!
Thanks for the fast response!
All the best,
Paul