Yoast SEO 19.1 & Premium 18.7: More crawl settings
As we’ve said many times, crawlability is an important aspect of SEO. Your site has to do away with technical hurdles that can hinder crawling while keeping the number of crawlable URLs in check. Crawlers are eager to crawl everything they can find on your site — even if you have a small site with content that hardly changes. The overhead is enormous because there are loads of URLs that make no sense for them to crawl. We’re doing something about that. In Yoast SEO Premium 18.7, we’re expanding our brand-new crawl settings with a long list of additional options.
More crawl settings in Yoast SEO Premium
In Yoast SEO Premium 18.6, we introduced the first addition to a new tab called Crawl settings. Within this section, you can find many toggles that let you turn off various things that WordPress automatically adds to your site and that most sites won’t miss. These WordPress additions generate URLs that search engines can and will crawl. At Yoast, we’re on a mission to drastically reduce the number of URLs that any given site outputs, as crawling all of these are a tremendous waste.
Only available in Yoast SEO Premium, these crawling features give you more control over what you want Google to crawl. The crawl settings are in beta, and we welcome your feedback. Together, we can help take your site to another level!
Go Premium and get access to all our features!
Premium comes with lots of features and free access to our SEO courses!
What can you expect from the crawl settings?
Here’s the basic crawling list with stuff added by WordPress that you can turn off to prevent search engines from crawling it. In our help documentation on the crawl settings, you’ll find all these options explained.
- Short links
- REST API links
- RSD / WLW links
- oEmbed links
- Generator tags
- Emoji scripts
- Pingback HTTP header
- Powered by HTTP header
You can also disable a huge amount of feeds generated by WordPress:
- Global feed
- Global comment feed
- Post comment feed
- Post authors feed
- Post type feed
- Category feeds
- Tag feeds
- Custom taxonomy feeds
- Search results feeds
- Atom/RDF feeds
These settings give you much more flexibility in deciding what you want Google to crawl. We don’t activate these settings for you automatically because it might be that your site benefits from or uses one of these things. We don’t want to break stuff!
Yoast SEO 19.1 enhancements
Yoast SEO 19.1 comes with several fixes and enhancements. For example, our content analyses improved, with support for em dashes as punctuation marks in the focus keyphrase field and correctly detecting caps in focus keyphrases with a period.
Among many other things, Yoast SEO recognizes keyphrases with hyphens in the slug. Our word forms feature and the filtering out of function words now works for hyphenated keyphrases. For example, the keyphrase
ex-husband will be recognized in the slug
ex-husband, but also the keyphrase
With the word forms feature, we recognize different grammatical forms of your focus keyphrase in various places in your content. You don’t have to write the same form of your keyphrase each time but can diversify your language by changing forms. This makes you write much more naturally!
WooCommerce SEO 14.9: Global identifiers for product variations + Schema
Working with product variations has always been a hassle. WooCommerce, for instance, doesn’t allow you to add global identifiers for product variations — making it hard to add valuable data for Google to discover. Moreover, there’s no simple way to add structured data to this correctly! In WooCommerce SEO 14.9, we’re fixing this.
WooCommerce SEO now outputs the necessary Schema markup for product variations on individual products. Of course, the plugin already outputs all the product variations Schema markup through the
offers property. Now, it will also output the global identifiers for product variations under the
offers property. In addition, we’ve also added the variations’ SKU and URL to a product’s Schema.
Open the product variations tab in WooCommerce and fill in the global identifier details for each variation in the new Yoast SEO section. The details will be automatically added to the
product structured data generated by WooCommerce SEO.
Learn how to add global identifiers on variable products with WooCommerce SEO.
Removing dates from product Schema
Google sometimes shows dates in your search results, which can help users find the most relevant content. But on your WooCommerce product pages, this doesn’t make much sense — so we remove some metadata from your page to discourage them from showing dates here. So, we now remove the
dateModified attributes on the
ItemPage Schema markup for a product.
Yoast SEO 19.1, Yoast SEO Premium 18.7, and WooCommerce SEO 14.9 are out today. In Premium 18.7, we’ve vastly expanded the possibilities of the new crawl settings we introduced in 18.6. You now have many options to manage crawling on your WordPress site.
Coming up next!
WordCamp Phoenix 2023March 24 - 25, 2023 Team Yoast is at WordCamp Phoenix 2023! Click through to see who we'll be there, what we'll do there and more! See where you can find us next »
Yoast SEO news webinar - March 28, 202328 March 2023 Our head of SEO, Jono Alderson, will keep you up-to-date about everything that happens in the world of SEO and WordPress. All Yoast SEO webinars »
3 Responses to Yoast SEO 19.1 & Premium 18.7: More crawl settings
Hi nice article and very useful features, thanks for sharing!
There is one question that instantly comes to mind:
If i disable a feed a redirect is created, so google will run into that redirect and will fetch let’s say the blogpost instead of the comments of that blogpost. My question is now, should i also disallow the disabled feeds in the robots.txt to avoid the crawler always running into the redirect? Is there a benefit from doing that?
All the best,
Hi Paul, thank you for your question!
I’ve consulted with our product team and this is our advice. In general, we would say no, but with a heavy dose of ‘it depends’. Some considerations:
– Your feeds might be getting linked to / have some degree of ‘equity’; redirecting them consolidates that to the page, but blocking them just ‘removes’ it.
– If it’s blocked, then Google (and other search engines) doesn’t know that it no longer exists, which feels inelegant.
– Redirects are (generally) ‘cheaper’ than full page loads (and 301s are heavily cached), so the efficiency/environment angle is lessened already.
We hope this answers your question, and do let us know if you have more questions!
Ok that clarifies it a bit, so lets keep the redirects for now!
Thanks for the fast response!
All the best,