Use robots.txt and .htaccess rules to control crawling and indexing
Control crawlability right from within Yoast SEO
Yoast SEO contains elaborate meta robots tags settings that allow you to decide where search engines go on your website. In most cases, these settings are enough to gain control of your site’s crawlability. But, we also have a solution for specific technical challenges that need access to a robots.txt or .htaccess file.
Yoast SEO gives you full control with a file editor
- Control your crawlability
- Fix specific technical challenges
- Powerful, but easy to use
Yoast SEO is set up to make crawlability easy out of the box. But, what if you have some specific technical challenges that need to be fixed with a robots.txt or .htaccess file? We cover those cases too! With the Yoast SEO tools, you can edit robots.txt and .htaccess files. By doing that, you can gain even more control over how your website gets crawled!
Why do I need to edit robots.txt or .htaccess files?
While we provide this option in the Yoast SEO plugin, we recommend you use it only if you know what you are doing. In most cases, it is enough to block search engines from particular pages by adding meta robots tags or robots HTTP headers. However, we recognize that some cases may face complex technical challenges that require the editing of the robots.txt file. For those cases, we made editing robots.txt and .htaccess files possible.
Rank with your high-quality pages
On large or complex sites, search engines may sometimes limit the number of URLs they’re willing to crawl and consume. This limit is sometimes known as “crawl budget”. Based on your website’s size and reputation, they may decide to allocate more (or less) resources to crawl your pages. If you have some pages on your site that need some SEO work, you might want to block crawlers from accessing them. In that way, you give crawlers more room to access the pages you want to rank with.
Block access to the pages you don’t want crawlers to access
There are also pages on your website that do not contribute to your ranking, but they are necessary for the functioning of your website. For example, you may have internal search results, admin pages, account pages, shopping carts, etc. You can block crawlers from accessing these pages with the robots.txt file. Doing that may improve your site’s technical SEO, and contribute to better rankings!
Manage server settings
Being able to edit your .htaccess file lets you control some aspects of how your server responds to requests. You can manage redirects, file handling options, and much more. This can be helpful in advance setups, or where it’s difficult to make changes on the server itself.
How does it work?
Depending on the server you use, you can edit your robots.txt or the .htaccess files in Yoast SEO > Tools > File editor.
For more information see:
Let Yoast SEO help you win the search rankings
Discover all the features of the Yoast SEO for WordPress plugin.