The ultimate guide to robots.txt
The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!
Read: "The ultimate guide to robots.txt"
Crawl directives
There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:
- tell a search engine to not crawl a page at all;
- not to use a page in its index after it has crawled it;
- whether to follow or not to follow links on that page;
- a lot of “minor” directives.
We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.
The ultimate guide to robots.txt »
The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!
Must read articles about Crawl directives
-
Preventing your site from being indexed, the right way »
Trying to prevent indexing of your site by using robots.txt is a no-go, use X-Robots-Tag or a meta robots tag instead! Here's why.
-
rel=canonical: the ultimate guide »
The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!
-
How to keep your page out of the search results »
Want to keep a page out of the search results? Ask yourself if it should be on your site anyways. If it should, use a robots meta tag to prevent it from being indexed.
Recent Crawl directives articles
How to keep your page out of the search results
If you want to keep your page out of the search results, there are a number of things you can do. Most of ’em are not hard and you can implement these without a ton of technical knowledge. If you can check a box, your content management system will probably have an option for that. …
Read: "How to keep your page out of the search results"
hreflang: the ultimate guide
This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.
Read: "hreflang: the ultimate guide"
Which pages should I noindex or nofollow?
Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there, as glue for other pages or simply because regulations require them to be accessible on your website. If you regularly read our blog, you’ll know …
Read: "Which pages should I noindex or nofollow?"
Block your site’s search result pages
Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal …
Read: "Block your site’s search result pages"
The ultimate guide to the meta robots tag
With the meta robots tag you can control what search engine spiders do on your site. Learn what it does right here!
Read: "The ultimate guide to the meta robots tag"
Media / attachment URL: what to do with them?
In our major Yoast SEO 7.0 update, there was a bug concerning attachment URLs. We quickly resolved the bug, but some people have suffered anyhow (because they updated before our patch). This post serves both as a warning and an apology. We want to ask all of you to check whether your settings for the …
Read: "Media / attachment URL: what to do with them?"
What are crawl errors?
Crawl errors occur when a search engine tries to reach a page on your website but fails at it. Let’s shed some more light on crawling first. Crawling is the process where a search engine tries to visit every page of your website via a bot. A search engine bot finds a link to your website and starts …
Read: "What are crawl errors?"
Yoast SEO & Ryte: Checking your site’s indexability
Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …
Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Pagination & SEO: best practices
Paginated archives have long been a topic of discussion in the SEO community. Over time, best practices for optimization have evolved, and we now have pretty clear definitions. This post explains what these best practices are. It’s good to know that Yoast SEO applies all these rules to every archive with pagination. Indicate that an …
Read: "Pagination & SEO: best practices"
Crawl directives
There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:
- tell a search engine to not crawl a page at all;
- not to use a page in its index after it has crawled it;
- whether to follow or not to follow links on that page;
- a lot of “minor” directives.
We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.