The ultimate guide to the meta robots tag

If you use meta robots tags on your pages, you can give search engines instructions on how you’d like them to crawl or index parts of your website. This page lists an overview of all the different values you can have in the meta robots tag, what they do, and which search engines support each value.

The different robots meta tag values

The following values (‘parameters’) can be placed on their own, or together in the content attribute of tag (separated by a comma), to control how search engines interact with your page.

Scroll down for an overview of which search engines support which specific parameters.

Allow search engines to add the page to their index, so that it can be discovered by people searching.
Note: This is assumed by default on all pages – you generally don’t need to add this parameter.
Disallow search engines from adding this page to their index, and therefore disallow them from showing it in their results.
Note: Informal messaging from Google suggests that, if a page is set to noindex for a long period of time, it may also be treated as if it were also set to nofollow. The precise mechanics of this are unclear, and it’s unclear whether other search engines behave similarly.
Tells the search engines that it may follow links on the page, to discover other pages.
Note: This is assumed by default on all pages – you generally don’t need to add this parameter.
Tells the search engines robots to ‘endorse’ (pass equity through) any links on the page.
Note: It’s unclear (and inconsistent between search engines) whether this attribute prevents search engines from following links, or just prevents them from assigning any value to those links.
A shortcut for noindex, nofollow.
A shortcut for index, follow.
Note: This is assumed by default on all pages, and does nothing if specified.
Disallow search engines from indexing images on the page.
Note: If images are linked to directly from elsewhere, search engines can still index them, so using an X-Robots-Tag HTTP header is generally a better idea.
Prevents the search engines from showing a cached copy of this page in their search results listings.
Same as noarchive, but only used by MSN/Live.
Prevents the search engines from showing a text or video snippet (i.e., a meta description) of this page in the search results, and prevents them from showing a cached copy of this page in their search results listings.
Note: Snippets may still show an image thumbnail, unless noimageindex is also used.
Prevents search engines from showing translations of the page in their search results.
Tells search engines a date/time after which they should not show it in search results; a ‘timed’ version of noindex.
Note: Must be in RFC850 format (e.g., Monday, 15-Aug-05 15:52:01 UTC).
Prevents the search results snippet from using the page description from the Yandex Directory.
Note: Only supported by Yandex.
Blocks Yahoo from using the description for this page in the Yahoo directory as the snippet for your page in the search results.
Note: Since Yahoo closed its directory this tag is deprecated, but you might come across it once in awhile.

Which search engine supports which robots meta tag values?

This table shows which search engines support which values. Note that the documentation provided by some search engines is sparse, so there are many unknowns.

Robots value Google Yahoo Bing Ask Baidu Yandex
index Y* Y* Y* ? Y Y
noindex Y Y Y ? Y Y
follow Y* Y* Y* ? Y Y
nofollow Y Y Y ? Y Y
none Y ? ? ? N Y
all Y ? ? ? N Y
noimageindex Y N N ? N N
noarchive Y Y Y ? Y Y
nocache N N Y ? N N
nosnippet Y N Y ? N N
notranslate Y N N ? N N
unavailable_after Y N N ? N N
noodp N Y** Y** ? N N
noydir N Y** N ? N N
noyaca N N N N N Y

* Most search engines have no specific documentation for this, but we’re assuming that support for excluding parameters (e.g., nofollow) implies support for the positive equivalent (e.g., follow).
** Whilst the noodp and noydir attributes may still be ‘supported’, these directories no longer exist, and it’s likely that these values do nothing.

Rules for specifics search engines

Sometimes, you might want to provide specific instructions to a specific search engine, but not to others. Or you may want to provide completely different instructions to different search engines.

In these cases, you can change the value of the content attribute to a specific search engine (e.g., GOOGLEBOT or MSNBOT).

Note: Given that search engines will simply ignore instructions which they don’t support or understand, it’s very rare to need to use multiple meta robots tags to set instructions for specific crawlers.

Conflicting parameters, and robots.txt files

It’s important to remember that meta robots tags work differently to instructions in your robots.txt file, and that conflicting rules may cause unexpected behaviors. For example, search engines won’t be able to see your meta tags if the page is blocked via robots.txt.

You should also take care to avoid setting conflicting values in your meta robots tag (such as using both index and noindex parameters) – particularly if you’re setting different rules for different search engines. In cases of conflict, the most restrictive interpretation is usually chosen (i.e., “don’t show” usually beats “show”).

Adding a noindex or nofollow to a post or page is a breeze if you’re on WordPress. Read how to use Yoast SEO to keep a post out of the search results.

The resources from the search engines

The search engines themselves have pages about this subject as well:

And of course there’s always the official robots.txt pages and Danny Sullivan’s big robots meta write up.

Read more: WordPress SEO: The definitive guide to higher rankings for WordPress sites »

Use structured data to win in Google!

  • Make your search results stand out
  • Increase traffic to your website
  • On-demand SEO training by Yoast
  • No technical skills required!
More info

11 Responses to The ultimate guide to the meta robots tag

  1. andrew bogutt
    andrew bogutt  • 1 year ago

    Hi Jono, Nice post, thanks for explaining different robots meta tag values useful for SEO optimization

  2. Noor Alam
    Noor Alam  • 1 year ago

    thank you for clarifying these robots meta tag values

  3. Ahmed
    Ahmed  • 1 year ago

    Great work, this post is very informative. It’s very useful for SEO.

  4. Ratul Roy
    Ratul Roy  • 1 year ago

    Very informative post Jono. It will help me every time when I am going to post something. Bookmarked it. It is certainly the ultimate guide to the meta robots tag.

  5. Rahul Yadav
    Rahul Yadav  • 1 year ago

    Such a great article, deeply explained and thank you for sharing

  6. Andy Roberts
    Andy Roberts  • 1 year ago

    that’s gotta be the most comprehensive post about robot tags i’ve ever read!!! :-)

  7. Havi
    Havi  • 1 year ago

    Thank you for the article! I just have a question. Under the Advanced section of Yoast, Meta Robots Advanced is set to Site-wide default: None
    Is this the same as:
    A shortcut for noindex, nofollow?

    (I do not see the meta robots on the page code anymore)

    Thank you!

    • انجام پروژه دانشجویی
      انجام پروژه دانشجویی  • 1 year ago

      Thank you for this question, Indeed I have this question in my mind also

    • Jono Alderson

      Ah, good question – there’s definitely some ambiguity there!

      That setting means, “don’t do anything differently from my default settings on this post”, not “set a value of ‘none'”. So, if your site-wide default is set to index/follow (or if you’ve not set any specific or restrictive rules), then it should be equivalent to ‘index/follow’,

      We’ll get that updated / make it clearer ASAP :)

      • Sarah
        Sarah  • 1 year ago

        Excellent, this is very informative and helps. Thanks

      • Salma Khatun
        Salma Khatun  • 1 year ago