google crawl speed

How to get Google to crawl your site faster

How to get Google to crawl your site faster

October 26th, 2016 – 5 Comments

A question that often pops up for sites that are chasing “hot topics” is: how to get Google to crawl our site faster? This could be either because you have a big site and want to see changes faster or because you’re optimizing a news site. There are a few things you can do to improve your crawl speed. We’ll cover them in this post.

There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The other reasons are technical: it has too much to crawl on your site, your site is too slow, or it’s encountering too many errors.

Your site doesn’t have enough authority

When your site doesn’t have enough quality inbound links, Google will not crawl your site very quickly. You can imagine it doesn’t necessarily want to spend much time on a site that it doesn’t deem important. When your site is brand new, this will be the case.

So, when you need more crawl action on your site, start doing some link building. We’ve written about that topic before, I’d suggest reading these three posts as I won’t go into it any further here:

  1. Link building from a holistic SEO perspective
  2. Link building, what not to do
  3. 6 steps to a successful link building strategy

If the above three posts are not enough for you, there’s more in our link building archive.

Technical reasons for slow crawling

The technical reasons for Google to crawl your site slowly can be divided into three groups: your site is too slow, you have too many errors, or you have too many URLs.

Your server is slow

The main reason why we see Google crawling sites slowly, is when the site itself is actually slow. Google actually indicates this to you when this is the case, on the “crawl rate” page of Google Search Console. If this is the case, Google might show you a “faster” option:

Crawl faster

Our advice: set it to “Faster”, but don’t stop with that: make your server respond faster, either by upgrading your hosting, or improving your site’s caching. Chances are that when Google is suffering long load times, your users are as well.

Optimize your site for Google News! Create XML News Sitemaps, editors’ picks RSS feeds and more with Yoast News SEO! »

News SEO for WordPress pluginBuy now » Info

Too many errors on your site

If you have a lot of errors on your site for Google, Google will start crawling slowly too. To speed up the crawl process, fix those errors. Simply 301 redirect those erroring pages to proper URLs on your site.

If you don’t know where to find those errors: use the Google Search Console integration in Yoast SEO. Or log into Google Search Console. If you have access to your site’s access logs, you can also look at those, preferably with a tool like Screaming Frog’s Log file analyzer.

To prevent your site from being crawled slowly, it’s important that you regularly look at your site’s errors and fix them. We have a more extensive article on fixing 404 errors.

When your site is down too much, Google will slow down incredibly hard. If this happens because of your hosting, switch hosting. Make sure to check on your site’s uptime with a tool like Pingdom.

Too many URLs

If you simply have too many URLs on your site, Google might crawl a lot but it will never be enough. This can happen because of faceted search navigation for instance, or another system on your site that simply generates too many URLs. To figure out whether this is the case for you, it’s always wise to regularly crawl your own site. You can either do that manually with Screaming Frog’s SEO spider, or with a tool like OnPage.org.

Tips to increase your crawl speed

A couple of simple tips to increase your site’s crawl speed:

  • Using the methods above, find and fix all the errors.
  • Make sure your site is fast.
  • Add an XML sitemap to your site and submit it to the search engines.
  • If all of that fails to improve your crawl speed, start link building!

Specifically for news sites, if you want to be indexed faster, you should also look at our News SEO plugin. When you’re in (or want to be in) Google News, it can help make sure you meet all the technical requirements.

Read more: ‘Site speed: tools and suggestions’ »


5 Responses to How to get Google to crawl your site faster

  1. Smitha
    By Smitha on 28 October, 2016

    Hi, I also noticed nowadays google crawling frequencies has been changed, I think so, because my website didn’t crawling frequently from past 2 months, Also i have used webmaster tools to recrawl but not! Dono why, Hope the above will work.

  2. Toby Balsiger
    By Toby Balsiger on 28 October, 2016

    Getting rid of too many urls really helped improve my rankings!

  3. ashish
    By ashish on 27 October, 2016

    Hi, from past one month , i noticed google indexing of webpages is not increasing , earlier it was 17 and after 1 month its still 17 , under my sitemap it shows 34 pages submitted and only 17 indexed.

    Also i couldn’t find that crawl rate option to make google crawl faster, normal or slower .

  4. Wagga Web Design
    By Wagga Web Design on 27 October, 2016

    What sort of target should we be aiming at for publish to crawl time?

  5. Amy
    By Amy on 27 October, 2016

    Thank you for the great information


Check out our must read articles about Analytics