How to get Google to crawl your site faster

Yet another question out of my referrers to answer: “how can i get google to crawl my site faster“. There are two possible reasons why Google is slow in spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The more quality links it finds, the higher it values your site, the more your site is crawled.

Another reason why Google is slow in spidering your site might be that your server is slow… Google actually indicates this to you, and I shortly addressed this in my post on how to use Google Webmaster Tools. If this is the case, Google shows you this in the “Crawl rate” overview of the Diagnostic tab:

Crawl faster

My advice: set it to “Faster”, but don’t stop with that: make your server react faster, either by upgrading your hosting, your hardware, or whatever is needed. Chances are that when Google is suffering long load times, your users are as well.

Now if both of these aren’t right for you, Google has found many great links to your site and it’s not telling you it would like to crawl faster, there is one solution to make sure it crawls your new articles or pages even faster: sitemaps.

I don’t use them on this site myself, as I want to know how Google crawls my site without me interfering in the process. However, if you think you have something to gain by Google crawling your new pages faster, an XML sitemap is a plausible solution.

Do consider though that you’re throwing data away, you’ll never know if and when Google would have found a page by itself if you hadn’t put it in the sitemap.

Tags: ,


Yoast.com runs on the Genesis Framework

Genesis theme frameworkThe Genesis Framework empowers you to quickly and easily build incredible websites with WordPress. Whether you're a novice or advanced developer, Genesis provides you with the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.

Read our Genesis review or get Genesis now!

76 Responses

  1. JulienBy Julien on 16 April, 2007

    Hum, according to you, what is for Google a fast enough site?

    Mine as an average of 350 ms, for a minimum of 330ms and a maximum of 380ms as of Google Webmester Tools… And I am not bbeeing proposed the “Faster” Option…

  2. GerbenBy Gerben on 16 April, 2007

    Well, that’s someting i would like to know to..

    I’ve got a max of 932, a average of 484 and a minimum of 283. Google crawl’s the complete site every 2 weeks. (max crawled pages is 8193, 96% of the entire site!).

    No, i do know that Google’s Spam reporter (MattCutts) has a max speed of 1769 so I always thought my speed was good, but as julien shows above, mine isn’t that fast at all?

  3. Joost de ValkBy Joost de Valk on 16 April, 2007

    Julien’s site is FAST. And your average sounds pretty good to me too Gerben, so I wouldn’t worry to much about it!

  4. JulienBy Julien on 16 April, 2007

    So, if I get your point, Google doesn’t propose me the “Faster” Option because my website doesn’t have enough “Quality links”… Hum, I will have to work for this!

    According to you, what is a “enough (quality) links”?

  5. GerbenBy Gerben on 16 April, 2007

    Thanx joost,

    But for you’re idea, it’s going to move to a dedicated server with 2 other sites…so it should be faster in the future too….

    And quality links…damn…a hell of a job! ;-)

  6. Joost de ValkBy Joost de Valk on 16 April, 2007

    Julien: no you’re not getting me right I’m afraid: you’ve got nothing, really nothing, to worry about if you look at the speed of your site crawl.

    Google will only start crawling faster (in this case meaning: more regularly) if you get more links, but it won’t give you any influence on it.

    Gerben: Good stuff :) and linkbuilding takes time, yes :)

  7. JulienBy Julien on 17 April, 2007

    Ok, I see… but the fact his that I sometime have the feeling that Google forgets so manu of the paages on my site!

    Anyway, I’ll try to linkbait too… for deep links mainly!

    Thanks for your help!

  8. MicrodesignBy Microdesign on 22 April, 2007

    cool, never knew you could setup speeds for google, really going to check it out!

  9. Joost de ValkBy Joost de Valk on 22 April, 2007

    Microdesign: good :) have fun with it!

  10. ronBy ron on 17 May, 2007

    I have a max daily crawl at 49133 and still no faster option. how fast can it be?

  11. jaiBy jai on 3 June, 2007

    Its really good to see the comments and the topic that is discussed in this website. It has lots of quality information available. I love to come agian and share my views and get information. Thanks webmaster!

  12. jaiBy jai on 3 June, 2007

    i have two domains hosted on a shared server one has maximum-13066 average-879 minimum-108 and the other one max-530 average-189 minimum-58. What could be the reason for such diffrence? Is it due to the links both website has or its a problem due to server?

  13. Joost de ValkBy Joost de Valk on 3 June, 2007

    @jai: that’s most probably due to links yes.
    @ron: that’s a good thing, not a bad one :)

  14. dwikristiantoBy dwikristianto on 5 September, 2007

    ( Maximum Average Minimum)

    Number of pages crawled per day
    507 195 21
    Number of kilobytes downloaded per day
    4029 1251 38
    Time spent downloading a page (in milliseconds)
    14941 2986 197

  15. ConnieTalkBy ConnieTalk on 22 September, 2007

    How do you avoid the catch-22, though? When you ask for links, or even exchanges, people are afraid it will lower THEIR ranking in Google.

    Take, for example, this e-mail exchange I had with a site owner similar to my site’s genre:

    Hello Connie,

    I’m afraid we can’t engage in any reciprocal link programs anymore, as it hurts our search result standings in Google. Google has announced that it will be penalizing sites with reciprocal links, due to the number of spammer and unethical sites which do it.

    Thanks,
    John Doe
    —————-
    Connie wrote:

    Bill,

    Where did you hear about Google penalizing sites for exchanging reciprocal links? They actually encourage it – they suggest under “Site Promotion” exchanging links with high-quality pages. I believe the penalization you are referring to applies only to sites that specialize solely in link exchanges?

    We don’t have a link exchange program – we just really want to be listed on your site!! We do everything we can to raise awareness about Smoker’s Rights. Please reconsider, or let us know what we can do for you to get a mention. We love your site, and there’s nothing we won’t do to be a part of it!!

    Sincerely,

    Connie
    —————————-
    John wrote:

    I’m sorry Connie, but the days of cross-linking are over. Please read this page in Google’s guidelines, specifically the part about “link to me and I’ll link to you” – this is now disallowed by Google. It happened on another website I have – I added 3 lnks to other sites who link to me, JUST 3!, and I dropped 8 places in search results:

    http://www.google.com/support/webmasters/bin/answer.py?answer=66356&query=link+exchange&topic=&type” TARGET=”_BLANK

  16. Surya Prakash LoonkerBy Surya Prakash Loonker on 24 September, 2007

    I have done everything. Submitted to free search engine submissions, paid search engine submissions, even submitted on google itself. its been 4 days, i saw it crawling on my site once. But its index does not seem to be updated as when i do searches, old pages are showing up and still my site is not showing up for several keywords even upto 1,000 searches. Any sure shot way of getting google to index quickly.

  17. David SalmonBy David Salmon on 24 September, 2007

    How come there aren’t any more articles on this topic which even give this much advise. Is SEO so secretive that you really have to buy something only to get proper advise, otherwise its only how they can help you do it, if you buy this eBook or take their services. Man, this is tough.

  18. Joost de ValkBy Joost de Valk on 24 September, 2007

    It’s not that hard, you guys are just thinking too hard :) Get more decent links and decent content, and you’re there :)

  19. ConnieTalkBy ConnieTalk on 24 September, 2007

    Well, I’ve been at this since November of ’06 on my site…only in the past few days has Google been showing me some love. I agree with David – SEO is so secretive and no one wants to share info without selling something! If anyone has good advice to share on this, please do!

    Also, Google News keeps using the wrong image with my stories – no matter how I rearrange my pages, it always uses my navigation bar (an image map) as the image that goes with the news article, instead of the corresponding image. Anyone know how to fix this?

  20. Gerben BouwhuisBy Gerben Bouwhuis on 24 September, 2007

    Hi Connie,

    Perhaps because it’s one of the first images in your source code?

    Gerben

  21. ConnieTalkBy ConnieTalk on 24 September, 2007

    Yes, but how am I supposed to avoid that? I even removed the sidebar completely from the news pages now, and now it’s pulling an image in the header of the page instead of from the article…

  22. davidBy david on 9 November, 2007

    I wouldn’t worry about how often google crawls your site. It was stated before, but make sure your sitemap is up-to-date. My site is crawled once every 10 days, and yet the content still comes up in google search.

  23. SirCommyBy SirCommy on 18 December, 2007

    I, for one, am using the Google Webmasters Tools, respectively, using sitemaps. I appreciate your nice technique of checking whether Google or other Search Engines crawls my website without sitemap(s) Joost, but I have too many websites to handle and it would take too much time. My sitemaps are doing a great job, plus, my favourite feature is seeing the TOP20 of searched keywords in google, whether they are my keywords or not. Also, how many times they were clicked. Very helpful on choosing more searched keywords where your site comes up.
    Cheers

  24. MariuszBy Mariusz on 30 January, 2008

    I taked option ‘faster’ in Google Webmasters Tools, but for changes I have waiting 90 days…thanks

  25. baliBy bali on 14 March, 2008

    Thanks. Ill try this tips

  26. fulldealsBy fulldeals on 14 March, 2008

    How come google stop indexing my page. the cache’s date are different date when i do cache:sitename
    it’s wierd… i don’t see my new content indexed..
    anyone know why?

  27. JassiBy Jassi on 14 April, 2008

    My website is stop by google reason stopbadware

    What should I do for recover it, Can you suggest me when google will cral my site again, what process to cral website again please tell me

  28. asianloungeBy asianlounge on 16 April, 2008

    how come using cache:http://asianlounge.com doesn’t show up :-/ but it’s still being index.. please advised?

  29. ShambooBy Shamboo on 25 April, 2008

    my web site has 600pages. Google is not crawling my website. in webmaster it say Error since i am providing a html page. please let me know how to solve this problem. Should i give a XML.

    let me know how to solve this issue.

  30. subsubBy subsub on 26 April, 2008
  31. EmilyBy Emily on 26 June, 2008

    Shamboo, the webmaster tools is there to submit an XML map, not an actual html map. That’s why you’re getting the error.

  32. AlldenverlimoBy Alldenverlimo on 31 July, 2008

    Nice Information regarding the Google crawler fast..Thanks
    I have also same problem and wait for Google crawler to crawl my site.
    I would like to more in detail about this issue and tips.

    Thanks agian.

  33. Cyrin ShellyBy Cyrin Shelly on 9 August, 2008

    Good and informative

  34. sem servicesBy sem services on 23 September, 2008

    i have change google crawl rate but haven’t get any benefit.

  35. RoadshowBy Roadshow on 24 September, 2008

    If you are using Google Webmaster Tools to tell you when your pages are being crawled, you may be assuming something that is inaccurate. In the overview, it tells you the last time your ‘homepage’ was crawled, not any other pages on your site. Other pages may be being indexed but your homepage is not. If you click on ‘more informaton’ you’ll see a more accurate representation of how often you are getting crawled. This is a bit confusing for some, and I wish Google would just pony up and list the last date each page was crawled to make life easier, but they haven’t done that to date.

  36. Matt RichesBy Matt Riches on 4 October, 2008

    Hi,

    I thought I would try and add a xml sitemap to a few of the sites I manage. They seem to be working ok but you have to update the sitemap whenever you add a new page (unless I am missing something)?

    Best Regards,
    Matt Riches
    http://www.richescreative.co.uk

  37. Designer JoeBy Designer Joe on 14 October, 2008

    Hey is posting here considered a quality link…? Cause I’m confused at the definition on the word “quality”. Hope it is and here’s my link http://www.4887777.com
    Linking to sites = 1

  38. Atv four wheelerBy Atv four wheeler on 17 October, 2008

    Thanks….. I got more information about How google crawl on my site and this its so much important because a site comes on top behalf of its crawling rate and visitors. I want more information about it.

  39. baia mareBy baia mare on 27 October, 2008

    i checked the crawl faster option, i have upload a xml sitemap, but still Google-bot crawls my site very very slow.
    what to do? and another problem is in my site content, it show me that i don’t have the word “web” in my site (google webmaster). But still the words “web” and “design” are the most used keywords. And keyword density is ok, under 6%. Any ideeas? thanx

  40. RayBy Ray on 5 November, 2008

    Hello,

    I too am having a hard time getting my site crawled. My webmaster switched us over to html from htm and for almost 2 weeks now my site hasn’t picked up the pages. My site is 2 years old. Any ideas?

    Ray

  41. RoboPapaBy RoboPapa on 20 November, 2008

    Hey bro,

    I’ve tried every trick in the book to increase my google crawl rate but google just aint indexing my site – My robotics website

    I’ve done the following:
    1. Added XML Sitemap and submitted to Google
    2. Modified robots.txt (to disclude some links to be indexed but still aint workin)
    3. Modified my linking structure

    I see Googlebot visiting my pages quite frequently but the search results and the pages it indexes remain the same.

    Need advice.

    Thanks !

  42. RayBy Ray on 20 November, 2008

    Hello RoboPapa,

    We recently changed over from html to PHP. Google took quite awhile to index the site. I did alot of work on the site while waiting and research. Check your titles and descriptions. Are they the same? They should all be different. Also if you did a 301 direct Google takes longer to pick it up. You might want to check site:www (your site).com to see if it is indexed but not showing. To see it is showing in the serps try site:www(your site).com/*. You should also submit to a social website or two and try to link to those pages not just the homepage. Google will index your site, just keep working on the site. Hope this helps.

  43. NeedsHelp!By NeedsHelp! on 5 May, 2009

    Dear Joost,

    I have a site, and hardly ever gets any visiters. I have JUST gotten it indexed in Google but only for extremely specific keywords. I want to know what I’m doing wrong! Could you maybe visit my site and tell me? You were talking about quality links, could you maybe link to me? As this is a quality site? :)

    My website is about World of Warcraft

    Some help would be much appreciated! Thank you!

  44. RoshanBy Roshan on 11 May, 2009

    Can you tell me why is there no activity in the graph that shows Google’s activity in the past 90 days in recent times of my site?

  45. Princess KikiBy Princess Kiki on 2 June, 2009

    this is good advice thanks

  46. michaelBy michael on 18 July, 2009

    I have over 7 million pages on my site, and Google is only showing 1300.
    I have backlinks, news articles, etc.

    Hmmm… how do I get this to increase??

    • JulianBy Julian on 10 November, 2009

      7 million give your head a shake.

  47. CraigBy Craig on 18 July, 2009

    I just verified it on google,bing and yahoo but not sure if they will index me or not. I had alot of problems with the verification tool with google but it finally went through.

  48. CraigBy Craig on 26 July, 2009

    Took me 2 days on google to get indexed without any hard work involved. I was surprised.

  49. SarahBy Sarah on 30 July, 2009

    Funny…it took me about 2 days also…but now i want the bot to come back because I have new stuffs! Same sort of advise for that? To get it to come back?

  50. CraigBy Craig on 31 July, 2009

    Nice, I have about 4 sites added now it seems like once the googlebot has explored your site it will come back pretty often without you having to do anything I think it comes by some of my sites every few days. I just started up a forum site so going to have to index that now I hope it goes aswel as the rest.

  51. Tech NukemBy Tech Nukem on 3 August, 2009

    My site http://technukem.blogspot.com was last crawled by google on 27/06/09 i want it to be back as i update my blog everyday but its not happening… what can be done for that. my site is added in webmaster also.

  52. JackBy Jack on 24 August, 2009

    Getting google to crawl your brand new site can be a mission, I’m currently doing it for 3 – 4 sites and there will still be more to come!

    Hopefully my portfolio will get crawled more often and hence my other sites will be.

    Good luck and good post!

  53. Ghulam HBy Ghulam H on 7 September, 2009

    Google recommends Normal Rate of crawling and that could be helpful for our website. If we set google rate on faster and our site opens slow due to server problem or messy coding it will create bad impact on bots.
    Thanks,
    Ghulam H

  54. Yousef GhazawiBy Yousef Ghazawi on 12 September, 2009

    Hi,

    Thanks for this great in formations, but the problem is I tried all of that it still not working. :(

  55. Jack BrownBy Jack Brown on 14 September, 2009

    Thanks, good and informative.

  56. Arrow SEOBy Arrow SEO on 17 September, 2009

    Thanks for the advice. I’ll give it a try.

  57. sem infotechBy sem infotech on 17 September, 2009

    Thanks for a nice article. It’s really helpful for new webmaster users.

  58. MoneyBy Money on 19 September, 2009

    I can do this in my webmaster tools. Its grayed out and says google has assigned a custom crawl rate for this website. Did I do something wrong? Anyone else have this issue?

  59. MEPBy MEP on 27 September, 2009

    I too have been trying to get the Google crawler to visit my new site. http://www.zavasites.com. It is still not indexing. Are there any other ways to speed up the crawlers process? I’ve created a site map and considered the other suggestions. Thanks.

  60. Joey CollinsBy Joey Collins on 30 September, 2009

    Nice, I will have to check this out… now if Bing could crawl my site more then once a month I’d be in business.

    Canada Card World

  61. WowZila FreebiesBy WowZila Freebies on 30 September, 2009

    Microsoft don’t crawl my site often probably once per month too. Yahoo and Google crawl often
    and a few others. I started doing adsense and ever since that Google is never away from my site.
    My website is http://www.wowzila.com we have a great freebies section!!

  62. Gotowanie w kuchniBy Gotowanie w kuchni on 5 October, 2009

    My page przepisy kulinarne google crawls approx. 30 pages a day (out of 100 pages in total). I works 1 month so I think it’s a good score.

  63. SellbitsBy Sellbits on 13 October, 2009

    If you want google to index and crawl your site faster, you have to get QUALITY backlinks!

    Here are a few tips that hopefully will help some of you.
    1. Get lots of quality backlinks. This won’t happen over night and you need to work at it.
    2. Write articles and submit to directories who have a decent PR ranking.
    3. If you can create press releases and submit those as well!
    4. Post on forums and blogs and use your website as a signature (again quality forums)
    5. Have well formatted XML sitemaps
    6. Join social networks such as facebook and twitter
    7. submit your site to all the major search engines
    7. Stick with this constantly for 6 months and you will see dramatic results. The longer you do this the better your site will become in the eyes of google.

    Remember everyone, Your site will never be at the top if you don’t work towards it.

  64. driverBy driver on 22 October, 2009

    i just launched my site http://www.traindrivertrainee.com 2 days ago, it was crawled within a couple of hours according to my visitor logs, however im yet to be indexed. How long does it take after being crawled to be indexed? Thanks

  65. dimaksBy dimaks on 24 October, 2009

    I am having the same issue. I just redesigned my blog a few weeks ago from almost a year of being on hiatus. well, Google crawls the site after i publish a post but when i do edits, it seems to take a very long while to re-visit.

  66. Bill ColemanBy Bill Coleman on 1 November, 2009

    A couple of months ago I completely updated my site which google has now crawled several
    times and things are looking good, apart from the fact that a number of now defunct old pages
    are still listed by google. This is a problem for me as some people are getting the 404 code
    and assuming that we are not a quality site or even dont exist anymore. Would anyone know
    how to get google to lose these old pages (http://www.spiritualquest.co.uk).
    tks

  67. sdmodule.comBy sdmodule.com on 7 November, 2009

    I’m having a similar problem at sdmodule.com. An initial index followed by no updates. Any advice on how to get the site re-indexed?

  68. dimaksBy dimaks on 11 November, 2009

    sdmodule,
    did you try using pingomatic to send pings to google bot? just a thought, it might help.

  69. Kirstin BaliBy Kirstin Bali on 11 November, 2009

    Well i have try to set the crawl faster. But after a couple of weeks, Google reset it to back to the normal. Do anyone know, why it can be like that?

  70. chris powellBy chris powell on 13 November, 2009

    my site has been on google for about 4 months and has not been crawled, i have a sitemap,
    keywords, metatags
    can you help me please?

  71. Steve BakerBy Steve Baker on 23 November, 2009

    I recently launched http://sitedirectorylinks.com and using Google’s webmaster tool and an XML sitemap, I was able to get crawled. My question is why it shows 1500+ urls submitted but only 9 have been included in the index? No matter how many times I resubmit, I can’t get the indexed number of pages to increase. Any information would be appreciated.

  72. Dani GorgonBy Dani Gorgon on 3 December, 2009

    How can I find out when was the last time google sent its crawlers to our website?

  73. RiyuBy Riyu on 7 December, 2009

    my site is not getting crawled since 2 months. I have applied all meta tags.

  74. Graphic Design DelhiBy Graphic Design Delhi on 7 December, 2009

    I have sever links to my website , and they are also shown in Crawling errors but due to some file extenstion problem a 404 error is shown in webmaster. Now that i have rectified the error still those links are still 404 error plz suggest.

Trackbacks