How to get Google to crawl your site faster
Yet another question out of my referrers to answer: “how can i get google to crawl my site faster“. There are two possible reasons why Google is slow in spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The more quality links it finds, the higher it values your site, the more your site is crawled.
Another reason why Google is slow in spidering your site might be that your server is slow… Google actually indicates this to you, and I shortly addressed this in my post on how to use Google Webmaster Tools. If this is the case, Google shows you this in the “Crawl rate” overview of the Diagnostic tab:
My advice: set it to “Faster”, but don’t stop with that: make your server react faster, either by upgrading your hosting, your hardware, or whatever is needed. Chances are that when Google is suffering long load times, your users are as well.
Now if both of these aren’t right for you, Google has found many great links to your site and it’s not telling you it would like to crawl faster, there is one solution to make sure it crawls your new articles or pages even faster: sitemaps.
I don’t use them on this site myself, as I want to know how Google crawls my site without me interfering in the process. However, if you think you have something to gain by Google crawling your new pages faster, an XML sitemap is a plausible solution.
Do consider though that you’re throwing data away, you’ll never know if and when Google would have found a page by itself if you hadn’t put it in the sitemap.