Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss what the possibilities are and how this might be a better …Read: "Playing with the X-Robots-Tag HTTP header"
Crawl directives archives
In 2015, Google Search Console already started to actively warn webmasters not to block CSS and JS files. In 2014, we told you the same thing: don’t block CSS and JS files. We feel the need to repeat this message now. In this post, we’ll explain why you shouldn’t block these specific files from Googlebot. Why you …Read: "Don’t block CSS and JS files"
What is a crawl budget? Crawl budget is the number of pages Google will crawl on your site on any given day. This number varies slightly from day to day, but overall it’s relatively stable. The number of pages Google crawls, your “budget”, is generally determined by the size of your site, the “health” of your …Read: "Crawl budget optimization"
The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!Read: "robots.txt: the ultimate guide"
The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!Read: "rel=canonical: the ultimate guide"
The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do. Over the years, especially Google changed …Read: "WordPress robots.txt example for great SEO"
A month ago Google introduced its Panda 4.0 update. Over the last few weeks we’ve been able to “fix” a couple of sites that got hit in it. These sites both lost more than 50% of their search traffic in that update. When they returned, their previous position in the search results came back. Sounds too good to be …Read: "Google Panda 4, and blocking your CSS & JS"
With the meta robots tag you can control what search engine spiders do on your site. Learn what it does right here!Read: "The ultimate guide to the meta robots tag"
Browse other categories
AnalyticsAll about analytical tools and how to use them »
Content SEOContent is King! Read about it »
eCommerceLearn how to optimize your shop »
SEO basicsSEO made easy for you »
Social MediaOur insights on how to use them best »
Technical SEOTechnically optimize your site to the fullest »
User eXperience (UX)How to give users an awesome experience »
WordPressRead about WordPress plugins, updates and SEO »