Crawl directives archives

Recent Crawl directives articles



SEO for a new website: the very first things to do

5 August 2021 | 8 Comments Marieke van de Rakt

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. …

Read: "SEO for a new website: the very first things to do"
SEO for a new website: the very first things to do

Preventing your site from being indexed, the right way

19 March 2021 | 18 Comments Joost de Valk

We’ve said it way back when, but we’ll repeat it: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. Do you know why it keeps amazing us? Because …

Read: "Preventing your site from being indexed, the right way"
Preventing your site from being indexed, the right way


What are sponsored, nofollow and ugc links, and why use them?

24 June 2020 | 32 Comments Willemien Hallebeek

Links are an important part of SEO. Without links, Google (or other search engines) may not discover your pages, or might not think that they’re important. Sometimes, though, you might want Google not to follow a link. Or you might want to tell them a particular is sponsored, or added to your page by a …

Read: "What are sponsored, nofollow and ugc links, and why use them?"
What are sponsored, nofollow and ugc links, and why use them?

Noindex a post or page in WordPress, the easy way!

22 June 2020 | 18 Comments Joost de Valk

Some posts and pages should not show up in search results. To make sure they don’t show up, you should tell search engines to exclude them. You do this with a meta robots noindex tag. Setting a page to noindex makes sure search engines never show it in their results. Here, we’ll explain how easy it …

Read: "Noindex a post or page in WordPress, the easy way!"
Noindex a post or page in WordPress, the easy way!


Help, I’ve accidentally noindexed a post. What to do?

9 January 2020 | 17 Comments Willemien Hallebeek

It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or …

Read: "Help, I’ve accidentally noindexed a post. What to do?"
Help, I’ve accidentally noindexed a post. What to do?

WordPress robots.txt: Best-practice example for SEO

7 November 2019 Jono Alderson

Your robots.txt file is a powerful tool when working on a website’s SEO – but you should handle it with care. It allows you to deny search engines access to different files and folders, but often that’s not the best way to optimize your site. Here, we’ll explain how we think site owners should use their …

Read: "WordPress robots.txt: Best-practice example for SEO"
WordPress robots.txt: Best-practice example for SEO