XML Sitemap in the WordPress SEO plugin

There have been some questions after the release last weekend of the new XML sitemaps implementation in WordPress SEO. Let me try to address most of those questions in one post as well as explain the ideas behind it.

The basic idea behind the rebuild of the XML sitemap functionality was simple: XML sitemaps needed to be more scalable, have a better API and cause less issues. Most of the issues were caused by two causes:

  1. the fact that I was writing XML sitemaps to disk as static files;
  2. the fact that I tried to generate one huge sitemap to catch everything.

Both of these were not really design decisions, just how it “turned out” when I had built the functionality and they turned out to be stupid. So, with the help of the incredibly talented Jon Cave, a WordPress core dev from the UK, the XML sitemap functionality has been rebuilt. It now has an index sitemap file that points to several sitemaps, one for each post type and taxonomy.

If you have more than 1,000 posts in a post type, for instance more than 1,000 posts, it will automatically start splitting these sitemaps up into 1,000 posts sitemaps, so as to reduce the load time per sitemap. These sitemaps are generated in a completely new way, on the fly, without caching them to disk. This means there won’t be a delay anymore (caused by the generation process) when you publish a new post.

The sitemap index file can always be found at sitemap_index.xml, a link to that can be found from the new XML sitemaps menu.

The new code also means there’s a great new API for XML sitemaps, which will first be used in the Google News module. Documentation for that is coming. Because of a new filter in the code, NextGen Gallery will now also be able to add its images into the sitemap, so all of your images can be found by Google.

FAQ regarding the new XML Sitemaps functionality

  • Why do I get a 404 when opening the sitemap_index.xml file?
    You probably have W3 Total Cache active, that is preventing 404 errors for static files to go through to WordPress. Update to the latest version of W3 Total Cache or add, under the browser cache page, this exemption:

    sitemap(_index|[0-9]+)?\.xml
  • Why do I get an error saying “not a valid feed template”?
    You’re probably using a Woothemes tumble-style theme,  check with Woothemes about an update, they’re using a very general rule for XML sitemaps in their old code. Or, if you’re a developer, go and find a file that’ll probably be called /includes/tumblog/theme-tumblog.phpand fix / remove the line that looks like this:

    '(.+).xml' => 'index.php?feed='. $wp_rewrite->preg_index(1)
  • Why does my sitemap show as type “images” in GWT?
    Because images are contained within the sitemap, and Google Webmaster Tools then immediately shows it as type “images”. There is no “midex” type in Google Webmaster Tools at the moment.
  • Which sitemap should I submit to Google Webmaster Tools?
    The index file, sitemap_index.xml, should be submitted to Google Webmaster Tools automatically if you have the ping setting for Google on. You’ll then find the sitemap under the “All” links, in the right hand side on GWT:
    You’ll then be able to click on the sitemap_index.xml file and see the sub sitemaps and indexation per sub sitemap.
  • So there’s no sitemap.xml file anymore?
    No, correct. This change was made to prevent collisions with other sitemaps, something that happened quite regularly. You can safely remove it from Google Webmaster Tools and other search engine portals.
  • What to do with the wp-content/uploads/wpseo/ folder?
    You can safely remove it, as said the plugin no longer writes static files.
  • The plugin doesn’t add anything to the .htaccess file?
    No it doesn’t, the rewrites for the files are handled with WordPress internals.
  • The plugin doesn’t add the sitemap to the robots.txt file!
    No it doesn’t. There’s no real good reason to do that when you’re pinging the search engine of changes with the sitemap. In my experience it only helps scrapers find new content on your site. The fact that another XML sitemap plugin does do that doesn’t make it better nor a requirement.
  • Is there a .gz version of the sitemaps file?
    No, but if you run a caching plugin like W3 Total Cache the output will be gzipped anyway.

Tags:


Yoast.com runs on the Genesis Framework

Genesis theme frameworkThe Genesis Framework empowers you to quickly and easily build incredible websites with WordPress. Whether you're a novice or advanced developer, Genesis provides you with the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.

Read our Genesis review or get Genesis now!

134 Responses

  1. Gautam A DBy Gautam A D on 30 May, 2011

    thanx…i have been really struggling with GWT issues. It was showing type as “images”. Thanx for addressing this issue, but i dont know why i am not able to submit my sitemap_index.xml to bing…it is not accepting it! :( anyway there was one more issue that was not corrected after the 0.4.1 update came i.e. the export feature. When i click on export the plugin generates some unusual error about ajax! please fix this. thanx :)

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      I’ll look into the export feature, not a top priority right now to be honest, want to solve all open xml sitemap issues first.

  2. HectorBy Hector on 30 May, 2011

    Thanks for clarifying all these issues which were starting to build up on the forums.

    One question though, I do not have Total Cache but have Nginx and getting a 404 error. Do you (or anyone reading this) know how to translate the exemption as a Nginx redirect so it is passed onto WP and not tried to be served from a static file?

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      Just asked someone on Twitter who said it was working for him with NGINX whether he’d had to do something specific to get it to work. Are you running another caching plugin perhaps?

      • HectorBy Hector on 30 May, 2011

        I did have WTC but I uninstalled and deleted completely. I have double checked and no cache dir or advanced-cache file left behind. I think it is unlikely that it is doing cache even after death?

        I do have DB Cache Reloaded now, although I believe it works differently and not intercept static file requests.

        But if you say Nginx is not the culprit, at least I can focus my hunt elsewhere and discard that option. Hmm….

        • Joost de ValkBy Joost de Valk on 30 May, 2011

          Did you clean up your .htaccess after removing W3TC? There’s probably stuff left in there. All the redirects for this plugin are done within WordPress, so nothing is added to the .htaccess for it other than the normal WordPress .htaccess lines.

          • HectorBy Hector on 30 May, 2011

            OK I just forced the request to Apache, which I am using for PHP behind the scenes (why hadn’t I tried this before? duh!) and still get the 404, which means it is NOT Nginx. Oh well… something left behind by W3TC….

            Thanks again for the super fast replies Joost. Awesome, phenomenal plugin.

      • HectorBy Hector on 30 May, 2011

        … although, something keeps bugging me, now that I kept thinking. Clearly there must be a redirect somewhere since there is no physical XML but rather a PHP that responds to such request. Are you not redirecting these requests through htaccess?

        Meaning, how will Nginx, without an htaccess redirect know that a .xml file should be parsed by a PHP?

        Thanks for the support mate.

        • Joost de ValkBy Joost de Valk on 30 May, 2011

          Are you running with index.php in your links perhaps Hector?

        • Ovidiu BicaBy Ovidiu Bica on 30 May, 2011

          WPSeo sitemap_index.xml works for me with Nginx as a standalone server, no proxy in front of it.
          You should take off nginx for a minute and see how apache handles WpSeo so Yoast can determine where exactly is the problem.

          If it works with apache make sure there’s no xml and xsl extensions in you nginx location declaration.

          I had to remove those extensions from browser cache to work with WpSeo.

          • HectorBy Hector on 31 May, 2011

            Hi Ovidiu. Thanks for the tips. I tried calling the XML directly from Apache by declaring its 8081 port and still got the error. So it is in fact NOT an Nginx issue.

            I installed W3TC AGAIN to see if the new version with the 404 exemption fixed it… nada. I have now uninstalled Total Cache again (was going for Super Cache to begin with). In fact I tried to disable Yoast’s sitemap altogether and try a separate sitemap plugin but the checkbox seems to persist… aaaaaaarrrrggghhhh!!

            Will keep trying. I hope I don’t have to uninstall the plugin altogether as I really like it.

  3. Simon SprankelBy Simon Sprankel on 30 May, 2011

    Thanks a lot for the Update, Joost – it is amazing how often you update your plugins, great!

    Unfortunately, if I try to open the sitemap, I only see the homepage of my blog. I do not have installed any caching plugin.
    If I submit it to Google anyway, they tell me:
    XML-Sitemap ist HTML
    Bei Ihrer XML-Sitemap handelt es sich offenbar um eine HTML-Seite. Bitte verwenden Sie stattdessen ein unterstütztes Format für XML-Sitemaps.

    Any help is appreciated.

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      You could try going to your permalink settings page and re-saving the permalink settings, let me know if that fixes it!

      • DavidBy David on 30 May, 2011

        Hi,

        Having this same problem. No caching installed. Re-saving the permalinks doesn’t fix it. Neither does clearing out my .htaccess down to the original WP entries. Still happens when I disable all my other plugins. Any ideas?

        Thanks!

      • DavidBy David on 30 May, 2011

        Hi,

        Having this same problem. No caching installed. Re-saving the permalinks doesn’t fix it. Neither does clearing out my .htaccess down to the original WP entries. Still happens when I disable all my other plugins. Any ideas?

        Thanks!

      • Sean SweeneyBy Sean Sweeney on 30 May, 2011

        I’m having the same issues. Completely turned off cashing, only get homepage.

        Please advise.

        Thanks,

        S

      • Brian CruikshankBy Brian Cruikshank on 30 May, 2011

        Hi Joost,

        I am having the same problem. The new sitemap_index.xml just redirects to the homepage. I have the latest version of everything (WP, WTC, WPSEO 4.1). Furthermore, when I try to disable sitemaps by unchecking the box and saving, it just re-enables itself!

        Thanks for your help!

      • Simon SprankelBy Simon Sprankel on 30 May, 2011

        Unfortunately, this does not fix this issue for me.

        Cheers,
        Simon

  4. Blake ImesonBy Blake Imeson on 30 May, 2011

    Sorry if this is not the best place to post issues but the export function now seems to be broken for me. On one site it gave no error and then on a dev site that is resolving to an IP address it gave:

    Warning: chdir() [function.chdir]: No such file or directory (errno 2) in /home/growth/public_html/wp-content/plugins/wordpress-seo/inc/wpseo-non-ajax-functions.php on line 93

    Any ideas? Thanks! (awesome plugin BTW)

  5. KAIBy KAI on 30 May, 2011

    How about w3 Super Cache? I’m using this cache and got a 404 error! How can I fix this ?

    • kaiBy kai on 30 May, 2011

      I think I have fixed this. I just noticed that my sitemap format has changed to index. So, I submit a new index sitemap to google and it works. Does it mean that I can ignore (delete) the sitemap.xml and sitemap.sml.gz?

  6. InfonoteBy Infonote on 30 May, 2011

    Hi,

    Thanks for the work on the plugin.

    I use an automatic translation plugin to translate my posts. Is it possible to add those posts to be included in the sitemap?

    Thanks

  7. ThomasBy Thomas on 30 May, 2011

    Thank you for your plugins, they really help me a lot. Unfortunately there’s an issue with the SEO Plugin regarding the Sitemap that’s not mentioned above.
    I get an error message from Google Webmaster Tools. It says that my sitemap is an html site. When I enter the Adresse http://feuerhaken.org/sitemap_index.xmlI'm redirected to my frontpage, but don’t know why. I’m not using any caching plugins. Maybe someone could help me?

  8. Pat DavisBy Pat Davis on 30 May, 2011

    I have updated to the newest version of the WP SEO plugin. Everything looks good with the new XML sitemaps EXCEPT one URL on my post sitemap page. It is showing up as an error in GWT because is only shows a slash and post name ie. /mac-capture-video instead of the full URL. How can I correct this?

    Thanks

  9. donnacha of WordSkillBy donnacha of WordSkill on 30 May, 2011

    Yoast, are there any problems using the XML sitemap feature on sites within a Multisite network?

    Thanks for your continued work on this excellent plugin.

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      No it should work completely. Let me know if you find any issues.

      • Dave ReadBy Dave Read on 4 June, 2011

        I’m getting the redirect to homepage for one of 3 sites on a multisite setup – other 2 work fine. Have tried various fixes suggested here. All latest versions – no cache plugins.

  10. LaurieBy Laurie on 30 May, 2011

    I imported my settings to the new version and it is not generating sitemap_index.xml anywhere within my blog’s file structure. I also can not change the XML settings, such as turning off the XML sitemap feature.

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      Seems to work for me

      • LaurieBy Laurie on 30 May, 2011

        After re-resaving the permalinks, I can see the sitemap now (“XML Sitemap generated by Yoast’s WordPress SEO plugin…). However, I tried again to change the XML settings Under SEO/XML Sitemaps and it won’t save the change. Sent you a screenshot via em.

  11. BarisBy Baris on 30 May, 2011

    hi Yoast,

    I have same problem like Thomas.I have two sites.One site with quick cache plugin and other without.I get an error message for both sites from Google Webmaster Tools too.

    Any idea?

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      I need more details :)

      - Which theme are you running?
      - Any other plugins shared by those two?
      - What do you get when you open the XML Index Sitemap URL?
      - Did you re-save the settings on the permalinks page?

  12. RikBy Rik on 30 May, 2011

    Unfortunately I also have problems getting XML-sitemap to work. Pointing my browser to domain.com/sitemap_index.xml results in showing my homepage.

    I use W3 TC plugin and used the settings mentioned above. Also saved permalink settings (as well in general settings as in WP SEO settings) but Nothomb works.

    I have the latest WP version and W3 TC installed… Using my own theme but that shouldn’t be a problem. Hope someone can give me that magical tip that gets the sitemap up and running again.

    Regards.

  13. BarisBy Baris on 30 May, 2011

    certainly ;)
    1- Expose theme and Directorypress Theme
    2- Keyword Winner Plugin,Statify,Antispam Bee and WP Google Fonts
    3-Redirect to my homepage
    4-Yes, I did it ;)

  14. BarisBy Baris on 30 May, 2011

    Yoast, should I send my admin access?
    If yes, which site? or both?

    thanks

  15. JohnBy John on 30 May, 2011

    Spotted this:

    Site uses www prefix.

    http://www.domain.com/sitemap_index.xml is fine

    but

    domain.com/sitemap_index.xml appears to be empty.

    • Joost de ValkBy Joost de Valk on 30 May, 2011

      It’s not empty, the XSL just doesn’t work because it’s not on the same domain. Next version will redirect :)

  16. MichaelBy Michael on 30 May, 2011

    Hello. I have a similar problem as others. It appears my sitemap_index.xml is not being generated:

    http://vodkabrands.org/sitemap_index.xml

    I’m using the Weaver theme. I re-saved my permalinks. I did not have a cache plugin (but did just install the W3 to see if it would kick it in, but no change). I previously had the Google Sitemaps XML plugin installed, which I have now deactivated, but /sitemap.xml is still there.

    • Michael BorgerBy Michael Borger on 31 May, 2011

      Btw, I added sitemap(_index|[0-9]+)?\.xml to the exemption page – no difference.

  17. Alex RodriguezBy Alex Rodriguez on 30 May, 2011

    This may seem like a random thing but would it be possible to split this into two plugins? One for SEO settings and the other the Sitemap? Or have an option to turning off the SEO part?

    Thanks!

    • Joost de ValkBy Joost de Valk on 31 May, 2011

      No. The point is that as soon as you noindex a post, for instance, you want it to disappear from the sitemap too. Good SEO is not something you can split up into small pieces that easily, it’s one big integrated approach.

  18. Marco BrennBy Marco Brenn on 30 May, 2011

    HI Joost,

    same on my sites. XML-Sitemap Options are not saved. And sitemap_index.xml will be not generated.

  19. DenizBy Deniz on 31 May, 2011

    Hi, originally I just wanted to send a thank you note for the plugin. But when I saw the issues with the sitemap I immediately checked mine to see if I had the same issues.I just loaded the plug in for the first time on my GPS Ratings blog and I’m not having any problems with xml sitemap at all.
    Of course this doesn’t help anyone but at least knowing that the plugin does actually work may be some help. I’m sure the issues will be worked out.
    I am running W3 Total Cache, Version 0.9.2.2.
    I have not disabled the Google XML Sitemaps so both site maps are being generated without issue. I will disable GglXmlStmps later.
    Everything seems to be working fine. I do wish I could export but I know that this is a much more important issue.
    I do have one question. I’m one of those in favor of putting the xml sitemap link in the robots text file so that there is one more chance for the engines to find it, Joost, would you say that this is a good or bad idea? I would really like your opinion.

    Thanks you for all your great work,

  20. Navjot SinghBy Navjot Singh on 31 May, 2011

    I am using WP SEO plugin on 2 of my sites on a shared hosting. On one site I have WP Super Cache enabled. The one where Super Cache is enabled gives a 404 for Sitemap file while on the other one, sitemap link gets redirected back to homepage. Tried saving the permalink options but it didn’t work. Where site gets redirected to homepage, I am using Magnitude theme [Child theme of Twenty Ten] and on the other one I am using Boldy theme by Site5.

  21. HectorBy Hector on 31 May, 2011

    Hey… can I overwrite with 0.3.5 just like that or will I lose data or something will break?

    I’m thinking to better go back to the old version which created a physical file. Recommendations?

  22. Dave from The Longest Way HomeBy Dave from The Longest Way Home on 31 May, 2011

    I’ve posted on the WP forums about this. Will do again here.

    4.1 issues. index Site map generated, categories, tags no problems. When I click on posts site map I get an internal 500 error.

    Later, I was getting 500 internal error on the admin screen too.

    Finally, when I tried to disable the sitemap function within the WP SEO settings, by un checking the box, and then clicking save – the page would reload, and the check box would be ticked. Meaning, I could not disable the sitemap option on the plugin.

    Finally, my long standing issue since around 3.5 has been that it’s not automatically updating the sitemap on scheduled posts. (I can’t check this in 4.1 due to the above issues)

    I’ve since reverted back to a previous version, and all is working fine again with the previous version. Not 4.1.

    • HectorBy Hector on 31 May, 2011

      Hi Dave. I plan to downgrade too. Did you just upload the old zip and overwrite the files? Did it work fine and nothing broke? No data (meta descriptions etc) lost?

      • Joost de ValkBy Joost de Valk on 31 May, 2011

        Hey guys, I’d love to check out what causing the issues, could you email me admin details @ joost at this domain?

        • HectorBy Hector on 31 May, 2011

          Oh cr*p I sent it to admin@.. If you didn’t receive it I can create the user again and resubmit credentials. Can you confirm if u got the emails?

          Thanks so much, really.

        • Marco BrennBy Marco Brenn on 31 May, 2011

          HI Joost,

          you got my mail right now. :-) Hope you could help me.

  23. MargaridaBy Margarida on 31 May, 2011

    Hi keep getting this message:
    [an error occurred while processing this directive]
    I have updated W3 Total Cache as you said, even though it has bugs and I cannot get it to gzip my minified files, but it still gives me the same error.
    Any ideas what could be going on here?

  24. Tony PageBy Tony Page on 31 May, 2011

    I’m having the “sitemap_index.xml redirects to blog front page problem”. This is on blogs within a multisite network using the single domain with sub-sites in folders set-up. However, the plugin works correctly a standalone blog on the same domain, generating the index file and the four sitemap files. This blog has exactly the same plugins as the multisite network but uses a different theme: the network uses the Studiopress Genesis + Lifestyle child theme. Hope this helps with a solution…

    Linux + Litespeed

    • Yip BopBy Yip Bop on 1 June, 2011

      I had the same issue and fixed this by going to the Permalinks setting of WordPress SEO, and unchecking “Redirect ugly URL’s to clean permalinks”

  25. BillBy Bill on 31 May, 2011

    Hi, one quick question. Is it possible to change the frequency (Ch. freq.) from daily to hourly?

  26. ThomasBy Thomas on 31 May, 2011

    I can confirm that. After deactivating the permalinks settings of the plugin the issues mentioned above are gone.

    • ThomasBy Thomas on 31 May, 2011

      This was a reply to a post by Marco Brenn, that appeares to be gone?

      • Tony PageBy Tony Page on 1 June, 2011

        Thanks for the reply, @Yip Bop, and as a matter of fact I actually confirmed that Marco Brenn’s solution (the same as yours) worked for me yesterday, it seems both his post, my post and I think one more post from Rich who had the same result seem to have vanished as Thomas says!

        • Joost de ValkBy Joost de Valk on 1 June, 2011

          Did a server move yesterday (you might have noticed the incredible slowness before that) apparently we didn’t sync up the comments correctly, very sorry for that!!

          • Tony PageBy Tony Page on 1 June, 2011

            Just for the record, after Joost’s plugin successfully created 23 site maps for my network, Google listed them all finding no errors and increasing the number of pages in the index by around 20%, all within an hour or so, the initial index files within minutes. So obviously if you get things working for you, you can expect a good return for your effort…

  27. Jonathan BlundellBy Jonathan Blundell on 31 May, 2011

    I updated the W3 Total Cache plugin and still had a 404 error but after adding the suggested script:
    sitemap(_index|[0-9]+)?\.xml
    On the browser cache page and making sure I had no warnings (red info boxes on the W3 plugin page) everything worked fine. Thanks!

    • MargaridaBy Margarida on 1 June, 2011

      Hi, I tried the same thing exactely as you say here: updated W3 Total cache, got rid of all the warnings without ruining my site, prevented the gzip of minified files that is not working in W3 Total Cache, added the ‘sitemap(_index|[0-9]+)?\.xml’ but I still get error when I try to create my sitemap.
      I keep getting the same error over and over:
      [an error occurred while processing this directive]
      Does anyone have the same problem or any ideas how to correct this?

      • Joost de ValkBy Joost de Valk on 1 June, 2011

        Which theme are you running?

        • MargaridaBy Margarida on 1 June, 2011

          It’s a childtheme of 2010 that I created myself.
          Is it possible I have some function there that it’s ruining everything here?

  28. Roger HustonBy Roger Huston on 31 May, 2011

    Hello,

    I have a pretty critical issue. On the XML Settings screen. I cannot uncheck NOT to include posts. I was checking items to see what they did, and now that I have checked this entry, I cannot uncheck it. I try. I uncheck, then hit save settings, and as soon as I do, it comes back.

    - Roger

    • Roger HustonBy Roger Huston on 3 June, 2011

      Hello,

      Do you know when the new version will come out? My biggest problem is that I cannot get the Sitemap to work for my posts. I had checked EXCLUDE Posts from the Sitemap, now I can’t uncheck it. No matter how many time I try to unset it, it resets itself when I try to save.

      If a new version isn’t coming soon, perhaps I will need to roll it back.

      - Roger

  29. Ünsal KorkmazBy Ünsal Korkmaz on 1 June, 2011

    Am i the only one have www redirect problem?
    Example:
    http://kremim.com/
    http://www.kremim.com/

    when i disable your plugin, www adress redirecting to non-www as normal..

    • Marco BrennBy Marco Brenn on 1 June, 2011

      Hey,

      how looks your .htaccess?

      • Ünsal KorkmazBy Ünsal Korkmaz on 1 June, 2011

        You can test it yourself too, just enable w3-total-cache and yoast seo plugin together and www stops redirecting.

        • Joost de ValkBy Joost de Valk on 1 June, 2011

          That’s a bug I’ve found and have a patch forthcoming for.

        • Ünsal KorkmazBy Ünsal Korkmaz on 1 June, 2011

          Maybe not.. not sure.. still trying to find whats problem.. Just i know, when i disable wordpress seo, www starting to redirect again. I am disabling options 1by1.. maybe i will find problematic section.

          • Ünsal KorkmazBy Ünsal Korkmaz on 1 June, 2011

            Oh.. didnt see your answer. Glad i am not alone and fix coming ^^

  30. LinoBy Lino on 1 June, 2011

    Hi, for first I want to congratulate you for your work.

    I have a problem with the module xml sitemaps. In practice I can not turn it off. Every time I save the module settings, they are not saved but returns as the default settings, even i save it I get the message: Settings updated. The file sitemap_index.xml is not generated.

    I made ​​several tries but to no avail:

    with default theme and with all plugin active and after with all plugin deactive.

    I have regenerated the permalinks and .htaccess file, and i do not use cache plugin.

    Joost or someone else can help me to solve this problem?

    • Joost de ValkBy Joost de Valk on 1 June, 2011

      An update for that is coming :) it is indeed impossible to disable it right now.

      • LinoBy Lino on 1 June, 2011

        Joost, Many thanks!!

  31. James JemieBy James Jemie on 1 June, 2011

    I have a concern. I run a few automated blogs that create quite a bit of content daily. Usually I would check “don’t update sitemap after every post” or whatever. Now that option is gone. I understand that you now seperate the sitemap into chunks of 1,000 posts. Still, since my blogs autopost hourly, and the sitemap rebuilds on every post, won’t this be a trainwreck for my website’s CPU cycles or whatever?

  32. A SamuelBy A Samuel on 2 June, 2011

    Great plugins Yoast, I have them running on lots of blogs…I have recently installed the new WordPress SEO on one blog and the sitemaps that get generated for posts and pages give a 500 internal server error. However the sitemap_index.xml and the sitemap_category.xml work. Any thoughts on why this might be?

    Cheers

  33. DonElleBy DonElle on 2 June, 2011

    I have the ping settings checked on for Google but cant see the sitemap_index.xml file when i click on “all” links in my GWT.
    please help.

  34. MattBy Matt on 2 June, 2011

    Hey, I have no sitemap now. I had one just fine before but now I get a 404 when I try to find it. I’m using your WordPress SEO plugin but there is still no sitemap. Any help?

    • MattBy Matt on 2 June, 2011

      just as a follow up, I’m now using the Google XML Sitemaps Plugin to build a sitemap, so I have one here, http://wegotthiscovered.com/sitemap.xml, but just curious as to why your WordPress SEO plugin doesn’t build one anymore?

      • DenizBy Deniz on 2 June, 2011

        I had the same issue and the same reaction, so I decided to cool down and came back an hour latter. Well the site map was actually created then. I guess, for me at least it took a while for it to get going. but now the site map comes up whenever I check them (10+ blogs)

        • MargaridaBy Margarida on 3 June, 2011

          Same happened to me. The sitemap seemed like it was not being created but coming back later to check it out, and now I do have a sitemap.
          Thanks for the tip and thak you Joost for creating this :)

  35. Bret WebeauBy Bret Webeau on 2 June, 2011

    Is there a class and function we could check to see when the sitemap is being called. I use a homegrown HTML5 formatter on the output of my pages. It checks for things like ‘is_home() || is_404()’ etc. I tried to taking out the is_404() check but it still seems to process this page. Is there a type specific to the sitemap I can exclude – something like ‘is_sitemap()’? Right now it generates the xml but the output is getting wrapped by HTML by my beautifier.

    Also, for those of you who are having 404 problems, check your server configuration. If xml files are being treated as static files (because they usually are most often because you want to determine your own expiration for them) then requests to xml files that don’t exist are never processed by wordpress and you get a 404 from the server software. You’ll be able to tell if that’s the case. A 404 from WordPress looks like a website page using your sites layout, a 404 from the server looks like a bunch of text.

    I had this in my nginx config that prevented proper execution:

    location ~* \.(?:xml|ogg|ogv|…

    Took xml out of this list and the plugin worked except for getting wrapped in HTML tags.

    • Bret WebeauBy Bret Webeau on 2 June, 2011

      Follow-up on my own post, the plugin returns True for is_home() when requesting sitemap_index.xml

  36. MattBy Matt on 2 June, 2011
  37. TammyBy Tammy on 2 June, 2011

    Hi, Joost -

    I’m experiencing a similar problem to what a few others have reported – no sitemap anymore on my main domain since upgrading the plugin. The /sitemap_index.xml just redirects to my home page. The sitemap was correct and functioning before.

    Not sure what to do..it’s kind of a big problem not to have a functioning xml file. Anything you can suggest?

    Thanks!

  38. MikeBy Mike on 2 June, 2011

    I’m in the same boat as several others here with no sitemap being generated. I get the 404 error on one of my websites at http://www.mycornerstone.org/sitemap_index.html

    I’ve added the code in the W3 Total Cache exceptions and still no luck. I’ve tried disabling plugins as well. Just wanted to let you know.

  39. StuartBy Stuart on 3 June, 2011

    Your page does not have a download sitemaps button. I check wordpress site but its not clear which one is yours??

  40. Jesse DijkstraBy Jesse Dijkstra on 3 June, 2011

    Hey Mike,

    I seem to have the same problem as you do. I’m getting a 404 error handled by Litespeed instead of WordPress. We’re running our sites on a multi-site environment with W3 Total Cache enabled.

    The exceptions don’t solve the issue. Seeing Mike is running Litespeed just as I am it seems to be a Litespeed problem (?).

    http://budeco.nl/sitemap_index.xml

    • Tony PageBy Tony Page on 4 June, 2011

      Hi Jesse, I’m using Litespeed with multisite in single domain multi subfolder blog format. I’m using Super Cache as W3 Total Cache has ongoing problem with switch_to_blog (causes redirection errors) AFAIK (I’m using SitewideTags to pull posts to the main blog).
      No problem with WordPressSEO after unticking the permalinks box (see my post above). I had all kind of weird problems with W3 TC (my preferred caching plugin) and finally narrowed things down to the switch_to_blog issue which Fred said would be fixed in a future release. So I’d have a look at that, you never know.
      BTW, if you’re running a multisite network in the same format as I am and NOT having other problems with W3 TC I’d sure like to hear about it…;-)

  41. AdrianBy Adrian on 4 June, 2011

    Hi Joost,

    Same problem here with sitemap going to homepage.

    Adrian

  42. EdBy Ed on 4 June, 2011

    With regards to the sitemap_index.xml page redirecting to the homepage issue.

    Uncheck the “Redirect ugly URL’s to clean permalinks. (Not recommended in many cases!)” options on the Permalinks page.

    • TammyBy Tammy on 4 June, 2011

      Hi, Ed – Thanks for the suggestion. Unfortunately, I don’t have that box checked and so I’ve still got the problem with the sitemap redirecting to my homepage. Hmm….

      • TammyBy Tammy on 4 June, 2011

        Spoke too soon — that did fix it! I was looking at the wrong site in my multisite setup when I checked. Thanks, Ed!!

    • SeanBy Sean on 5 June, 2011

      Thanks, turning off the “Redirect ugly URL’s to clean permalinks.” setting fixed the sitemap redirecting to homepage issue for me. Works fine now.

  43. AdrianBy Adrian on 4 June, 2011

    Cheers Ed for that, shouldn’t have ticked it in the first place. Also found it affected the google mashup plugin.

    Really delighted to get it working, great plugin.

  44. RIck DawsonBy RIck Dawson on 4 June, 2011

    It appears that a lot of database garbage left behind from an uninstall of all-in-one webmaster had something to do with it for me. Once I took those entries out and toggled the “replace with pretty links” checkbox as mentioned above, it worked.

  45. MariusBy Marius on 5 June, 2011

    Hi
    I have the same issue as most of the people above, but cannot make out from all the posts on here what the solution is – if any. The sitemap does not want to generate and I get an 404 error. I dont use ay cashing and my template is Magazine Premium from c.bavota.
    I will swicth now to another sitemap generator and will try this one maybe again in the next decade .. or so – unless there is one confirmed solution to fix the problem. Is there?

  46. Joergenf FroelnerBy Joergenf Froelner on 5 June, 2011

    Hi Joost it works with domains BUT not with subdomains. What can I do?

  47. Doug SmithBy Doug Smith on 5 June, 2011

    I’ve recently been testing the new bbPress beta, which has changed to be a WordPress plugin instead of stand-alone forum software. I love what they’ve done with it but I had a big concern about generating a sitemap with the plugin I’m currently using.

    The site has many thousands of pages on an active forum with a few thousand users. Updating a sitemap file every time a user posted was just not going to be sustainable. I had already turned off automatic sitemap generation on my stand-alone bbPress forum because of the server load and delays with each post.

    When I saw your announcement of switching to dynamically generating multiple sitemaps in a scaleable way I knew that was the solution to being able to move to the new bbPress plugin.

    Thanks for all your hard work and for taking the time to explain how you’ve implemented it!

  48. SkonkaBy Skonka on 6 June, 2011

    Would this make my XML Sitemap Generator plugin, unnecessary?
    Just want to know if I go with yours if it would be better to remove that plugin?

    • Aaron HBy Aaron H on 9 June, 2011

      Yes, remove that plugin. It generates the sitemap to disk (/sitemap.xml or /sitemap.xml.gz) where this sitemap (/sitemap_index.xml) is dynamically generated so cannot be found on disk through FTP. Its still up to you which one you use but there is no point in having them both enabled.

  49. MordyBy Mordy on 6 June, 2011

    Hi,

    I was wondering why no content has been added to the robots.txt file?

    How do the search engines know not to follow specific links?

    Also, there are no robot tags in the source code of each page?

    Am I missing something.

    Thank you!

  50. Kevin LearyBy Kevin Leary on 6 June, 2011

    I was having issues with the latest version of W3TC and sitemap 404′s, even after adding the recommended exceptions. I opened up my .htaccess and removed the following lines:

    # BEGIN W3TC Skip 404 error handling by WordPress for static files

    RewriteEngine On
    RewriteCond %{REQUEST_URI} !(robots\.txt|sitemap\.xml(\.gz)?)
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} \.(css|js|html|htm|rtf|rtx|svg|svgz|txt|xsd|xsl
    |xml|asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|exe|gif|gz|gzip|ico|
    jpg|jpeg|jpe|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|
    odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|swf|tar|tif|
    tiff|wav|wma|wri|xla|xls|xlsx|xlt|xlw|zip)$ [NC]
    RewriteRule .* - [L]

    # END W3TC Skip 404 error handling by WordPress for static files

    And that fixed it. Server setup is CentOS/WHM cPanel with Lightspeed.

    • Joost de ValkBy Joost de Valk on 6 June, 2011

      Did you make sure W3TC updated your .htaccess file before you had to do that?

  51. Jeff ChappellBy Jeff Chappell on 8 June, 2011

    Just adding my two cents: I had to add the exception to the W3TC browser cache, and in Yoast WordPress SEO, in the permalinks section, uncheck “redirect ugly URLs” option to get the sitemap to generate. Following the advice above, I unchecked everything in the permalinks section, and then tried the various combinations — but it was the ugly URLs option that was causing the problem with sitemaps and redirecting to the homepage.

    Does the new version that was pushed just now — 0.4.2 address the above issue? Just curious.

    BTW, it’s a great plugin; it helps simplify a lot of SEO related detaisl. I’ve been using it for some time now, and this is the first issue I’ve ever had with it. Just wanted to say thanks.

    P.S. Don’t know if it’s pertinent, but I use various themes from ElegantThemes on all three of my sites, and was having the same sitemap issue across all three until I figured out the above solution.

  52. Greg FieldingBy Greg Fielding on 9 June, 2011

    Any way to make the sitemap show the buddypress activity stream? Most of my content is there – it’s should be my highest priority.

    Really appreciate your help!

  53. SimonBy Simon on 11 June, 2011

    I’m getting nothing but a blank page for my sitemap. Any idea what I could do to fix it? I read in the support forums that someone else had the issue, and then it just magically resolved itself. My sitemap has still not become magical… :(

  54. Andy from WorkshopshedBy Andy from Workshopshed on 13 June, 2011

    Is it better to have all this functionality in one plugin or perhaps a series of plugins? I can see that the XML sitemap functionality would be good when my site gets larger but I’m not sure I want ALL of the other functionality

    • Joost de ValkBy Joost de Valk on 13 June, 2011

      The reason I made the plugin so comprehensive is not that I want it to do all per se: good on-site SEO is the combination and integration of several aspects.

      • Andy from WorkshopshedBy Andy from Workshopshed on 13 June, 2011

        I agree with that but it means that the one plugin has to be the best for all the functions provided or I’d end up with duplicated functionality between your plugin and anothers.

        Perhaps you could get the plugin to write engaging articles and automatically respond to people’s comments? Unless you’ve already implemented that…

  55. Marco BrennBy Marco Brenn on 15 June, 2011

    Hey Joost,

    could you say anything about Local SEO and Video SEO Modules? Could i buy those modules at this time?

    Regards
    Marco

  56. Adam HaworthBy Adam Haworth on 16 June, 2011

    I have just been playing around with GWT and got my sitemap up and running this SEO is brilliant I have just taken all in one SEO off :D

  57. Paul OgierBy Paul Ogier on 17 June, 2011

    Hi there, I have used your plugin on 1 other site and now am trying it on a very large database and nothing is being generated. Does it take a while to generate? Should I give it a while? There are now cache plugins. I have resaved the permalink settings, I have excluded everything except for the categories and nothing is coming up. I have also ftped into the site to see if anything was generated and nothing.

    • Paul OgierBy Paul Ogier on 27 June, 2011

      Please could someone give me something that i can try?

      • Joost de ValkBy Joost de Valk on 27 June, 2011

        what’s your domain?

        • Paul OgierBy Paul Ogier on 27 June, 2011

          Will send an email.

  58. George HalsteadBy George Halstead on 20 June, 2011

    I just installed the latest and everything went through fine. What is weird is that the plugin i was using only had 87 links submitted. This on nearly tripled it and no errors. Great jobs, thanks friend.

  59. Jeroen SlagtBy Jeroen Slagt on 23 June, 2011

    Another problem that arises in W3TC is with the cache preloading on the page cache settings page.

    Setting the Sitemap URL to sitemap_index.xml obviously does not result in the generating of cache files, since it does not contain any URL’s.

    An additional feature where in W3TC where it finds the different sitemaps linked from sitemap_index.xml itself and then uses the links in all of those sitemaps would be great.

    Or a feature in Yoast SEO with a possibility for a single sitemap to load into W3TC…

  60. NileshBy Nilesh on 24 June, 2011

    So we dont need sitemap.xml and sitemap.xml.gz i submitted my sitemap_index.xml to google,bing,yahoo without any error. In comments you said that if wp total cache plugin is active it generates sitemap.xml.gz . But i didn’t find anything in options of wp total cache to build that sitemap.xml.gz . just let me know how to build that sitemap.xml.gz and do we need that? as i already submitted sitemap_index.xml to search engines.

  61. FalkBy Falk on 25 June, 2011

    Installed it on my Site (based on THESIS theme); WP Multi-site install; latest versions of WP, Thesis & Yoast’s WP SEO plugin (and W3 Total Cache): But… Yoast’s title will NOT show for my home page… (http://learnjazzheaven.com), even though I checked “Force rewrite titles” under TITLE SETTINGS of the WP SEO plugin…

    It still wanted to show THESIS’ title… I deleted that.

    But now it shows the title from WordPress’ own GENERAL SETTINGS…

    => Any pointers what I could be missing?

  62. FalkBy Falk on 25 June, 2011

    PS: to last post: now I deleted the info for site title & tag line in WP’s GEN. SETTINGS & cleared my browser’s cache: still shows the previous entries from WP’s GEN. SETTINGS – and not the title I entered on the home page with Yoast’s WP SEO plugin… Mhhhh.

  63. MuzzaBy Muzza on 29 June, 2011

    Where is sitemap_index.xml located in WP MU so I can submit these to google. I have some sites indexing and some not. Using hyper cache disabled.

Trackbacks