The honest Truth about Plugin Testing

After my post about content analysis with the WordPress SEO plugin, which covered the recent updates to that plugin, I got some comments that we should test better, especially because the commenter had issues with a certain slider plugin. I laughed when I read that comment, but then it dawned on me that most in the WordPress community might not have the insight that we here at Yoast have into the WordPress world. Most people are (of course) unaware of the enormous amount of other plugins used by our users of the WordPress SEO plugin.

We’ve been doing anonymous and opt-in tracking of WordPress installs for just over a year now. This allows us to check whether or not people update our plugins and much more. Also, it allows us to see which other plugins people use on their websites. Of course, we test compatibility of every plugin update with the most commonly used plugins. To test compatibility with every plugin, however, is impossible. Let me show you why by giving you some numbers.

The WordPress SEO plugin runs on at least 1 million and in fact probably more than 2 million sites. All of these sites use different plugins. The 691,797 sites we currently track in our tracking system use approximately 83,000 different plugins, in innumerable combinations. After the 10 most used plugins, none of them are run on more than 10% of those sites, yet most sites run more than 5 plugins. In fact, let me show you the distribution of number of plugins per site:

number of active plugins per site

This doesn’t even take into consideration the fact that all of those people run a different theme too, and that they each have different settings for each plugin and theme as well. All of these can (and actually do) interact and interfere with each other. Because of that, testing if a certain update works for all of our users and their plugins is – and I hate saying this - impossible.

What to do if you’re updating?

If you are installing an update of a plugin, any plugin, - especially when you have a webshop or are otherwise making money from your site – you should always test the new version first on a staging environment. Be aware of plugins with very little users and of plugins developed by people with limited WordPress experience. Of course, this plugin may just have the exact functionality you need, but these plugins will come with a higher risk of compatibility issues.

What you should do is figure out the most important things people should be able to do on your website, create specific test plans to reproduce them and then test them against every (plugin/theme/core) update before you put it live. We as plugin developers are not responsible if something breaks because of incompatibilities, you are. Of course we’ll try and prevent it, but… well, you’ve read the above. This means that installing another plugin for feature X does always come with a cost, even when you’re not paying for the plugin.

This is also why I don’t really get the free versus premium debate. Some people seem to hate paying for plugins, but they don’t realize the biggest investment is actually their time, not the money. Paying money so that a plugin developer can properly test it because he has the resources is actually a good thing.

A few WordPress hosts get this particular issue quite well, especially WP Engine (aff) seems to have taken their one-click staging to such a level that it makes this kind of testing really easy. I assume from having conversations with other hosting parties that they will soon follow.

A word on versioning

I thought it’d be wise to add a note to this post on our versioning system. We try to stick with Semantic Versioning, which means that upgrading from 1.4.17 to 1.4.18 for instance should be painless as the upgrade only contains bug fixes. We have in the past made the mistake of adding functionality in a minor release, we won’t do that again. Those minor releases are for bug fixes, new features will be marked by upgrading from 1.4 to 1.5, like the upcoming release.

When we go to 2.0, which we are planning for, we will drop some (though only a small part) of our old API, so at that point we increase the major version number.

Check out our WordPress SEO Premium plugin

Yoast.com runs on the Genesis Framework

Genesis theme frameworkThe Genesis Framework empowers you to quickly and easily build incredible websites with WordPress. Whether you're a novice or advanced developer, Genesis provides you with the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.

Read our Genesis review or get Genesis now!

16 Responses

  1. KarenaBy Karena on 30 September, 2013

    Understandable, for sure! I am curious though. What are “10 most used plugins” your stats show?

    • Joost de ValkBy Joost de Valk on 30 September, 2013

      They’re highly skewed, which is why I didn’t share them, as number 1 and 2 are my own plugins as they contain the tracking code :)

    • KimBy Kim on 30 September, 2013

      Karena, you can find the top 15 plugins across 700,000 website in this report:

      http://blog.presstrends.io/2013/08/recent-stats-for-the-wordpress-org-developer-community/

      If I’m not mistaken, this would be the most accurate data available on themes, and plugins, albeit with an inflation for themes and plugins which include PressTrends, the tracking tool.

  2. TarielBy Tariel on 30 September, 2013

    Thank you for useful insights,

    They additionally confirmed the way I approach [I am careful] installing plugins. I am now in the process of planning a new WP website. I not only plan the functionality and blocks of info on the blog [choosing the fee based skin based on those], but also plugins [try to keep them under 10] and possible future update headaches [try to make sure they will be driven to minimum].

  3. HassanBy Hassan on 30 September, 2013

    That versioning scheme always confused me. I was like why do they add a handful of new features and stuff and the version number says 1.4.15 to 1.4.16?! I’m glad you’re set to improve on this.

    Joost, do you have any plans to move the plugin translation project to Transifex? Way better than running your own GlotPress, IMO, and easy for us, too :)

    • Joost de ValkBy Joost de Valk on 30 September, 2013

      No we’re not going to migrate to Transifex. It’d cost us $299 a month and GlotPress is a lot cheaper than that :)

      • HassanBy Hassan on 30 September, 2013

        No, they host open source projects for free! Many WordPress plugins are already on there. From their pricing/plans page, scroll down a bit and you’ll a section titled “Open source projects” with a link to “Signup for free”.

        Hidden for obvious reasons :)

        • Joost de ValkBy Joost de Valk on 30 September, 2013

          Well we have our premium plugins on our glotpress too, so that still doesn’t work :)

          • KimBy Kim on 30 September, 2013

            Joost, I thought I’d point it out: your Premium plugins are open source, so, it is still free on Transifex.

  4. MichaelBy Michael on 30 September, 2013

    I ditched a premium plugin that I was using when it broke on an update of yoast’s. When it comes to picking and choosing in this open source world, you definitely have to outweigh the long term benefits of each and every one of them.

  5. Jay DorseyBy Jay Dorsey on 1 October, 2013

    Thanks for sharing some of the metrics along with your explanation. In the past I’ve purchased some premium plugins that don’t work well together. Thankfully many premium plugins offer no questions asked refunds.

    As a developer who makes premium plugins I wonder which is preferred: troubleshooting the edge cases or refunding the client? I usually go straight for the refund because the few times I’ve tried to help troubleshoot (setting up demo sites, isolating cause, sometimes even providing the solution) the author can’t seem to be bothered to fix it. Thanks to your post I now realize why.

    Thanks again!

  6. Couteau SurvieBy Couteau Survie on 1 October, 2013

    Thanks for sharing some of the metrics along with your explanation. In the past I’ve purchased some premium plugins that don’t work well together. Thankfully many premium plugins offer no questions asked refunds.

    As a developer who makes premium plugins I wonder which is preferred: troubleshooting the edge cases or refunding the client? I usually go straight for the refund because the few times I’ve tried to help troubleshoot (setting up demo sites, isolating cause, sometimes even providing the solution) the author can’t seem to be bothered to fix it. Thanks to your post I now realize why.

    Thanks again!

  7. Prasad AjinkyaBy Prasad Ajinkya on 1 October, 2013

    Great point on why we should go for premium plug-ins. Pay for the work and the testing that the developer has done, instead of relying on some free plug-in where the work has not been done.

    Joost, what advice would you give to a webmaster on how to test different plug-ins before installing it on their respective WordPress platforms?

  8. Rhythm ShahriarBy Rhythm Shahriar on 1 October, 2013

    Excellent Points.. Thanks for sharing.

  9. Couteau SurvieBy Couteau Survie on 2 October, 2013

    Great point on why we should go for premium plug-ins. Pay for the work and the testing that the developer has done, instead of relying on some free plug-in where the work has not been done.

  10. JamesBy James on 9 October, 2013

    Great advice, when a client installs a plugin on a shop I don’t think they know the full extent of conflicts that could be caused.