Yoast SEO 14.0.x; or “Why you should never bypass wpdb”

Yoast SEO 14.0 was a release that needed quite a few patches so far. We’re sorry about that. In building Yoast SEO 14.0 we made a mistake: we bypassed wpdb. That caused issues and errors for some users when they tried to upgrade, which we’re very sorry about. This post explains the mistakes we made, and our reasoning. I hope it explains a bit of how this happens, and, for the developers among you, prevents you from making these same mistakes.

Let’s start with what we got right, which paves the way for an exciting future:

  • We started using a custom database table for our metadata. As posts tend to have a lot of SEO metadata, this sped up our frontend performance significantly.
  • To access that data, we started using an ORM (which stands for Object Relation Mapper). Think of that as a more convenient way of mapping the data you use in your application to the data in the database.
  • We didn’t remove the data from wp_postmeta, we just copied it to where we could access it most efficiently, but left it around (and even updated it there) for when other people relied on it being there.
  • The vast majority of websites upgraded with no problems, errors or issues.

Note that all the stuff described below was done because a portion of our users was affected. At the same time, the vast majority was not affected, and in fact, saw an immediate speed improvement on their sites.

Picking our ORM

In choosing our ORM we were severely limited as we had to support PHP 5.6. The problem here is that while WordPress has only recently increased its minimum requirements to PHP 5.6, the rest of the PHP world considers that version ancient history. We would have preferred to use Doctrine, but unfortunately, due to this PHP 5.6 requirement, this was unfeasible. Which is why we ended up with Idiorm. It’s not perfect, but it was still a lot less work than building our own. We also started using Ruckusing for database migrations. This allows us to easily set up and change our database schema.

Idiorm and Ruckusing both had a dependency though: they are built on top of PDO, a way of connecting to the database that is slightly different from the mysqli and mysql interfaces WordPress normally uses in wpdb. And this is where we made the wrong decision: we decided to open a second connection to the database, using PDO, bypassing wpdb.

Releasing 14.0

Timeline of events

As we discovered issues and edge-cases, our team worked tirelessly to diagnose, debug, and release patches. This meant that we launched multiple small updates, which resulted in some users having to (re)index their sites multiple times. We’re sorry about the inconvenience that caused.

  • April 28th, 08:22 – Released 14.0
  • April 28th, 17:35 – Released 14.0.1
  • April 29th, 13:00 – Released 14.0.2
  • April 30th, 14:04 – Released 14.0.3
  • April 30th, 17:05 – Released 14.0.4

The full details of each of these releases (and their changelogs) can be found at our plugin page on wordpress.org.

We thought we were ready to release our 14.0 update. We’d tested this extensively, across multiple configurations and systems, and had asked hosts and other plugin developers all over the world to test along with us. We can’t always catch every obscure edge-case, but we weren’t really expecting a lot of problems. Well. We were wrong. Unfortunately, when you have the number of installs we have, even a problem that affects 0.1% of our users becomes an iceberg.

It turns out loads of people have very non-standard DB connections. Ranging from setups where DB_HOST or DB_CHARSET aren’t defined, to larger issues where they are running encrypted database connections, which of course our PDO solution didn’t know about.

We encountered lots and lots of small little issues. We could have fixed all that, potentially, but we’d be spending a lot of time doing that and new issues would inevitably keep popping up. So on Tuesday we made a very hard call. We released version 14.0.1 with fixes for all the things we could fix immediately, and then started working on the bigger and more complex challenges, which became version 14.0.2.

The fix, 14.0.2

So we started working on a better solution: still using Idiorm, but passing it all through wpdb. We’ve forked Idiorm and we’ve “semi”-forked Ruckusing: both now connect to the database through wpdb. This was quite a bit of work, but in hindsight, it wasn’t as bad as we thought it would be. Had we known we could do it “this easily”, we would have immediately made that step and used this approach – but this is new territory; there are few WordPress plugins which extend the platform in this way, at this scale, with this number of unknowns.

The results

The upside of not connecting through wpdb was that it had none of the wpdb overhead. On the other hand, we no longer have to open a connection. In our tests, we see mixed results, but overall, not a whole lot of impact. The added positive side effect is that we can once again inspect our queries through our favorite development plugin, Query Monitor.

Next problem: updates not working as they should

The next problem we ran into was a vexing one: some people were updating to Yoast SEO and they had all the new files, but they were reporting errors with files that no longer existed in the plugin. Note that this does not happen if you update through the WordPress admin. That was baffling. In the end, it turned out that the problem was that some of the WordPress site management tools simply copy files over the old files, instead of removing the directory entirely and replacing it with a directory that contains the new version. This caused errors, but it took a while to diagnose what was going on. That’s because this isn’t how modern hosting or plugin systems should behave, so it never crossed our mind that they would behave in that way. We got there with the kind help of Ipstenu and Otto in the WordPress forums team, though. Note that the upgrade process for plugins is dealt with by WordPress core, not by our plugin, so this was out of our hands. Regardless, we had to find a workaround.

Because of how Ruckusing, which I mentioned above, works, we included files in a directory to run migrations. If the old files in that directory didn’t get deleted, they’d reference classes that we had since removed, and everything breaks. To fix that, we simply renamed that directory, which prevents this from happening for now, and released 14.0.3, this afternoon.

wpdb bites back

Lazy loading

When you update to 14.0 or later, we add a row to our indexables table for every post, page, tag etc. on your site. We can either do this in one go, through our indexing process, or we do it lazily. Every time a URL for which there is no indexable is loaded, we create it, and store it. The next time you open that URL it can immediately take the data from our table and thus be faster.

And then came the final problem, on Thursday 30th April, which caused us to do yet another update. Turns out that when you wpdb::prepare a query which has null in it, wpdb escapes it into 0, which turns out to be a long standing issue in WordPress core. This is typically something we would have caught if we had not fixed this in a patch release, so had we made our decision to use wpdb sooner, we wouldn’t have caught this problem this late. Unfortunately, now it meant that we were storing the wrong values for noindex, nofollow and other robots meta tags in the wrong way. So… We had to do another patch release, 14.0.4 and re-index everybody’s sites to make sure we stored the data right. The re-indexing is possible because we left all the data in wp_postmeta, a decision I’m now very thankful we made.

Is it worth it?

After reading all this, you might think: is this worth it all? And despite the fact that I and a part of our dev team have been working incredibly hard the last 3 days, the answer is still a resounding: yes! We’re laying a foundation for a much better future and we’re making sites instantly faster. I hope we’ve caught all the major issues now, something I of course can’t guarantee, and once again, I’m sorry if we caused you trouble!

Coming up next!


13 Responses to Yoast SEO 14.0.x; or “Why you should never bypass wpdb”

  1. Rob
    Rob  • 4 years ago

    Dear friends,
    What’s happening…….
    One of the best (dutch) plugins SEO by Yoast……I guess you read the reviews and your score is dropping to almost a zero rating……And no comment on your site about this.
    What I see is a copy past answer for every problem presented by your costumers…….It would be responsible (business wise) to put a special memo, bulletin about this Problem on your site.
    Seriously these customers are not all wrong………
    Anyhow I do believe you do the best you can, but is it possible you communicate a bit more…..
    Instead we are doing fine and these are minor issues…….

    Greetings Rob de Koning

    • Willemien Hallebeek
      Willemien Hallebeek  • 4 years ago

      Hi Rob, We’re very sorry for the issues we caused. Like Joost says above, there are so many sites using Yoast SEO that even if a small percentage of the users experience problems, lots of sites are affected. We do feel bad for every site and take these issues very seriously though. We’ve also worked hard and are still doing so to fix issues asap, in patch releases, and our current 14.1 release. Thanks for your suggestions on improving our communications, we take them to heart.

  2. Barbara
    Barbara  • 4 years ago

    NO MERCY!! :) No i’m joking.. when i saw all these problems i deactivated the plugiin and waited for some days. Fortunatly your developer team answered also in forums so people understood you where working on it and was everything ok ;) Maybe creating a team of 50 webmaster that could be your test for 24h before releasing an upgrade could solve this problem next time.

    • Edwin Toonen
      Edwin Toonen  • 4 years ago

      You had me there, Barbara. We’re working diligently to improve the plugin and are getting ready for our next release on Tuesday. We had a lot of people and companies testing Yoast SEO 14.0, but it’s very hard to catch everything. Still, we’re hoping to do better on our next releases. Thanks for sticking with us!

  3. Ashish Sharma
    Ashish Sharma  • 4 years ago

    i’m also getting this error in WordPress I tried to reinstall the plugin and delete it as well and reinstall it but still getting this error I’m getting this error after the 14.0 updates and it’s still not resolved please fix it as soon as possible thanks

    “Yoast SEO was unable to create the database tables required and as such will not function correctly.
    Please read this help article to find out how to resolve this problem.”

    please release a patch of this error ASAP!
    it’s really causing me problems a lot

    • Edwin Toonen
      Edwin Toonen  • 4 years ago

      Hi Ashish. I’m sorry we’ve caused you problems. Have you updated to the latest version? If that didn’t work, could you try to reset the indexables tables and migrations by using the Yoast Test Helper plugin? Once you install and activate the relevant plugin, please go to the Tools → Yoast Test → Yoast SEO → Reset indexables tables & migrations options to reset it.

  4. Vincent
    Vincent  • 4 years ago

    Hi Yoast, after reading this I have a few questions: a lot of trouble and you say it makes sense as it makes sites faster. But does it actually? When all sane WordPress users have caching to speed up their sites?

    Were there only no-index meta’s removed or also added: in other words was there a chance the plugin de-indexed, parts of a, website?

    Great work on the transparency. Cheers

  5. Sean
    Sean  • 4 years ago

    This is what I’ve been thinking, not having experienced any issues with the latest Yoast updates, and that is way too many developers insist on hanging on to outdated, complicated legacy systems and refuse to standardise and move forward with the rest of the world. When the rest of the world does move forward, problems inevitably occur between the new and old systems, yet those still hanging on to said old systems are the first ones to cry fowl. But you do have backups, right?

    So yes, I have seen the light and I am taking Yoast’s side on this one.

    • Curmudgeon
      Curmudgeon  • 4 years ago

      That’s assuming that the “legacy” was more complicated than the “modern” one.
      Which quite frankly is not always true.
      But anyway.

  6. Roger
    Roger  • 4 years ago

    still not work on my site hd to deaktivate.

    • R M Charlton
      R M Charlton  • 4 years ago

      I’m glad you write this and I read it eventually. All the updates and then the constant problems in forums still made me think you guys had lost the plot but now I’m very understanding and even excited again about the prospects of your new indexables.

      It helped me a lot to understand why there were many updates and also why some of the common and terrifying sounding reported problems were happening. It even helped realising that part of the trust and brand doubt is because most of the internet uses yoast, so the level of reported incidents makes more sense now.

      I think it would help you a great deal if you make a link to this explanation post a lot more visible. Maybe sticky for now at the top of the WordPress support forum, at the top of all your landing pages with any traffic and as a standard 1st sentence to any replies to any reviews. People are watching those places like the news at the moment, in disbelief and also to see when the issues stop being reported, not realising how small % are now actually having issues.

      https://arcproject.org.uk/

      • Edwin Toonen
        Edwin Toonen  • 4 years ago

        Thanks for the kind words, RM! And thanks for the suggestion to spread the message.

  7. David McDonald
    David McDonald  • 4 years ago

    It looks like Yoast SEO Premium (which deactivates Yoast SEO standard plugin) does not activate with the new YoastSEO() surfaces.

    If this is the case, how do we do access post_meta now if we are running Yoast SEO Premium?