Yoast SEO 14.0.x; or “Why you should never bypass wpdb”

We thought we were ready to release our 14.0 update. We’d tested this extensively, across multiple configurations and systems, and had asked hosts and other plugin developers all over the world to test along with us. We can’t always catch every obscure edge-case, but we weren’t really expecting a lot of problems. Well. We were wrong. Unfortunately, when you have the number of installs we have, even a problem that affects 0.1% of our users becomes an iceberg.

It turns out loads of people have very non-standard DB connections. Ranging from setups where DB_HOST or DB_CHARSET aren’t defined, to larger issues where they are running encrypted database connections, which of course our PDO solution didn’t know about.

We encountered lots and lots of small little issues. We could have fixed all that, potentially, but we’d be spending a lot of time doing that and new issues would inevitably keep popping up. So on Tuesday we made a very hard call. We released version 14.0.1 with fixes for all the things we could fix immediately, and then started working on the bigger and more complex challenges, which became version 14.0.2.

The fix, 14.0.2

So we started working on a better solution: still using Idiorm, but passing it all through wpdb. We’ve forked Idiorm and we’ve “semi”-forked Ruckusing: both now connect to the database through wpdb. This was quite a bit of work, but in hindsight, it wasn’t as bad as we thought it would be. Had we known we could do it “this easily”, we would have immediately made that step and used this approach – but this is new territory; there are few WordPress plugins which extend the platform in this way, at this scale, with this number of unknowns.

The results

The upside of not connecting through wpdb was that it had none of the wpdb overhead. On the other hand, we no longer have to open a connection. In our tests, we see mixed results, but overall, not a whole lot of impact. The added positive side effect is that we can once again inspect our queries through our favorite development plugin, Query Monitor.

Next problem: updates not working as they should

The next problem we ran into was a vexing one: some people were updating to Yoast SEO and they had all the new files, but they were reporting errors with files that no longer existed in the plugin. Note that this does not happen if you update through the WordPress admin. That was baffling. In the end, it turned out that the problem was that some of the WordPress site management tools simply copy files over the old files, instead of removing the directory entirely and replacing it with a directory that contains the new version. This caused errors, but it took a while to diagnose what was going on. That’s because this isn’t how modern hosting or plugin systems should behave, so it never crossed our mind that they would behave in that way. We got there with the kind help of Ipstenu and Otto in the WordPress forums team, though. Note that the upgrade process for plugins is dealt with by WordPress core, not by our plugin, so this was out of our hands. Regardless, we had to find a workaround.

Because of how Ruckusing, which I mentioned above, works, we included files in a directory to run migrations. If the old files in that directory didn’t get deleted, they’d reference classes that we had since removed, and everything breaks. To fix that, we simply renamed that directory, which prevents this from happening for now, and released 14.0.3, this afternoon.

wpdb bites back

And then came the final problem, on Thursday 30th April, which caused us to do yet another update. Turns out that when you wpdb::prepare a query which has null in it, wpdb escapes it into 0, which turns out to be a long standing issue in WordPress core. This is typically something we would have caught if we had not fixed this in a patch release, so had we made our decision to use wpdb sooner, we wouldn’t have caught this problem this late. Unfortunately, now it meant that we were storing the wrong values for noindex, nofollow and other robots meta tags in the wrong way. So… We had to do another patch release, 14.0.4 and re-index everybody’s sites to make sure we stored the data right. The re-indexing is possible because we left all the data in wp_postmeta, a decision I’m now very thankful we made.

Is it worth it?

After reading all this, you might think: is this worth it all? And despite the fact that I and a part of our dev team have been working incredibly hard the last 3 days, the answer is still a resounding: yes! We’re laying a foundation for a much better future and we’re making sites instantly faster. I hope we’ve caught all the major issues now, something I of course can’t guarantee, and once again, I’m sorry if we caused you trouble!