YoastSEO.js design decisions

In the upcoming 3.0 release of Yoast SEO (for WordPress), we’re moving the entire content analysis from the server to the client side. This way our users can get instant feedback on the content they’re writing, while they’re writing it. In the past six months we’ve been working hard on a JavaScript library called YoastSEO.js that is going to replace a ton of ancient PHP code. This transition brings along quite some challenges, most importantly with regard to plugins and themes that integrate with us.

The problem

The old content analysis contained a couple of widely used WordPress filters that other plugins could hook into. The most widely used filter was wpseo_pre_analysis_post_content which was a hook others could use to alter the content before it was analyzed. Such hooks are meant for others to use, they’re part of an officially maintained API. Removing those will cause dozens of integrations to no longer work. Then again, we’re simply removing PHP code and replacing it with JavaScript. Should we still support hooks used in code that doesn’t even exist anymore? Is that even possible?

Can we retain the WordPress filters?

Let’s say hooks represent a possibility to manipulate or integrate with behavior in a structured way. In this case, the behavior is not gone, it has only been moved from the server to the browser. If hooks represent a possibility to manipulate or integrate with behavior, then WordPress hooks represent a possibility to manipulate or integrate with behavior on a server, within the context of a WordPress app. If we want such hooks to have access to behavior in the browser, we can of course solve that through AJAX. We simply take a trip down the server every time we need the server to have a say in things, right? So the technical answer is yes, it is possible to retain the WordPress filters.

Should we retain the WordPress filters?

Let’s start investigating the hybrid solution in which we use AJAX to run the server side hooks. We’ll focus again on the wpseo_pre_analysis_post_content filter. It is fired every time an analysis is run and allows others to add or alter something in the content before it is analyzed. A common use case was for example to parse certain shortcodes in the content before it is analyzed. We were now confronted with 3 major issues:

1. In the old situation, the content analysis was only done once per request. With the new content analysis tool, it could be done hundreds of times while your are editing your content. The server would start receiving a ton of extra requests for every time someone is editing content.
2. A hard requirement for the analysis was for it to be blazing fast and performed synchronously. AJAX calls are slow and asynchronous by definition. It would be virtually impossible to keep the analysis fast and consistent at the same time, while maintaining a trip down the server on every run.
3. The idea of going hybrid just doesn’t feel right. It’s simply a less maintainable and far too coupled approach. One of the big benefits of moving this functionality over to the client side, is also that we’re making our content analysis tool portable to other platforms. Therefore we want to decouple from the underlying platform (in this case WordPress) as much as possible.

All of the considerations mentioned above made us decide we are going to completely remove the WordPress filters and come up with a client side alternative instead.

Gathering requirements

Of course we still wanted the content analysis to be pluggable. By investigating the option of a hybrid solution, three major requirements had already become clear:

• We needed YoastSEO.js to have its own hook mechanism.
• We needed hooks to be synchronous (fast).
• We needed to guarantee consistency across different runs of the analyzer.

We also had to accept that data from the server is often still required to be able to alter the content. Some data can be fetched once and doesn’t change anymore during the process of content editing, while some data needs to be generated and fetched dynamically during the process of content editing. A third option is that the data is static, but there is just a lot of it and you only need a small subset in the browser to be able to integrate with the content analysis. A couple of extra considerations came up:

• WordPress has a way of localizing data for usage in the browser called wp_localize_script. This is a great way of making data from the server available in the browser. However, it would be better if plugins would not have to slow down the page request on the content editing pages for the sake of integrating with our content analysis tool.
• We cannot allow plugins to dynamically fetch data from the server while hooking into our analysis. That breaks the requirement of having synchronous analysis and could impact constistency negatively. We need to come up with an alternative way of dealing with that use case.

Our solution

We ended up devising a client side API that takes into account all of the above considerations and requirements. The revelation for us was the idea of preloading. We decided we should allow some time for plugins to load their data. In that time they can register themselves with YoastSEO.js, register modification callbacks and preload the datasets they need to perform their modifications.

We encourage plugins to preload data with AJAX, and keep their data set as small as possible. The modification callbacks will only be executed if they are registered for a plugin that declared ready within the maximum load time of 3 seconds and only if they execute synchronously. A plugin is responsible for keeping his dataset up-to-date in the background. If a plugin needs to dynamically update its dataset during the process of content creation, it can simply tell YoastSEO.js it has reloaded and a new analysis will be run.

Go check it out

You can find the API documentation on GitHub. To minimize the impact of removing the WordPress filters, we have written a shortcode integration ourselves that parses all shortcodes in the content before it is analyzed. It actually covers all of the problems that were addressed in this post. If you have any questions or feedback you would like to share, be sure to drop us a line in the comments!