Testing the new Yoast SEO settings interface

As you may already know, we recently released a brand new version of the Yoast SEO plugin with a new settings user interface. We covered the topic multiple times in different blog posts including one where we provided the thought process behind the redesign in general and, more specifically, why now. In this article, I as a QA Lead at Yoast, want to highlight all the various steps we took to ensure the quality of the new Settings UI.

This redesign not only looks impressive but also required a massive amount of collaboration between different subdivisions within the Yoast company. This was truly a team effort in many aspects – because we clearly understood the effect it will have on the user experience of Yoast SEO users.

It was utterly important for us to deliver a new vision for our settings as smoothly as possible. That kind of change was a little bit scary for us but we trust each member of our team – that everyone will do their best to deliver the best quality in each respective area: be it development, content, videos, support, etc.

Agile approach

When you rewrite a big part of a plugin, there are several approaches you can take:

1. Starting from scratch and completely removing the old parts.
2. Iterative improvements with both parts (new and old) being present in the plugin simultaneously.

Both approaches have their own pros and cons but, as the saying goes, we decided to eat the elephant one bite at a time. We started with a single page redesign that lived inside the plugin under a feature flag YOAST_SEO_NEW_SETTINGS_UI.

Now, what is the best way to test drastic changes in the plugin? For team Yoast, it’s deploying everything to a production environment with tons of everything: users, custom post types, taxonomies, etc. So we were dogfooding (meaning: using your own product to test) from the very beginning by using and testing everything on this same site – yoast.com. That’s what we always do with each and every release.

This approach allowed us to work on the redesign for months while still delivering new features and quality improvements, as well as bug fixes, to our users.

As part of our agile approach, we also shared an early alpha version of the new Settings UI with Yoast partners and various agencies. For anyone who was interested to give it a try, it was also shared on our developer blog. This gave them almost 2 months to update their documentation, videos, screenshots, etc. and make sure they are ready, compatible, and familiar with the big changes.

Company-wide acceptance testing

For final acceptance testing before the Yoast SEO 20.0 release, we decided to engage people from all departments, so everyone’s knowledge and experience could help ensure that we deliver the highest quality product.

The process started in December 2022, a month before the release day on January 24th, 2023. All the areas that require manual testing were converted into individual GitHub issues, and different team members or the whole departments have been assigned to one or several such tasks. So, all in all:

• 30+ people were actively involved in this final round of testing.
• Almost 70 test tasks were distributed across the involved people.
• With everyone involved, almost the full month (150+ hours) was spent on testing.

We paid attention to a lot of areas which I will outline below.

Functional testing

We wanted to make sure that there is no SEO impact for upgrading to v20. To achieve this we checked that all the options conform to our specifications. But not only that, backward compatibility has the utmost priority for us, so we were determined to make sure our users upgrading from the previous versions of the Yoast SEO plugin do not lose any settings as well.

Every single option and its combinations have been tested and re-tested to work properly in the new Settings UI. That’s why the testing required a team-wide effort for a prolonged period of time.

User experience (UX)

To make sure our changes were as good as possible, we had several rounds of UX review by different teams, including UX and support teams. The support team review was especially important because they deal with Yoast SEO users on a daily basis and they know their pain points, and can predict potential confusion. But in general, almost everyone on the Yoast team dedicated some time to test the user experience.

Obviously, the UX review was performed during the initial development phase, and also as a final check during the acceptance testing.

Accessibility

At Yoast, we always try to be as inclusive as possible. That includes the need to make our updated Settings pages accessible. So we asked our accessibility specialist Andrea Fercia (who is also part of the WordPress core Accessibility team) to review our updated Settings UI, and thanks to his feedback we introduced some very important improvements in this area.

Performance & security testing

Being installed on more than 13 million sites means that we should triple-check our changes to avoid any performance and/or security issues. Affecting even 1% of our customers results in more than 130 thousand sites potentially being insecure or working with degraded performance.

That’s why we have a dedicated team that helps the development team be mindful of all the changes. They tested and reviewed the new code and provided suggestions and ideas that were promptly implemented.

Multi-language support testing

The huge variety of sites in different parts of the world that use our product requires us to make sure that every big change is still working properly in languages other than WordPress’ default English. With the help of our dedicated linguistics team, we tested the plugin in a lot of different languages to make sure that all the strings are displayed properly in places where they should be, that search is working across different languages, etc.

Automated tests

Here at Yoast, we believe that automating repetitive tasks is the correct way to simplify our lives and free up additional time for creative work. That’s why we have an extensive list of automated tests that we use to check on a daily basis that major parts of our plugin user interface are working as expected. You can read more about our processes and reasoning in the How to automate repetitive daily tests post by my colleague Daria.

While working on the new Settings UI, we adjusted our test cases to be compatible with the latest changes. This way we had an old set of tests running for Yoast SEO 19.x versions of the plugin, and a new set created specifically for the redesign and planned to be released in 20.0 – all running and used at the same time. Better safe than sorry.

Post-release support and monitoring

Despite all of our efforts and energy put into developing and testing the new Settings UI, there is always the possibility of a compatibility issue or other issues due to the number of sites using our plugin. There are literally millions of variations of plugins, themes, custom code snippets, etc. all potentially affecting the way the Yoast SEO plugin works in each individual case.

That’s why right after any release we closely monitor any issues being mentioned across various forums and social networks to catch any potential problems that our users face. We are prepared to promptly react and support anyone having complications. We hope you enjoy the new Yoast SEO settings interface and if you do encounter any bugs, make sure to let our support team know!