The story behind our Site Kit by Google integration

Yoast SEO offers a seamless integration with Site Kit by Google, and we’re so excited that we wanted to share some background information on the whats and hows of that ambitious technical feat.

What is the integration exactly?

With this new integration, site admins and SEO managers now have the ability to see data from Search Console and Analytics, right in their Yoast dashboard. Not only that, but they now get the opportunity to see a unique display of combined Google and Yoast data, aggregated in a way that has never been done before!

Once an administrator completes the onboarding process from Yoast’s dashboard, they are greeted with our new widgets, showing them analytics and search console metrics of their website. Those widgets, at this first iteration of the feature, are:

• A list of key search metrics (Impressions/Clicks/click-through rate/Average Position) of the last 28 days, combined with a comparison of the preceding period, to identify trends.
• A graph of the Organic Sessions of the last 28 days, combined with the comparison of the organic sessions of the preceding period.
• A list of the website’s top 5 search queries, which are the queries that produced the most clicks to the website in the last 28 days.
• A list of the website’s top 5 most popular content, which are the URLs that produced the most clicks in the last 28 days. This is also the widget that aggregates Google and Yoast data, which possibly makes it the most interesting one, especially from a technical standpoint.

Expanding on the top 5 most popular content widgets, Yoast SEO is able to match URLs coming from Google data with their respective WordPress entities whenever possible (posts, terms, archives, etc.), allowing the user to quickly edit each corresponding entity, where possible. More technical details about that are below.

This is where it gets really interesting. Yoast is also able to match those URLs with the SEO scores that might have been calculated for the corresponding posts. All that means that the admins have the ability to see their best-performing content while getting actionable information around them!

We’re very happy with the results of our first iteration of the integration, which is close to what we envisioned from the moment we started planning for it. In fact, there are plenty of reasons why that integration was always the natural next big feature of Yoast SEO. 

Site Kit by Google + Yoast SEO = the perfect match

Yoast’s historical position in the SEO circles of the WordPress ecosystem is something that would be sufficient to make an integration with Google’s WordPress plugin a natural next step. What’s more, the great overlap between Site Kit by Google and Yoast SEO users made this a “natural” integration.

But even beyond that, there are technical aspects of the two integrating systems that truly make this collaboration a mutually beneficial one. So, let’s dive into some of those:

How Yoast SEO takes advantage of Site Kit

One of the main benefits of integrating with Site Kit is how seamless the onboarding process can be for new users. Specifically, because Yoast SEO makes use of Site Kit’s secure and custom OAuth proxy service, it basically takes advantage of the past hard work the Site Kit team had put into building a seamless experience for users, one that doesn’t need them to create separate Google Cloud Platform projects. 

Due to the WordPress infrastructure, there’s no path to easily storing client secret API keys in the plugin, which means there’s an inherent challenge in authenticating with Google services. Yoast SEO being able to make use of the existing sophisticated authentication flow of Site Kit was a game-changer for us, in terms of the feasibility of the project.

What’s more, Site Kit has a feature that allows Yoast SEO to have a super-friendly flow for multi-user websites, the Dashboard Sharing feature. Imagine being an SEO manager in a website whose admin has already set up Site Kit access but you don’t have a Google account yourself. If the admin has already shared read-only access to Site Kit data, this means that you’re able to access the new Yoast SEO widgets without the need to authenticate yourself. 

Considering that our users often have delegated SEO managers who are not the admins of the websites themselves, hooking into Site Kit’s Dashboard Sharing makes this common use case work seamlessly out of the box, without much hassle.

How Site Kit is enhanced by Yoast SEO

Remember the top 5 most popular content widgets mentioned above? Time to actually explain why it’s so interesting, from a technical perspective!

We have to first consider that Google works with URLs, which means little in the WordPress context. Meaning, it’s tough matching URLs to WordPress, non-post entities like terms, archives, etc. (and even for posts, you are dependent on the url_to_postid() function that’s prone to errors and heavy calculations).

But our previous work with indexables paid dividends for us here. Actually, we couldn’t explain the problem better than the release post of the release that introduced the concept of indexables way back in 2020:

“Information on the web is addressable via URLs. Anything that has a URL is something that could be discovered, scraped, indexed, and shown in the search results. WordPress has posts, pages, custom post types, categories, tags, custom taxonomies, different types of archives, special pages, and maybe even more types of content. Do you think a search engine like Google cares about that? It doesn’t really. It just looks for things with a URL, that it can scrape and index.”

That major overhaul of our data model has served us well all these years, and it once again became prominent with this integration (thank you, past selves from 5 years ago!). It basically allowed us to translate the 5 most clicked URLs into WordPress entities. From the user’s perspective, that means allowing us to point users to the edit pages of those entities, turning that Google data into actionable insights!

On top of that, Yoast SEO applies its SEO Analysis to all relevant WordPress entities. This means that we were also able to match the top-performing URLs to SEO scores! The scores, calculated by our carefully crafted algorithms, being matched to Google’s data is giving a POV that has never been given before to site owners and we believe that it will be a powerful tool in their SEO strategic plans.

But even from a non-technical point of view, this integration was also greatly shaped by Yoast’s designated SEO team, which identified some of the most powerful metrics we can display, in a way that shares a complete and concise story of the website’s SEO recent performance. Site Kit by Google provides so much to explore and add, that we selected what we thought would be most useful for a first iteration of the feature and we think that this initial selection of widgets gives site owners a great summary of actionable and useful insights on where they can improve their SEO.

Despite those hand-in-glove attributes of the integration though, the importance of the project and its inherent technical restrictions meant that we faced numerous challenges that required hard technical decisions and even changes of direction during the implementation.

Want to know what tried to break our team’s spirit?

With a project of this size and complexity, there were a handful of technical decisions to be made that would directly impact the success of the project. 

The impact of those decisions was magnified by the sheer size of the Yoast userbase, the security and privacy considerations that Google data inherently comes with and the fact that no similar integration had been attempted before.

Note to selves: Make sure we don’t DDoS Google APIs

Having over 13 million installs is something we’re grateful for every day, but it’s also a responsibility that we don’t take lightly.

And exactly because of that huge userbase, we had to be extra careful not to overflow Google’s APIs with unprecedented traffic, because that could have meant outages and degraded performance of Google services, not only for our integration but potentially for other Google users too. No pressure, right?

So, aside from trying to work with best practices in mind and making our implementation as efficient as possible, which is a mindset we operate with anyway, we came up with two separate action points that we believed would alleviate concerns about all that.

First, it was decided that we would do a staggered rollout of the feature. Which basically means that we are gradually increasing the number of users who have access to the feature, instead of opening it up for over 13 million users. This is allowing us to identify early warning signs of over-usage of the Google APIs, which is vital to a project of this size.

Second, we understood that it was imperative to apply some sort of cache around our requests to the Google APIs. But that came with its own set of challenges.

A key aspect we had to consider was that since Google services deal with data that can be sensitive from a privacy perspective, we didn’t have the choice to store any data in local databases, considering the uncontrolled nature of WordPress APIs, database access (any plugin can literally do anything it wants) and general infrastructure. Which meant that we couldn’t use the WordPress Transient API, which would have been ideal otherwise, as it would have been able to combine the logic for caching and invalidating aggregated data very well. Those privacy restrictions prevented us from considering encryption as well.

So, we ended up implementing a frontend, browser caching layer, which is a solution that takes care of all privacy and security considerations. We even used different expiry times for aggregated data, to make sure we didn’t serve rapidly-changing data as stale.

But that wasn’t the end of the technical hurdles we had to overcome, as there was a lot of uncharted territory in terms of technical implementation that we had to navigate through.

No precedent made things interestingly tough (but close collaboration always wins!)

Since we were trying to integrate with Site Kit by Google in a way that wasn’t able to rely on documented technical processes, the collaboration between the Yoast and Site Kit teams proved to be a vital part of this project.

For example, the initial implementation of the requests to Site Kit’s APIs meant instantiating Site Kit classes, and we didn’t find out that those are not intended to be instantiated by third parties until late in the process. After getting the relevant recommendations from Site Kit’s developers, we switched to perform server-side HTTP requests to Site Kit’s REST endpoints. Even though that required a big refactor, rather late in the process, it was totally worth it, since it reduced coupling on Site Kit’s internal implementation. At the end, being as future-proof as possible made the extra effort worthwhile.

Another unique challenge that we had to overcome was that there are multiple states when users are getting onboarded and set up with Site Kit and we had to consider every single one of them, so that we could properly display our own onboarding wizard, for all sorts of users, admins and not.

Especially for multi-user setups, those numerous states meant a lot of cases to be dealt with. And the fact that developer documentation for those was scarce, we had to reverse-engineer extensively, which was time-consuming and error-prone. Luckily, another round of strong communication between Yoast and Site Kit dev teams managed to make our requirements clear, and that resulted in equally clear suggestions that were implemented, which made the whole solution robust and documented.

So, worth re-iterating: When in doubt (or struck by unprecedented circumstances), rely on communication!

Conclusion

All in all, integrating Site Kit by Google in Yoast SEO has been an incredible journey so far, presenting a unique set of challenges for our dev team, but also great opportunities to put the capabilities of our product to good use. 

We believe that the result of that will enable users to get great, actionable insights about the performance of their website and we’re looking forward to enhancing the integration even further in the future!

Read more: Site Kit by Google insights now available for all Yoast SEO Premium users »