Migrating Yoast.com’s webshop from EDD to WooCommerce

At Yoast, we optimize every aspect of websites’ performance. Our goal is to make the web a better place by making websites more usable, easier to navigate, faster, and more reliable. In September 2017 we migrated our webshop from Easy Digital Downloads (EDD) to WooCommerce — it’s where we sell all the tools, products, and content available on yoast.com. In this post I’ll explain why we did it, what we built, and what the benefits will be for us and for our users in the future. WooCommerce has provided us the technical foundation to build on for years to come.

This article originally appeared on the WooCommerce blog »

Why the change?

Yoast.com is experiencing incredible growth driven by two main components — the popularity of our plugin, and the expanding audience of our SEO blog. The combination of these two elements, however, makes yoast.com a challenging website to manage.

Historically, the site has evolved from a simple WordPress installation. Easy Digital Downloads (EDD) was added by Yoast founder Joost de Valk to facilitate selling premium plugins. We also used EDD to serve updates for all the Premium installations out there in the world.

Over the years we’ve made more and more tweaks and relied on hacks to make sure EDD could still serve our needs. Making it possible for our customers to pay in either euros or dollars, for example, was an enormous effort. Since we wanted to add other currencies in the future, we needed to find a different solution.

When we looked into other options, we considered several needs:

  • Being able to support multi-currency purchases.
  • Being able to support recurring payments.
  • The cost of building integration and the maintainability of the platform.
  • Being able to support user accounts.
  • Creating an SEO platform.
  • Taking into account future compatibility, since we want to have a site that will still work in 5-10 years.

To achieve all of the above, we had to come up with a plan.

Ideation

In January 2017 we scheduled a meeting with our team of architects: Joost, Omar, Jip, and myself. We discussed what our needs were and how we could best meet them.

Our first conclusion was that we didn’t want to move away from WordPress. Even though we now have plugins for other platforms, we are WordPress fans and deeply care about its mission. WordPress is where our roots are, and it has worked very well for us as a CMS.

Why WooCommerce?

We then considered which eCommerce platform we wanted to use. We asked ourselves what the best eCommerce solution for WordPress is, and came to the conclusion that it’s WooCommerce. But, like EDD, WooCommerce didn’t support multiple currencies. So we needed a solution.

I’m a big fan of the idea that everything is a remix. We were inspired by the idea of using a different site for each language, an idea that’s been executed well by the MultilingualPress plugin. So we remixed it: to avoid having to write a ton of custom code, we decided to use a multisite installation.

As a result, WooCommerce doesn’t need to manage different currencies. On the dollar site, WooCommerce does everything in dollars. On the euro site, WooCommerce does everything in euros. This also makes it relatively easy to add a new currency. We’d just add a new site and copy the settings. In combination with MultilingualPress, in the future this will also make it possible for us to support different languages.

yoast.com woocommerce currency switcher

yoast.com offers users a choice between U.S. dollars and euro, via a switcher

WooCommerce ended up being a great fit for yoast.com for several reasons:

  • A good data model.
  • A large ecosystem.
  • Built-in REST API and Webhook support.
  • The possibility to dogfood our plugin in combination with WooCommerce.
  • WooCommerce solves the eCommerce domain. We didn’t want to spend much time working on the eCommerce domain. SEO is our expertise, so we want to focus on SEO.
  • WooCommerce has a vibrant community in which we can participate.

eCommerce: SKUs, historical records, and refunds

I want to focus for a moment on the eCommerce domain. One aspect to consider here is the SKU. Every product has an SKU — a unique identifier. An SKU might sound like something superfluous. Why do I need an SKU when I have an ID in the database? But then you realize that every financial department of every company already has this system in place. SKUs are not auto-incremented — you need a way to track products in a way that is unambiguous. Names are not suited for this. So on yoast.com the SKU is now the same number our finance department uses to track products.

Another requirement was to have a historical record of every purchase. WooCommerce does this by default. Once an order is complete, that’s it. All the data is immutable. If a customer changes their name, address, or email, the new information is only used for new orders. For a developer this may sound strange, but it is actually a strength. You wouldn’t want your bank to change transactions after the fact either — once they are done, they are done. Any mutation is a new transaction.

Having a historical record makes the biggest difference when it comes to refunds. In the old setup, refunded orders would disappear from our exports, which would change that month’s revenue. In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

In WooCommerce, every refund has its own date and amount. So once a month is done, it is really done.

Technical details of migrating Yoast.com to WooCommerce

Migration
We have a lot of historical data, which meant we had to migrate a large number of orders. We chose a tool called pandas, a tool designed to handle large datasets, and a good choice for migrating it all at once. If we planned another migration we would probably go in a different direction — we would slowly format historical data over months. That would drastically reduce the amount of data to migrate on the day of the actual move.

There was one main pain point that made the process more complex than it needed to be. Because all the data was saved in one meta value in the database, it needed to be unserialized. Python doesn’t have an efficient and correct way to do PHP unserialization, so we ended up shelling out to PHP to be able to unserialize this data.

Aggregating the data
You might have noticed that we now have two sites. But we still want to have an overview of all the data in one location. This is one of the reasons we build MyYoast. All orders we receive are synched to MyYoast. This means that MyYoast knows everything.

WooCommerce has a system to sync all orders: webhooks. The biggest downside of webhooks, in general, is that if the receiving system is down, the webhook is not received. When looking at webhooks’ code, you find the following snippet:

/*
Let other plugins intercept deliver for some messages queue like rabbit/zeromq
*/
return apply_filters( 'woocommerce_webhook_should_deliver', $should_deliver, $this, $arg );

We considered using a message queue as suggested in the comment, but we went with a simpler approach.

There is a delayed_job project from the Ruby on Rails community. We found a PHP port which suited our needs. Delayed job does what its name suggests: it schedules a job to be executed later. This means a job can never be lost. And if the job fails it will be retried four more times. After that, the job can be used for debugging what went wrong. This gives us a really robust setup.

yoast woocommerce webhookjobs

We configured WooCommerce to try to send webhooks to our custom dummy URL: http://my-yoast-job.url. This ensures that we can catch these requests from the code. We have one Scheduler class which is responsible to schedule jobs. It catches the webhook requests and turns them into jobs instead.

yoast woocommerce jobclasses

We have several classes that handle jobs, one for each of the fundamental building blocks of yoast.com. The jobs are then handled by a worker in the background. This proces is quick, so a customer will instantly see their products in MyYoast.

WordPress as an application
If you have never read The Twelve-Factor App, I would highly recommend you do so. It suggests a very robust framework for web-application development. How does this translate to a WordPress context? Better than you might think.

WordPress has a few unique features that make it harder to set up. But you can work around those to get a robust WordPress installation. If you only do one of these, it should be the dependencies factor. Start using Composer! It is here, and it works. Rarst created a good overview of how to use Composer for WordPress development. Auto-updating is not an option for a complex web application. So you need a different solution, and Composer does a very good job.

Performance
We started out without using object cache. That turned out to be a bad idea. If you handle a substantial volume of sales using WooCommerce you need object caching. Without it, an order model needs to be completely reinstantiated on every request concerning that order. It’s very costly, because the order model is required on a many pages. At the very minimum, all checkout pages need to access the order.

We also created a few pull requests on WooCommerce itself to improve performance without caching. One of these was a modification to cache the currencies. Before the modification, all currencies would be translated every time any piece of code needed just one currency. That is very inefficient considering that each shop only runs in one currency.

One thing we could not simply fix inside WooCommerce itself was searching for orders and subscriptions. Doing this would result in a query that could take down our whole website — this query contains many searches for post meta, and those take a long time. We worked around this issue by building our own search functionality on top of MyYoast.

The meta search issue would also be solved if WooCommerce used custom tables, which is luckily already on the WooCommerce technical roadmap. This issue also informed our thinking about our own plugin. Can we reduce the amount of meta keys that Yoast SEO creates? As a result, we’ve added the creation of a custom table to our own technical roadmap, too.

Current list of WordPress plugins and WooCommerce extensions
To give a comprehensive idea of where we landed, here is the list of plugins we currently use that are relevant to our site’s function as a store. The full list contains 55 active plugins.

Current list of WordPress plugins and WooCommerce extensions

Takeaways and results

We launched this new platform on August 29th, 2017. In the weeks that followed we have greatly improved the site’s performance. We’ve achieved all of the goals we wanted to hit. Since the migration we have also started to build a lot of other things. Our support team now has a much easier time refunding and transferring accounts. We have migrated our Yoast Academy to the multisite. And we have a bunch of exciting plans for 2018, including:

  • Adding a shop where we sell in British pounds. Thanks to our past choices and our move to WooCommerce, we can do this in a day, it will require almost no additional code.
  • Building a sales dashboard for our sales team.
  • Adding Composer support to our premium plugins.

WooCommerce has given Yoast the technical foundation on which to build and improve our platform into the next decade. Are you considering migrating an existing webshop to another platform? Have you done this already? Share your experience in the comments.

Read more: Why every website needs Yoast SEO »

Make Yoast SEO & Woo work together smoothly

  • Make your products stand out in Google
  • Get more buyers to your online store
  • Make products easier to find on your site
  • Have a cleaner XML sitemap & more
More info

6 Responses to Migrating Yoast.com’s webshop from EDD to WooCommerce

  1. Chris
    Chris  • 9 months ago

    Great article, Anton.
    Thank you.

    Did I get you right? You have 55 plugins running to get this solution? Man, WordPress indeed is awesome. But it comes with a terrible nightmare called plugin conflict.

    How can you sleep with that monster hiding behind every corner? Would love to hear your thoughts on this.

    • Anton Timmermans

      Thank you.

      Yes, at the time of writing we were running 55 plugins. Of course, we also have to deal with plugin conflicts. We have few ways of managing this:

      * We update all our plugins using git and Composer, that means we always know which plugin caused a conflict.
      * We look at the code quality of plugins we use. Higher quality plugins have fewer plugin compatibilities. This is because they are written more robustly and are less likely to fall over if something isn’t exactly as the plugin expects.

  2. kenny
    kenny  • 10 months ago

    Thanks for this breakdown of your plugins and thinking behind them Yoast. I was also wanting to use woo commerce as opposed to EDD, but simply for the practicality of being able to do physical products should the need arise… (like the Yoast t shirts?).
    Thanks for always being so open with your information.

    • Anton Timmermans

      Always happy to share our thought processes.

      Great to hear you are considering the same switch Kenny. I agree that WooCommerce is also really great to sell physical products.

  3. melanieloomos
    melanieloomos  • 10 months ago

    I am so glad I found you guys!
    I did not know about the lack of record-keeping on the internet; and now I understand why Woo is required by a lot of companies.
    It is odd to me that transactions were changed backward in time by default historically. I totally agree people would not want their bank changing records backward in time and a merchant should not be able to do that.
    Right now, I am in the process of re-doing my old web site that I printed on actual paper when I took it down, and didn’t know you had other plug-ins. Very excited to hear you have a Woo / Yoast / Word Press Plug-In!!!

    • Anton Timmermans

      We are glad to have you! 😀


Check out our must read articles about WooCommerce