How to automate repetitive daily tests

As a tester in a continuous development cycle, you know that you test your application not only before a release, but also on a daily basis. You’ll want to automate most of your basic (though very important) short repetitive test cases, which consume lots of your time when testing manually. That way, you can focus on other test tasks. When you deal with web applications, it’s useful to automate user interface use cases. UI automation very often receives the criticism that it is a development job to automate testing, not a task for a tester. Well, automation does not need to be complicated. With the following tool set – a test tool, GitHub and Travis CI accounts – we’ll have a look at how to set up an automation framework.

Define your test cases

First, you need to figure out which test cases to automate. Try to write them as simple as possible. These tests should not cover edge cases, as you want to have a 100% predictable result at the end of your test. In the case of WordPress, you can imagine that you have to test the same things all over again for all content types: posts, pages, taxonomies, custom post types, etc. So, when you have nearly the same object of testing, this is a very good candidate to be automated.

Test framework

The next step is to figure out which UI automation tool best suits your needs. The choice depends on your coding skills, working frameworks of your development teams and personal preferences, of course. There are a lot of out-of-the-box frameworks that help you to get started. At Yoast, we have chosen CodeceptJS with Puppeteer webdriver for automated testing of our plugins. Why? We want to use JavaScript and we want an easy and quick setup that works with Google Chrome directly, without requiring additional tools like ChromeDriver. In this case, we lose cross browser testing support, but it suits the goal of our testing.

After the choice is made, you proceed to the installation of a test framework. You need basically:

npm install codeceptjs puppeteer --save-dev

That is it, you just need to follow Quickstart documentation provided by CodeceptJS to get your framework ready for writing tests.

BDD-style syntax

You have probably heard about the Feature/Scenario approach in describing test cases. Well, it is indeed a very convenient way of storing your use cases, otherwise testers face the nightmare of giving unique names to tests. It is much easier to store your test under the full description of the steps in a clear language that everyone can understand. CodeceptJS supports BDD-style syntax, so automating your tests from input to result would be in a way of Feature/Scenario:

Feature( "[ConsoleLogs] In order to see Yoast plugin does not throw errors\n" +
       "As an Yoast SEO user\n" +
       "I need to check console logs" );


Scenario( "Given I am logged in WordPress admin\n" +
          "When I check console logs\n" +
          "Then no errors are shown",
async( I ) => {
    const logs = await I.grabBrowserLogs();
    const res = logs.filter( ( log ) => log.type() === "error" );
    assert.deepEqual( res, [], "Console contains errors" );
} );

The main principles in automating tests are: setUp, actions, tearDown. In setUp, you prepare your test conditions, in actions you run your test, in tearDown you clean the leftovers of your test actions. CodeceptJS supports Before and After hooks, which get triggered before and after your Scenarios. It also supports the Fail hook.

Tests output

Once you start running your new scenarios locally, you’ll notice that reading output on a console is a far from a nice user experience. That is why you can try one of the reporting tools. We have chosen mochawesome:

npm install --save-dev mochawesome

CodeceptJS supports reporting outputs so you will get your results in nice html files:

Continuous Integration setup

Up to now, you have run your new scenarios locally. But you are not going to use your machine to run them on a daily basis. As mentioned in the beginning, you have GitHub and Travis CI accounts. In this case, you can make a nice Continuous Integration system. Assuming your web application source code is pushed to GitHub, then you can also create a repository with your test framework in GitHub. Just add a .travis.yml to your repo and start making the basic configuration:

addons:
  chrome: stable
  hosts:
    - local.wordpress.test
language: php
sudo: required
dist: trusty
php:
  - 7.2
branches:
  only:
    - master

In our CI system, we fetch latest WordPress installation in before_install step and then we launch Chrome in headless mode:

- google-chrome-stable --headless --disable-gpu
--remote-debugging-port=9222 http://localhost &

The next supported Travis CI step is before_script, where we install and configure WordPress. As we develop plugins, here at Yoast, we also include building of our plugin to our CI system pointing which branch we want to use. Our goal is to benefit from early feedback of our development, so we run automated scenarios against the development branch of our plugins. In the step script you execute codeceptjs which triggers to run your Feature and Scenarios in a headless mode in Travis CI box.

Once you have tried mochawesome reporting, you’ll love using it for all scenario executions. So as a last step, you’re going to want to store test results of newly built CI system. You need to have a storage which you can reach from Travis CI. At Yoast, we send and store our test results in S3, which is possible to configure in .travis.yml 

deploy:
  provider: s3
  access_key_id: "YOUR AWS ACCESS KEY"
  secret_access_key: "YOUR AWS SECRET KEY"
  bucket: "S3 Bucket"

And, do not forget to add deployment of the results, also after failure, as by default the deploy step is being executed if your build succeeds in Travis CI. So, if some of your scenarios are failing, and you also want to receive test results, you add:

after_failure:
  - dpl --provider=s3 --access-key-id="YOUR AWS ACCESS KEY" --secret-access-key="YOUR AWS SECRET KEY" --bucket="S3 Bucket"

One last touch: if you have already started with the CI system, then you’ll want to implement the most important condition: tests are triggered automatically. At Yoast, we have decided to run on a nightly basis, and Travis CI supports cron option in Job Settings:

To sum up

There it is, you have now automated your basic tests that require time every day and that you can run locally on your machine. You store your tests in Features and Scenarios, thus keeping a clear structure of your web application behavior. The CI system runs on a nightly basis verifying development branch of your web application. And you can enjoy your coffee while reading your morning test reports!