Scaling visual testing with BackstopJS (Part 1)

As a free and open source tool for conducting visual tests, BackstopJS is a good way to start exploring visual testing before buying into one of the many paid services. I’ve found it does face some limitations in scaling to even a modest number of scenarios, for two reasons:

1. Testing scenarios are defined in a single flat config file.
2. Test results are presented in a single long report.

Interestingly, these aren’t just a limitation of it being free; I’ve seen paid tools that scale just as poorly. But luckily, as an open source tool, we have more flexibility than with other tools if we’re willing to put the work in.

In this post, I’m going to walk through how I get around the first problem, by introducing a way of breaking up the config. Later, I’ll post a second part to address the reporting problem.

Better organization of test cases

In vanilla BackstopJS, the config file might look something like this:

{
    "viewports": [{
        "name": "mobile",
        "width": 320,
        "height": 480
    },{
        "name": "desktop",
        "width": 1920,
        "height": 1080
    }],
    "paths": {
        "bitmaps_reference": "backstop_data/bitmaps_reference",
        "bitmaps_test": "backstop_data/bitmaps_test",
        "html_report": "backstop_data/html_report",
        "ci_report": "backstop_data/ci_report"
    },
    "scenarios": [{
        "label": "first",
        "url": "http://example.com/first.html"
    }, {
        "label": "second",
        "url": "http://example.com/second.html"
    }]
}

This has two tests in the scenarios array, that will be run on two viewports, with output images and reports saved in the directories set by the paths object. This is fine for a few cases, but very quickly you run into scenarios like the following:

• If not all tests should run on the same set of viewports, you need to start defining a new viewport object in every scenario that differs from the default, even if they’re all the same.
• If scenarios need special properties, like ignoring a certain css selector or waiting for a particular selector to be visible, that too needs to be added to every scenario that needs it.
• As you start approaching even a dozen or two dozens scenarios, how do you order them in the scenarios array? How do you find tests that are in there? How do you accurately express the purpose of each test?
• If you have multiple domains you use for testing, such as separate staging and development environments, there’s no easy way to change all the hard coded urls.

A lot of this boils down to the fact that the entire config is just a giant JSON. The solution is to provide all this same information in a more manageable format, and then compile a config for BackstopJS to consume.

A new format for scenarios

To introduce some level of organization, reduce duplication of config information, and allow more flexibility, we’re going to set up configs as multiple JS files that can later be imported. Each one looks like this:

module.exports = {
    "someSharedProperty": "value",
    "scenarios": [{
        "label": "first",
        "url": `${process.env.BASE_URL}/first.html`
    }, {
        "label": "second",
        "url": `${process.env.BASE_URL}/second.html`
    }] 
}

We’re using an environment variable, BASE_URL, to tell Backstop which domain to test. Changing the domain to be tested is as simple as changing this value. We’ve also abstracted out someSharedProperty – it might be delay, readySelector, or any of the other scenario properties that Backstop looks for. When we parse this new config object, we apply any properties in it to each of the scenario objects, to construct the flat scenario objects that Backstop will understand:

const getScenarios = config => {
    const scenarios = config.scenarios;

    // pull out all the properties in this config other than scenarios
    const meta = {};
    for (property in config) {
        if (
            config.hasOwnProperty(property) 
            && property !== "scenarios"
        ) {
            meta[property] = config[property];
        }
    }

    // now apply those properties to the scenarios
    return scenarios.map(scenario => {
        return Object.assign({}, meta, scenario);
    });
};

For the above config, this returns an array of two objects, both of which contain a copy of the someSharedProperty key. Because of the order of the final Object.assign({}. meta, scenario) call, we still allow each scenario to override the defaults at the top of the file where needed.

This format also allows us to add more complicated logic if we need it, extracting any duplicate information into variables, and adding comments to explain more thoroughly what each test is actually for to help others understand the coverage.

Combining the new config files

Let’s say now we’ve got a series of these new, smaller config files, organized like so:

tests/
  - homepage.js
  - profile.js
  - widgets/
      - one.js
      - two.js
  - gizmos/
      - alpha.js
      - beta.js

This relatively simple site has a homepage, a profile page, two widgets and two gizmos, each with their own test scenarios. In our script, we need to find all these files and combine them. Let’s use a glob pattern:

const glob = require("glob");
const pattern = "./tests/**.js";
const configFiles = glob.sync(pattern); 

And then, we can loop over all the files and get their scenarios using our getScenarios function from earlier:

const allScenarios = configFiles.reduce((scenarios, filename) => {
    const thisPath = path.join(process.cwd(), filename);
    const thisConfig = require(thisPath);
    const theseScenarios = getScenarios(thisConfig);
    scenarios.push(...theseScenarios);
    return scenarios;
}, []);

Since simply iterates over the entire list of files, imports that module, parses it into an array of vanilla Backstop scenarios, and pushes them all into one big array.

To help us understand where each test in the results came from, and ensure we don’t accidentally have duplicate labels on scenarios, it’s good to add an extra bit before the scenarios.push line to set the labels:

theseScenarios.forEach(scenario => {
    labelPrefix = filename.replace(/^\.\//, "").replace(/^\//, "");
    scenario.label = labelPrefix + "/" + scenario.label;
});

This uses the filename, stripping off any leading / or ./, so that the scenario label reflects the file it was found in. For example, if one of the scenarios in gizmos/alpha.js contains this label:

    "label": "a very interesting scenario"

then the object in the allScenarios array will actually contain:

    "label": "gizmos/alpha.js/a very interesting scenario"

Creating the final Backstop config

Now with our array of scenarios, all that remains is to create the config file that Backstop can read. For that, we still use Backstop’s config file, with one difference: it doesn’t contain the scenarios property any more:

{
    "viewports": [ ... ],
    "paths": [ ... ],
    "report": [ ... ]
}

To read this in and add our scenarios is quite simple:

const baseConfig = require('./backstop.json');
const conspiracy = Object.assign(
    {},
    baseConfig,
    {scenarios: allScenarios}
);

From here, you can either write that out to a file — make sure not to overwrite backstop.json, or use a different name for that — or run backstop directly from the script:

const backstop = require("backstopjs");

fs.writeFile("out.json", JSON.stringify(conspiracy, null, 4), () =>
    console.log("wrote config to out.json")
);

backstop("test", { config: conspiracy })
    .then(() => console.log("done!"));

And that’s all there is to it.

In practice, we’ve also added a simple command line interface using commander, so that the user can directly run either test, reference, approve, or even test and reference at the same time (taking advantage of the referenceUrl property of scenarios).

It was also helpful to add filtering of scenarios, since inevitably people will want to run subsets of tests as they’re developing. All it takes is passing in a search argument and checking label.indexOf(searchPattern) after reading in all the scenarios.

In Part 2, I’ll address the second half of this problem: how to write a better report file to make reviewing all these results easier.

But wait: why is that final variable called “conspiracy”?

You may have noticed Backstop’s mascot is a lemur. Since I am developing a tool to essentially herd a group of Backstop config files together, it made sense to name it using the proper group noun: a conspiracy of lemurs. So, internally, we call this tool Backstop Conspiracy.