A Reference Manual for Automating Accessibility Testing in React/Gatsby(iii)

From various testing framework integrated with axe-core, to linting plugins and full page auditing…

Photo by Alex Perez on Unsplash

The last bit of accessibility tooling comes down to full page reporting. (While you can achieve with the same result using E2E testing framework integrated with axe-core, this is more straight forward).

And the two tools I’d like to introduce to you are

• Pa11y (Pa11y CI)
• a11y

comparison of download volume in Pa11y and a11y

So first start with Pa11y, which runs accessibility tests on your pages via the command line or Node.js. It runs against full page and show result in desired format (json, csv, html … — good to share with the team) with helpful information on how to fix the errors to comply with a11y standards.

Under the hood, it uses HTML CodeSniffer in Headless Chrome to evaluate web pages based on the WCAG. (HTML CodeSniffer is a client-side script that run analysis against HTML source code and detects violations of defined criteria).

This is useful, but a big hassle if you need to run from command line every time. So there’s also Pa11y-CI, which is built with Pa11y but can run accessibility tests against multiple URLs, and follows a declarative model by looking for a .pa11yci config file.

In order to use the Pa11y CI, you need to create a JSON file .pa11yci in the root directory, where you define a list of configs, some of them being:

• standard : The accessibility standard to use: Section508, WCAG2A, WCAG2AA (default), WCAG2AAA
• level : The level of issue to fail on (exit with code 2); options include error, warning, and notice.
• timeout : Time in milliseconds that a test should be allowed to run before calling back with a timeout error; defaults to 30,000.
• threshold : The number of errors, warnings, or notices permitted before failing the test with exit code 2.

For a full list, please check documentation.

An example config file:

{
   "standard": "WCAG2AA"
   "defaults": {
    "timeout": 3000,
    "threshold": 2
  },  
    "urls": [ 
       "url": "http://pa11y.org/",
       "viewport": { "width": 344, "height": 480 },
       }
],

You can also have a separate config file for different environment for Pa11y-ci to run on, for example, the url to test in staging is different from the one in development. (More on this here)

Among its many configs, you can also define user actions, which might be sufficient for a few e2e test on trussle.com

pa11y('http://example.com/', {
    actions: [
        'click element #tab-1',
        'wait for element #tab-1-content to be visible',
        'set field #fullname to John Doe',
        'check field #terms-and-conditions',
        'uncheck field #subscribe-to-marketing',
        'screen capture example.png',
        'wait for fragment to be #page-2',
        'wait for path to not be /login',
        'wait for url to be https://example.com/',
        'wait for #my-image to emit load',
        'navigate to https://another-example.com/'
    ]
});

In contrast, another similar tool developed by Addy Yosmani (addyosmani) — a11y. It is used behind Chrome Accessibility Developer Tools, and require less configuration (personal opinion).

But the biggest drawback of this is that it only checks common accessibility problems including WAI-ARIA but not specifically cover WCAG standards likle Pa11y CI does.

Worth keeping an eye on it if you don’t need to enforce strict WCAG standards and want lighter configuration.