Accessibility test digest

Synopsis

Page: Bank of the West (90%)

URL: https://www.bankofthewest.com/

Score: 3989

Tested by: Testaro, procedure tp15

Scored by: Testilo, procedure sp15c

Digested by: Testilo, procedure dp15a

Introduction

The Testaro application used its tp15 testing procedure to test the accessibility (barrier-free design and coding) of the Bank of the West (90%) web page at https://www.bankofthewest.com/ on 2022/08/12. The procedure performed 1230 tests. Of these, 24 are custom tests defined by Testaro, and the others belong to these seven other packages (programs that perform collections of tests):

Alfa by Siteimprove
Axe-core by Deque
Continuum by Level Access
HTML CodeSniffer by Squiz Labs
Equal Access by IBM
Nu Html Checker
Tenon by Level Access
WAVE by WebAIM

Testaro produced a report enumerating the test results.

Testilo processed the report and used the sp15c scoring procedure to compute partial and total scores for the page. The total score is 3989 (where 0 is the best possible score). The scored report is appended below.

Finally, Testilo used procedure dp15a to produce this digest, briefly explaining how sp15c computed the scores.

Score summary

Score components
total	3989
log	2630
preventions	1350
tabFocusability	6
visibleBulk	3

Issue summary

Special issues

log: Score 2630. This is based on the amount of browser error logging and miscellaneous logging during the tests.

preventions: Score 1350. This is based on tests that the page did not allow to be run. That impedes accessibility progress and risks interfering with tools that users with disabilities need.

Classified issues

tabFocusability: Score 6. Issues reported by tests in this category:

Package testaro, test focAll, score 4 (Discrepancy between elements that should be and that are Tab-focusable)

visibleBulk: Score 3. Issues reported by tests in this category:

Package testaro, test bulk, score 1 (Page contains many visible elements)

Discussion

Although there are widely accepted accessibility standards, there is no unanimity about how to define, test, and quantify accessibility. The failures reported in this digest merit investigation as potential opportunities for improved accessibility. Investigation may lead you to conclude that some of the reported failures do not actually harm accessibility. Conversely, some substantial accessibility faults can escape detection by any of these tests. You may question the attempt to assign an accessibility score to a web page, or you may prefer weightings and formulas different from those used by sp15c. You can modify and extend Testaro and Testilo to fit other theories and priorities.

Here, in brief, is how sp15c computes a score for a page.

It finds all the defects and warnings (let’s call them issues) recorded in the report.
It classifies them according to type. For example, a link that looks like the text around it is one issue category, while a video that has no captions is another issue category.
It also classifies the issues according to severity. For example, an issue that prevents a transaction is more severe than an issue that only complicates the transaction, and a warning about a possible issue is less severe than a definite finding of an issue. (Some packages rate the severity of each issue; for the other packages, sp15c assigns a severity weight to the issue type and uses that weight.)
It assigns quality ratings to particular tests that are judged abnormally reliable or unreliable.
It assigns a score to each issue reported by each test of each package.
It aggregates the issue scores, weighting them by severity, test quality, and redundancy. Redundancy occurs, and causes downweighting, when two or more packages contain tests that are designed to discover the same or mostly the same issues. So the score for a category is not simply the sum of the scores of the tests in that category.
It assigns a score for issues in the page logged by the browser.
It assigns an estimated score each time the page prevents one of the packages or one of the Testaro tests from being run on the page.
It adds the scores together to obtain a total score.

The precise rules of sp15b are found in the code itself.

Report

{
  "id": "4yiew-bankofthewest",
  "host": {
    "id": "bankofthewest",
    "which": "https://www.bankofthewest.com/",
    "what": "Bank of the West (90%)"
  },
  "log": [
    {
      "event": "startTime",
      "value": "2022-08-12T20:46:49"
    },
    {
      "event": "endTime",
      "value": "2022-08-12T20:47:43"
    }
  ],
  "script": {
    "id": "tp15",
    "what": "Alfa, Axe, Continuum, HTML CodeSniffer, IBM, Nu Html Checker, Tenon, WAVE, and 22 custom tests",
    "strict": true,
    "timeLimit": 500,
    "commands": [
      {
        "type": "launch",
        "which": "webkit",
        "what": "Webkit browser"
      },
      {
        "type": "url",
        "which": "https://www.bankofthewest.com/",
        "what": "Bank of the West (90%)"
      },
      {
        "type": "tenonRequest",
        "id": "a",
        "withNewContent": true,
        "what": "Tenon API version 2 test request"
      },
      {
        "type": "test",
        "which": "motion",
        "what": "spontaneous change of content; requires webkit",
        "delay": 2500,
        "interval": 2500,
        "count": 5
      },
      {
        "type": "launch",
        "which": "chromium",
        "what": "Chromium browser"
      },
      {
        "type": "url",
        "which": "https://www.bankofthewest.com/",
        "what": "Bank of the West (90%)"
      },
      {
        "type": "test",
        "which": "bulk",
        "what": "count of visible elements"
      },
      {
        "type": "test",
        "which": "embAc",
        "withItems": true,
        "what": "active elements incorrectly embedded in each other"
      },
      {
        "type": "test",
        "which": "focAll",
        "what": "Tab-focusability"
      },
      {
        "type": "test",
        "which": "focInd",
        "revealAll": false,
        "allowedDelay": 250,
        "withItems": true,
        "what": "focus indicators"
      },
      {
        "type": "test",
        "which": "focOp",
        "withItems": true,
        "what": "focusability and operability of elements"
      },
      {
        "type": "test",
        "which": "hover",
        "headSize": 40,
        "headSampleSize": 20,
        "tailSampleSize": 15,
        "withItems": true,
        "what": "hover impacts"
      },
      {
        "type": "test",
        "which": "labClash",
        "withItems": true,
        "what": "unlabeled and mislabeled form controls"
      },
      {
        "type": "test",
        "which": "linkUl",
        "withItems": true,
        "what": "underlining of inline links"
      },
      {
        "type": "test",
        "which": "menuNav",
        "withItems": true,
        "what": "keyboard navigation within true-focus menus"
      },
      {
        "type": "test",
        "which": "radioSet",
        "withItems": true,
        "what": "grouping of radio buttons in fieldsets"
      },
      {
        "type": "test",
        "which": "role",
        "what": "validity and necessity of role assignments"
      },
      {
        "type": "test",
        "which": "styleDiff",
        "withItems": true,
        "what": "style consistency of headings, buttons, and links"
      },
      {
        "type": "test",
        "which": "tabNav",
        "withItems": true,
        "what": "keyboard navigation within tab lists"
      },
      {
        "type": "test",
        "which": "zIndex",
        "withItems": true,
        "what": "elements with non-auto z indexes"
      },
      {
        "type": "test",
        "which": "alfa",
        "what": "Siteimprove alfa"
      },
      {
        "type": "test",
        "which": "axe",
        "detailLevel": 2,
        "rules": [],
        "what": "Axe core, all rules"
      },
      {
        "type": "test",
        "which": "continuum",
        "what": "Continuum"
      },
      {
        "type": "test",
        "which": "htmlcs",
        "what": "HTML CodeSniffer"
      },
      {
        "type": "test",
        "which": "ibm",
        "withItems": true,
        "what": "IBM Accessibility Checker, with page content and again with URL"
      },
      {
        "type": "test",
        "which": "nuVal",
        "what": "Nu Html Checker"
      },
      {
        "type": "test",
        "which": "wave",
        "reportType": 4,
        "what": "WAVE, report-type 4"
      },
      {
        "type": "test",
        "which": "tenon",
        "id": "a",
        "what": "Tenon API version 2 result retrieval"
      }
    ]
  },
  "acts": [
    {
      "type": "launch",
      "which": "webkit",
      "what": "Webkit browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "Bank of the West (90%)",
      "result": "https://www.bankofthewest.com/"
    },
    {
      "type": "tenonRequest",
      "id": "a",
      "withNewContent": true,
      "what": "Tenon API version 2 test request",
      "url": "https://www.bankofthewest.com/"
    },
    {
      "type": "test",
      "which": "motion",
      "what": "motion",
      "delay": 2500,
      "interval": 2500,
      "count": 5,
      "url": "https://www.bankofthewest.com/",
      "result": {
        "bytes": [
          453854,
          453854,
          453854,
          453854,
          453854
        ],
        "localRatios": [
          1,
          1,
          1,
          1
        ],
        "meanLocalRatio": 1,
        "maxLocalRatio": 1,
        "globalRatio": 1,
        "pixelChanges": [
          0,
          0,
          0,
          0
        ],
        "meanPixelChange": 0,
        "maxPixelChange": 0,
        "changeFrequency": 0
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "Chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "Bank of the West (90%)",
      "result": "https://www.bankofthewest.com/"
    },
    {
      "type": "test",
      "which": "bulk",
      "what": "count of visible elements",
      "url": "https://www.bankofthewest.com/",
      "result": {
        "visibleElements": 781
      }
    },
    {
      "type": "test",
      "which": "embAc",
      "withItems": true,
      "what": "active elements embedded in links or buttons",
      "url": "https://www.bankofthewest.com/",
      "result": {
        "totals": {
          "links": 0,
          "buttons": 0,
          "inputs": 0,
          "selects": 0
        },
        "items": []
      }
    },
    {
      "type": "test",
      "which": "focAll",
      "what": "focusable and Tab-focused elements",
      "url": "https://www.bankofthewest.com/",
      "result": {
        "tabFocusables": 131,
        "tabFocused": 127,
        "discrepancy": -4
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "focInd",
      "revealAll": false,
      "allowedDelay": 250,
      "withItems": true,
      "what": "focus indicators",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "focOp",
      "withItems": true,
      "what": "focusability and operability of elements",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "hover",
      "headSize": 40,
      "headSampleSize": 20,
      "tailSampleSize": 15,
      "withItems": true,
      "what": "hover impacts",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "labClash",
      "withItems": true,
      "what": "unlabeled and mislabeled form controls",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "linkUl",
      "withItems": true,
      "what": "underlining of inline links",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "menuNav",
      "withItems": true,
      "what": "keyboard navigation within true-focus menus",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "radioSet",
      "withItems": true,
      "what": "grouping of radio buttons in fieldsets",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "role",
      "what": "validity and necessity of role assignments",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "styleDiff",
      "withItems": true,
      "what": "style consistency of headings, buttons, and links",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "tabNav",
      "withItems": true,
      "what": "keyboard navigation within tab lists",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "zIndex",
      "withItems": true,
      "what": "elements with non-auto z indexes",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "alfa",
      "what": "Siteimprove alfa",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "axe",
      "detailLevel": 2,
      "rules": [],
      "what": "Axe core, all rules",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "continuum",
      "what": "Continuum",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "htmlcs",
      "what": "HTML CodeSniffer",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "ibm",
      "withItems": true,
      "what": "IBM Accessibility Checker, with page content and again with URL",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "nuVal",
      "what": "Nu Html Checker",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "test",
      "which": "wave",
      "reportType": 4,
      "what": "WAVE, report-type 4",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    },
    {
      "type": "launch",
      "which": "chromium",
      "what": "chromium browser"
    },
    {
      "type": "url",
      "which": "https://www.bankofthewest.com/",
      "what": "URL",
      "result": "ERROR: Attempts to visit https://www.bankofthewest.com/ failed"
    },
    {
      "type": "test",
      "which": "tenon",
      "id": "a",
      "what": "Tenon API version 2 result retrieval",
      "result": {
        "error": "ERROR: No page identified",
        "prevented": true
      }
    }
  ],
  "timeStamp": "4yiew",
  "startTime": "2022-08-12T20:46:49",
  "presses": 0,
  "amountRead": 0,
  "testTimes": [
    [
      "motion",
      13
    ],
    [
      "focAll",
      4
    ],
    [
      "bulk",
      0
    ],
    [
      "embAc",
      0
    ]
  ],
  "logCount": 127,
  "logSize": 9102,
  "errorLogCount": 122,
  "errorLogSize": 8564,
  "prohibitedCount": 111,
  "visitTimeoutCount": 0,
  "visitRejectionCount": 50,
  "visitLatency": 30,
  "endTime": "2022-08-12T20:47:43",
  "elapsedSeconds": 54,
  "score": {
    "scoreProcID": "sp15c",
    "logWeights": {
      "logCount": 0.5,
      "logSize": 0.01,
      "errorLogCount": 1,
      "errorLogSize": 0.02,
      "prohibitedCount": 15,
      "visitTimeoutCount": 10,
      "visitRejectionCount": 10,
      "visitLatency": 1
    },
    "soloWeight": 2,
    "groupWeights": {
      "absolute": 2,
      "largest": 1,
      "smaller": 0.4
    },
    "preventionWeights": {
      "testaro": 50,
      "other": 100
    },
    "packageDetails": {
      "testaro": {
        "bulk": 2,
        "focAll": 8
      }
    },
    "groupDetails": {
      "groups": {
        "visibleBulk": {
          "testaro": {
            "bulk": {
              "score": 1,
              "what": "Page contains many visible elements"
            }
          }
        },
        "tabFocusability": {
          "testaro": {
            "focAll": {
              "score": 4,
              "what": "Discrepancy between elements that should be and that are Tab-focusable"
            }
          }
        }
      },
      "solos": {}
    },
    "preventionScores": {
      "testaro-focInd": 50,
      "testaro-focOp": 50,
      "testaro-hover": 50,
      "testaro-labClash": 50,
      "testaro-linkUl": 50,
      "testaro-menuNav": 50,
      "testaro-radioSet": 50,
      "testaro-role": 50,
      "testaro-styleDiff": 50,
      "testaro-tabNav": 50,
      "testaro-zIndex": 50,
      "alfa": 100,
      "axe": 100,
      "continuum": 100,
      "htmlcs": 100,
      "ibm": 100,
      "nuVal": 100,
      "wave": 100,
      "tenon": 100
    },
    "summary": {
      "total": 3989,
      "log": 2630,
      "preventions": 1350,
      "solos": 0,
      "groups": [
        {
          "groupName": "tabFocusability",
          "score": 6
        },
        {
          "groupName": "visibleBulk",
          "score": 3
        }
      ]
    }
  },
  "scoreProcID": "sp15c"
}