Evaluation procedures and conformance
This section describes the procedures and their objectives for expert and automatic evaluation and provides information about conformance claims based on the UWEM. The evaluation procedures in this document have the following objectives:
-
To improve replicability of the evaluation results by definition of a repeatable sampling scheme in: Scope and Sampling of resources.
-
To clarify interpretation, facilitate automatic evaluation and improve replicability of its test results (see Tests for conformance evaluation).
-
To support aggregation of the evaluation results for individual Web pages, Web sites and even sets of Web sites (Aggregation model for test results). The UWEM will also support aggregations across geographical regions as well as economical sectors, as indicated in section Reporting of test results.
-
To support the reporting of test results (Aggregation model for test results and Reporting of test results)
-
To support large scale evaluation of Web sites, to identify problem areas in Web sites.
-
To evaluate conformance to the checks of: Tests for conformance evaluation. It includes procedures for evaluation down to very specific tests for expert and automatic evaluation.
Types of evaluation
Accessibility testing may be done via automatic, expert and user testing. The different types of evaluation methods have a number of strengths and weaknesses.
describes three different evaluation methods, from which two (automatic and expert) are covered in the UWEM. The figure shows, for example, that automatic evaluation (Tool 1 or Tool2) can only test for conformance to a subset of the checkpoints (such as the set of tests marked as "fully automatable" in: Tests for conformance evaluation), which further means that only a subset of all possible accessibility barriers can be identified reliably by using automatic testing. Therefore, coverage of automatic evaluation as an overall indicator of accessibility is low, however it can identify many barriers reliably. It may also be applied efficiently to test very large numbers of Web resources within one Web site as well as multiple Web sites. Tool 1 and Tool 2 are here two fully automatic assessment tools that focus on checking accessibility issues, possibly with some overlap of functionality.
Some tools can also act as support systems in an expert evaluation process. The tools provide reliable results for a subset of tests and can not only speed up the process by performing some tasks automatically, but also, by providing hints about barrier locations, indicate areas the expert evaluators should focus on.
User testing is able to identify barriers that are not caught by other testing means, and is also capable of estimating the accessibility for tested scenarios. However, user testing is quite specialised, thus it is not generally suitable for conformance testing, since it is not able to test all aspects of the tests of: Tests for conformance evaluation. The best approach to ensure both accessibility and UWEM conformance is to use a combined approach encompassing all evaluation methods: automatic, expert evaluation and user testing of the Web site.
Figure 1: UWEM Evaluation Types.
User testing is not covered in this version of UWEM. How to involve users in the evaluation of web content is described also on the W3C/WAI website in the evaluation suite. 1
The main advantages of automatic testing are:
-
The method is replicable (although the dynamic nature of some Web technologies mean complete replication is impossible [LEVENE99]).
-
It may be applied to very large number of resources on a Web site and to multiple Web sites.
Conformance claims
A conformance claim determines if a web site meets the accessibility standards described in: Tests for conformance evaluation. To claim conformance with the UWEM, it is minimally required that:
-
The resources sample and the scope for the evaluation is defined according to: Scope and Sampling of resources.
-
All resources in the sample pass all applicable tests to the corresponding conformance level.
The conformance levels to the UWEM replicate those of WCAG 1.0, i.e.:
-
Conformance Level 1: all tests relevant to WCAG 1.0 Priority 1 checkpoints are satisfied (see Tests for conformance evaluation).
-
Conformance Level 2: all tests relevant to WCAG 1.0 Priority 1 and Priority 2 checkpoints are satisfied (see Tests for conformance evaluation).
More information on the meaning of priorities can be found in: WCAG 1.0 priorities.
The claims of accessibility conformance according to the UWEM methodology must use the following form:
-
The UWEM version and its URI identifier, i.e., http://www.wabcluster.org/UWEM1/
-
The URI to a document detailing the scope and the sample (see: Scope and Sampling of resources) to which the claim refers.
-
The level of conformance.
Tool conformance
Evaluation tools can also claim conformance to UWEM 1.0. In that way, experts evaluating web sites according to UWEM 1.0 will be able to rely on the results of the tool for the fully automatable tests of the methodology.
To claim conformance to UWEM 1.0, the tool MUST implement all fully automatable tests of: Tests for conformance evaluation, to the corresponding conformance level. This conformance claim must use the following form:
-
The UWEM version and its URI identifier, i.e., http://www.wabcluster.org/UWEM1/
-
The URI to a document detailing a set of public test files where this conformance claim has been verified.
-
The level of conformance.