Scorecard report

The two evaluation types outlined earlier support the identification of possible barriers. A single value describes the probability of creating a barrier by violating any of the tests outlined in section 5. Aggregation of single values allows to compute the barrier probability for a single Web site (Aggregation model for test results). Scoring is based on this data and allows to compare the results obtained via the same evaluation type. In the following, we describe in more detail the scorecards for monitoring the status and progress of single Web sites and groups of Web sites.

Scorecard report

The balanced scorecard is an approach to strategic management developed by Kaplan and Norton [KAPLAN96]. The balanced scorecard method is amongst others used by the European Commission in the Bologna process to measure the progress made in three areas in higher education. This is done to visualise the status and improvements in the process.

Scorecards give the big picture of status and progress, and allow to monitor a set of indicators. In UWEM, we use a simple scorecard to show status at a given point in time and progress since the previous evaluation, if available. The overview perspective is useful to direct additional evaluation resources to areas where the results are especially interesting or alarming.

Scores indicating barrier status

explains the colour and letter codes used for the UWEM barrier status score. The barrier scores are coded with letter codes and coloured fields with A and light blue indicating the best score, and E and dark blue indicating the worst (no data is indicated with “n/a” and white field). The coloured codes were based on the shades of blue as this is the only set of colours that ensures a clear distinction regardless of colour-vision impairment and on the black-and-white printout.

Table 1: UWEM barrier score represented by letter and colour to indicate barrier probability. Note that the levels chosen are preliminary values that will be evaluated when real data measurements from the EIAO project are available.
Scorecards	RGB	Barrier probalility Fs	Level of accessibility	Interpretation
A	204/236/255	Fs = 0.00%	Accessibility is fully supported	Full accessibility
B	153/204/255	0% < Fs <= 25%	Within levels of confidence no or very few accessibility barriers have been identified	Very good accessibility
C	0/153/204	25% < Fs <= 50%	A few accessibility barriers have been identified	Medium accessibility
D	51/102/153	0% < Fs <= 75%	Several accessibility barriers have been identified	Poor accessibility
E	0/51/102	75% < Fs <= 100%	Web Site is inaccessible	Very poor accessibility
n/a	255/255/255	Not measured	Not tested or not applicable	n/a

The scorecards presented in can be used to describe the assessment of single Web sites or groups of Web sites based on UWEM. They may also be used to describe the results of both automatic and expert testing. However, it is important to remember that only the results for different web sites or groups of web sites may be compared only if they are obtained using the same testing method.

The score should always be presented together with information about what tests have been used. The set of tests that can be automatically evaluated is a subset of all defined tests (Tests for conformance evaluation). The other tests can be conducted mainly manually by experts.

Scores indicating barrier change

If a set of web resources was previously evaluated using the same assessment method, the score for indicating the barrier probability change can be determined. This score simply indicates direction of change compared to the previous evaluation, as shown in . An increase in accessibility is indicated with an arrow pointing up, a decrease with an arrow pointing down. No change is indicated with a horizontal bar, and no previous data with the text “n/a”.

Table 2: UWEM barrier change score represented by a symbol indicating change in barrier probability.
Code	Barrier probability (Fs) change since previous evaluation	Accessibility change
↑	Status score decreased	Accessibility improved
—	Status score unchanged	Accessibility unchanged
↓	Status score increased	Accessibility declined
n/a	No previous evaluation available	No previous evaluation available

To provide consistency between status and change scores, the change score relates to a change in the actual status scorecard (see ) rather than to changes in barrier probability Fs (a change in the scorecard colour is needed to trigger a decrease/increase in the change score).

Aggregation of statistics

The single web sites scores can be grouped according to:

NACE business sectors and
NUTS regions.

The scores are then computed as averages over the barrier probabilities of the web sites in the NACE or NUTS group.