4 Evaluation

Evaluation had two objectives: to measure the progress in attaining correct output documents and to evaluate the impact of using the final method in document production. The iterative evaluation and adjustment of the transformation methods, first at a component level and then at an integrated document level, using an ’expert user’ to carry out all evaluations, is akin to software testing methodologies in which access to users is limited. The evaluation goal was to locate and record defects in the output (or, if no output was produced to determine the cause). Output was checked visually and with relevant assistive software, for correctness by comparison and for usability via a heuristic inspection.

The baseline for document production was established from staff input. None of the sample documents could be used to automatically produce multiple formats and most could not be transformed successfully without alteration. Of the 45 survey respondents 15 had had resources produced in an output format specifically for a disabled student and a further 9 were unsure whether this had occurred. Only 1 member of staff reported that they had been able to produce the required format without assistance.

To evaluate the impact of using the methods we asked an expert user to encode a series of handwritten sample documents. We evaluated the outputs produced and the expert user answered a series of questions to ascertain their views on the methods, the additional costs involved and the challenges experienced.

The expert user took one hour to comprehend the main restrictions of the process but frequently needed to search the guidance documents for details. On the first attempt details were missed which caused all diagrams to be lost. Her feedback also exposed an incomplete understanding of the restrictions. Her difficulties highlighted the exacting technical nature of the task even for an expert user. The searches for technical information imposed an extra cost to producing the documents. She felt she would eventually learn the restrictions and, asked to disregard the overhead, stated “I believe this is as long as it would have taken me working in the usual way [to create one specific format]”. On reaching full understanding of the methods, she concluded they were flexible enough and saw they could be expanded in the future. She noted that “establishing a common standard for producing notes is the best way to not only produce accessible notes but to ensure easy transition between lecturers for a course”. The main costs beyond production overheads were software set up on two operating systems and performing the transformations for each document. These overheads are not insubstantial and again required instructions to be followed precisely.

The experience of the expert user and the integrated document evaluations confirmed that while production of correct documents is possible the methods are technically exacting (even the smallest deviations can lead to no or incorrect output). The required precision is due to missing or incorrect software functionality. Staff specialising in accessible mathematical document preparation have experience in following precise procedures in this work. We concluded that they will be best placed to produce the master documents in the first instance, to transform the documents as required for individual students and to advise teaching staff updating masters. The final stage of the evaluation would have involved user tests by teaching staff. The conclusions already drawn suggested that this was unrealistic. It was inappropriate to ask staff to spend a substantial amount of time attempting to test the methods so the plan was abandoned.

We have derived and documented structure, command, style and symbol constraints which permit multiple formats to be produced. Staff specialising in mathematical document production for disabled students will be able to produce, from a single master (figure 4), formats including large print PDF (figure 5), also suitable for small screens such as some e-book readers, browser independent formats Figure 3: text-to-speech enabled format suitable for small screens and large print produced from L^aT_eX  master on screen, formats accessible to text-to- speech technologies (figure 6), editable formats (figure 7) and formats suitable for screenreader use (speech and/or some form of Braille).

The project has positively impacted practice by:

• increasing the knowledge and expertise of staff who produce and advise on the production of accessible learning materials in mathematical sciences and

• providing a method to reduce resource production and maintenance costs, helping to ensure sustainable access for disabled students to formats suited to their needs.