Do patterns help novice evaluators? A comparative study
De Angeli A
MetadataShow full item record
Evaluating e-learning systems is a complex activity which requires considerations of several criteria addressing quality in use as well as educational quality. Heuristic evaluation is a widespread method for usability evaluation, yet its output is often prone to subjective variability, primarily due to the generality of many heuristics. This paper presents the pattern-based (PB) inspection, which aims at reducing this drawback by exploiting a set of evaluation patterns to systematically drive inspectors in their evaluation activities. The application of PB inspection to the evaluation of e-learning systems is reported in this paper together with a study that compares this method to heuristic evaluation and user testing. The study involved 73 novice evaluators and 25 end users, who evaluated an e-learning application using one of the three techniques. The comparison metric was defined along six major dimensions, covering concepts of classical test theory and pragmatic aspects of usability evaluation. The study showed that evaluation patterns, capitalizing on the reuse of expert evaluators know-how, provide a systematic framework which reduces reliance on individual skills, increases inter-rater reliability and output standardization, permits the discovery of a larger set of different problems and decreases evaluation cost. Results also indicated that evaluation in general is strongly dependent on the methodological apparatus as well as on judgement bias and individual preferences of evaluators, providing support to the conceptualisation of interactive quality as a subjective judgement, recently brought forward by the UX research agenda.
Showing items related by title, author, creator and subject.
Donnellan D; Pahl C (MCIL, 2001)Virtual courses often separate teacher and student physically from one another, resulting in less direct feedback. The evaluation of virtual courses and other computer-supported educational systems is therefore of major ...
Evaluating mtDNA patterns of genetic isolation using a re-sampling procedure: A case study on Italian populations Anagnostou P; Capocasa M; Dominici V; Montinaro F; Coia V; Destro-Bisol G (2017)BACKGROUND: A number of studies which have investigated isolation patterns in human populations rely on the analysis of intra- and inter-population genetic statistics of mtDNA polymorphisms. However, this approach makes ...
Student teachers’ self-evaluation and perceived social evaluation of their future profession in South Tyrol – Italy = L’autovalutazione e la percepita valutazione sociale della professione dell’insegnante da parte degli studenti delle Scienze della Formazione in Alto Adige – Italia Gross B (2019)Focusing on the student teachers’ evaluation of the prestige of their future profession is needed to include this aspect of the professionalization process in teacher education and training. Hence, knowledge on student ...