MSC Assessment Alliance

The Assessment Alliance is a partnership to improve undergraduate assessment practice. This partnership consists of all undergraduate members of the Medical Schools Council. Its aim is to help ensure the confidence of the public, employers, and the regulator in the quality of medical school graduates by:

• Developing the highest quality assessments for undergraduate students, in turn enhancing their learning experiences
• Sharing experience and adding value through collaboration rather than competition
• Exploring the equivalency of the standards applied by medical schools

One of the main achievements of the Assessment Alliance is the development and running of a communal bank of examination questions that are accessible to all partner medical schools. These questions are developed collaboratively by assessment experts from the medical schools, then bespoke software is used to assist in a rigorous quality-assurance process. Medical schools can draw from the bank when arranging their formative and summative assessments. This is efficient both in terms of cost and staff time. It also enables psychometrically valid comparisons between the passing standards of different medical schools. 

The majority of items in the Question Bank are in the single best answer format, but other formats such as OSCE stations and ‘multiple-mini interviews’ are also being developed and shared.

The principle of the question bank originated in the Universities Medical Assessment Partnership (UMAP), which began in 2003 and was supported by HEFCE’s Fund for the Development of Teaching and Learning. This grew into a consortium of 15 medical schools, all members of the Medical Schools Council, working together to build a shared database of single best answer questions of applied basic knowledge in medicine. UMAP was incorporated into the Medical Schools Council a few years later and renamed the Assessment Alliance to reflect the broader remit and an increased membership.

In 2010, the General Medical Council requested evidence that the standard required to pass written examinations at graduation level is comparable across all UK medical schools. The Assessment Alliance commissioned a project to explore the acceptability, feasibility and methodology for a comparison using ‘common content’ single best answer items inserted into medical school finals-level papers. Following a successful pilot, the project was rolled out across medical schools and is still in place today, providing important data and feedback for schools.

The Assessment Alliance also establishes working groups and commissions research to learn about particular assessment methods. For example, currently underway is a qualitative study of standard setting in UK medical schools. Another project is applying Rasch modelling to evaluate passing standards and identify common strengths and weaknesses.

Structure

The Assessment Alliance consists of a Reference Group which meets twice yearly and is attended by two senior assessment experts from each medical school. Elected from the Reference Group is the Board, which meets four times a year to lead the activities of the Assessment Alliance as a whole. Board members are as follows:

• Professor Mark Gurnell (Chair), University of Cambridge
• Professor David Kluth (Deputy Chair), University of Edinburgh
• Dr Andrew Blythe, Bristol University
• Professor Adrian Freeman, University of Exeter
• Dr Ellie Hothersall, University of Dundee
• Dr Elizabeth Metcalf, Cardiff University
• Dr Nimesh Patel, Queen Mary University of London
• Professor Amir Sam, Imperial College London
• Dr Phil Smith, Cardiff University
• Dr Rachel Westacott, University of Birmingham

Ex officio:

• Professor John Atherton, Medical Schools Council
• Dr Katie Petty-Saphon, Medical Schools Council
• Professor Malcolm Reed, Medical Schools Council

Twice a year the Assessment Alliance runs ‘expert item review’ events. These are attended by assessment experts from across the medical schools and are where questions are written and quality-assured by peers, overseen by members of the Final Clinical Review Group. After review and approval, they are added to the common bank of questions from which medical schools can draw. 

To contact the Assessment Alliance team, call the Medical Schools Council or email directly: mscaa@medschools.ac.uk.

Publications 

The following publications have emanated from or been facilitated by the work of the MSC Assessment Alliance:

• High-stakes, remote-access, open-book examinations - Professor Amir Sam et al
• National inter-rater agreement of standardised simulated-patient-based assessments - Professor Amir Sam et al
• Exploring differences in individual and group judgements in standard setting - Dr Peter Yeates et al
• Validity of very short answer versus single best answer questions for undergraduate assessment - Professor Amir Sam et al
• Variation in passing standards for graduation-level knowledge items at UK medical schools - Dr Celia Taylor et al
• Comparing single best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study. - Professor Amir Sam et al
• The influence of candidates’ physical attributes on assessors’ ratings in clinical practice - Professor Amir Sam et al
• Variation in performance on common content items at UK medical schools - Dr David Hope et al
• Thinking differently – Students’ cognitive processes when answering two different formats of written question - Professor Amir Sam et al
• Making it fit: Examining the assessment of contextual knowledge and understanding in the positivist assessment modality of medical education - E.J. Hothersall, V. Rodrigues, M. Gordon, J.C. McLachlan, S. McAleer