MRCGP assessment pass mark too low, academics tell GMC

Around 150 extra candidates qualify as GPs each year due to an erroneously low MRCGP exam pass mark, according to a group of leading psychometricians. 

The group have formally complained to the GMC about the robustness of the Simulated Consultation Assessment (SCA) – which GP trainees must pass as part of the MRCGP exam to gain their certificate of completion of training (CCT). 

In its letter of complaint, it claims the SCA uses ‘flawed statistical analysis to assess the quality of each exam sitting.’ 

The letter’s authors include Dr Richard Wakeford, who was involved in developing the assessment’s two predecessors (CSA and RCA), as well as Dr Julian Hancock who is the chair of the applied knowledge test (AKT) of the GMA’s Medical Licensing Assessment (MLA). 

Dr Wakeford claimed the RCGP has overstated the reliability of the exam because the college incorrectly calculated Cronbach’s alpha and standard error of measurement – the accepted measures of an assessment’s reliability. 

During the SCA, candidates complete 12 simulated consultations with actors role-playing as patients. 

At each of the 12 consultation ‘stations’, an examiner scores the candidate in three areas: data gathering and diagnosis; clinical management and medical complexity; and relating to others. 

In calculating alpha, the RCGP treats each domain score as independent, as though there are 36 independent scores given for a candidate, rather than 12 separate ‘station level’ scores – inaccurately, the group argues. 

This inappropriately high coefficient creates an erroneously low standard error of measurement (SEm), which in turn leads to a lower pass mark (equivalent to 2-3 marks’ difference). 

‘The effect of this error has been to (a) overstate the reliability of each exam sitting (meaning the defensibility of pass/fail decisions is reduced) and (b) lower the pass marks (meaning a significant number of candidates have passed this ‘exit’ exam who would have failed, had the correct calculations been made)’, the letter states. 

‘Bluntly, this means that an estimated 150+ candidates are passing the SCA each year who, had the statistics been correctly calculated, would have failed it’, Dr Wakeford told Pulse. 

Last week (5 November), the group wrote to the GMC’s Corporate Review team because it said the council ‘abrogated its regulatory duty’ by not taking action.  

When Dr Wakeford had first raised the issue with the regulator, it told him their standards did not require a detailed explanation of how reliability is calculated. 

‘This is the responsibility of the RCGP as part of their local processes, which we don’t approve, or quality assure’, it said. 

Responding for this story, a GMC spokesperson told Pulse: ‘We have received the letter from Dr Wakeford and his colleagues, which raises some very serious concerns.  

‘It is essential that awarding bodies have assessments and processes that meet a high standard. We are therefore looking into the matter carefully and will respond as soon as we are able.’ 

RCGP chair Professor Kamila Hawthorne told Pulse that while the exam’s ‘statistical methodology differs from some other postgraduate clinical assessments, it is robust and evidence-based’. 

Professor Hawthorne said: ‘The College is aware of this complaint, and it would be inappropriate to comment further whilst it is being considered. However, it is inaccurate to say the College has not engaged with the complainant and we will continue to do so, as well as remaining in regular contact with the GMC.  

‘The Simulated Consultation Assessment is one part of the tripartite MRCGP exam, which is designed to ensure a GP Registrar’s competence in the skills necessary to practise independently as a GP in the UK in order to ensure patient safety.  

‘The SCA has been approved against the GMC’s standards and while our statistical methodology differs from some other postgraduate clinical assessments, it is robust and evidence-based.’ 

The SCA is a permanent replacement for the Remote Consultation Assessment (RCA), a temporary examination brought in during to the COVID-19 pandemic to replace the Clinical Skills Assessment (CSA). 

Last month, GP leaders raised concerns after FourteenFish Consult, a free tool for trainees to record consultations, was withdrawn by the app’s developers.  

The tool had been developed for the now-replaced RCA, but the BMA argued it was still useful as a cost-effective and GDPR-compliant recording tool for trainees.  

And last year, the college compensated trainees unable to complete the RCA due to an IT outage in 2023.