Leaked draft of independent review of MRCGP exam rejects bias claims but calls for changes
A leaked draft copy of the GMC-commissioned review into the MRCGP exam has called for changes to the recruitment of examiners and more support for overseas graduates – but has concluded ‘significant differences’ in failure rates between different ethnic groups in the clinical skills examination are unlikely to be the result of bias.
The draft report found that even when controlled for age, gender and AKT component score, white UK graduates performed ‘significantly’ better than all other groups. While the greatest difference was with black and ethnic minority international medical graduates, even non-white UK graduates were nearly four times more likely to fail the CSA examination at their first attempt than their white UK graduate colleagues.
The report called for the recruitment of a more diverse selection of examiners and CSA actors, and for cases presented in the CSA to be more representative of the ‘norms of general practice in a multicultural society’.
But it concluded that the RCGP had taken steps to mitigate any potential bias in the CSA, and stressed that the method of assessment ‘was not a reason for the differential outcomes’ seen in failure rates.
The draft of the long-awaited review by University of Manchester academics Professor Aneez Esmail and Professor Christopher Roberts was posted on the University of Manchester’s website with the file name ‘final report’, although the GMC said the report was a draft version and it did not wish to comment on its findings until a final version was published.
It comes as the RCGP readies itself for a judicial review sought by the British Association of Physicians of Indian Origin to be held on 18 October, which will decide on whether the exams are directly or indirectly discriminatory.
The draft independent review found all other ethnic groups did worse than white UK candidates, even when the results were adjusted for other factors.
It said: ‘The most important finding is that even when controlling for age, gender and AKT component score, all groups do significantly worse than white UK graduates in CSA failure rates.’
But it rejected claims that the differences were the result of bias, stressing that the CSA was not a ‘culturally neutral examination’ but designed to ensure doctors are safe to practise in UK general practice.
On the CSA exam, the review concluded: ‘The nature of the examination is such that it is open to subjective bias. We cannot ascertain if the standardised patients (played by actors) behave differently in front of candidates from non-white ethnic groups. Nor can we confidently exclude bias from the examiners in the way that they assess non-white candidates.
‘However, having observed (by AE) the examination and read the background documentation, it is clear to us that the RCGP is aware of these potential biases and takes steps to mitigate them.’
It added: ‘The differences are much greater between UK and non-UK graduates suggesting that it is the preparedness of UK graduates that may be an explanation for the differences between these two groups.’
The report pointed out that British graduates ‘have much more exposure and training in general practice than the majority of international medical graduates’, and highlighted major differences in pass rates on the machine-marked applied knowledge test.
‘It is difficult to attribute this to bias because of the nature of the test and the reasons for the differential pass rates are likely to be complex,’ it concluded.
‘The AKT is an applied knowledge test relevant for UK general practice. The vast majority of IMG candidates come from the Indian subcontinent and from other countries where the discipline of general practice is poorly developed. IMG candidates will therefore have much less direct experience of general practice than their UK counterparts.’
‘In our view, this must disadvantage this group in subtle ways and explain the much larger differences in outcomes between UK and non-UK graduates. This will also be one of the reasons that there are significant differences in outcome in the CSA examination.’
However the report did also make a series of recommendations for changes to the MRCGP in light of the difference in pass rates. It said:
- the criteria for the selection and recruitment of MRCGP examiners ‘should be reassessed’ to make the pool of examiners more diverse
- there were ‘very few actors representative of ethnic minority backgrounds’ in the CSA exam despite RCGP efforts to recruit examiners from a more diverse range of backgrounds
- the type of cases presented in the CSA should be revised to me more representative of the ‘norms of general practice in a multicultural society’
- more detailed feedback should be provided to candidates. The review said the RCGP’s reason for not giving individualised feedback – that the number of candidates precludes this – was ‘not acceptable especially in an examination that charges candidates £1,694 and which is a high stakes exam such as the MRCGP’
- the GMC should consider providing ‘additional support’ to international medical graduates to better prepare them for taking the exam.
Dr Krishna Kasaraneni, chair of the BMA’s GP trainee subcommittee said the draft document echoe the BMA’s own concerns.
But he added: ‘There are wider questions about the selection of GP trainees and how those who require additional support or training are identified earlier.
‘We would welcome a further study demonstrating what happens to trainees who are forced to withdraw from training, why they are asked to withdraw, and whether ARCP processes are as fair as we would hope they are.’
A spokesperson from the RCGP said they took quality and diversity issues very seriously.
She said: ‘We have always strongly refuted any allegations that the MRCGP exam is discriminatory. The early draft report of the independent review appears to support this. Our assessment procedures are designed to ensure safe patient practice and we are always looking to enhance and improve our processes.
‘We recognise many of the issues raised in this draft and have been working to address these for some time. Once we receive the final report, we will ensure that if any new issues emerge, we develop a response to these as well.’
Please note Pulse is not currently accepting comments on this story.