This site is intended for health professionals only

At the heart of general practice since 1960

MRCGP examiners 'do not favour their own', claims study

A statistical analysis has shown no evidence systematic bias by examiners in the clinical skills assessment is the reason for the difference in failure rates for international or black or ethnic minority candidates, in direct contradiction to the conclusions of a recent GMC-commissioned review into the MRCGP exam.

The study, published in the British Journal of General Practice, found significant differences between the grades awarded to international medical graduates (IMGs), UK graduates, black and ethnic minority (BME) candidates and male candidates depending on the characteristics of the examiner.

But these differences were not seen after they were put through an analysis of variance (ANOVA), apart from the impact of gender on the results gained by candidates, with male examiners marking more harshly.

The researchers concluded that there was no support for the suggestion that the diversity of examiners must be improved, instead recommending that each examiner is subjected to an ‘equality audit’ to ensure they are not favouring one group over another.

The findings are in direct opposition to the conclusions of GMC-commissioned research conducted by racism expert Professor Aneez Esmail who said that that increasing the diversity of the examiners to be ‘more reflective of the nature of GPs in the country’ would improve the way the CSA is run.

A different version of his research, published in the BMJ, concluded that subjective bias ‘due to racial discrimination’ may be a cause of the higher failure rates for UK trained candidates from non-white ethnic groups and IMGs.

This new study – led by the clinical research lead at the RCGP, Dr Mei Ling Denney – included data from almost all of the candidates taking the CSA in 2011/12, and found no consistent evidence of examiners ‘favouring their own’.

In fact, the raw data from 52,000 CSA attempts and 251 examiners showed that examiners who were IMGs gave lower marks to both UK graduates and fellow IMGs and there was no significant association between grades from white examiners marking white candidates.

They showed that a ‘slightly lower mark’ was awarded to candidates of the opposite ethnicity than to those of the same; but the impact of this was minimal, accounting for 0.06% of total variance in case score.

The authors concluded: ‘The effect size of the potential significant raw effects found in this study, regarding any individual candidate case, could result in, for example, male candidates receiving a 0.9% enhancement of their case score under a female examiner and any candidate receiving, irrespective of their source of degree, a 2.4% enhancement of their case score under a UK-graduate examiner as opposed to an IMG examiner.’

They added: ‘Examiners show no general tendency to “favour their own kind”. With confounding between variables, as far as the impact on candidates’ case scores, substantial effects relate to candidate and not examiner characteristics. Candidate-examiner interaction effects were inconsistent in their direction and slight in their calculated impact.’

‘This study provides no support for equating examiner representation to that of candidates from the point of view of delivering a fair assessment to all groups of candidates. Nevertheless incorporating a variety of subgroups of examiners in the examiner panel has benefits for collegiality and examination development, and incorporating approaches to practice which may themselves vary between these subgroups.’

 

Readers' comments (38)

  • Will the other party (In this case RCGP) accept tehy are at fault?
    They are not happy to accept result of independant review but will accept from one of their own researcher how equitable is that?

    Unsuitable or offensive? Report this comment

  • RCGP study finds that their exam is not biased. In other news it transpires that the pope is catholic.

    If anyone seriously believes that you can explain away a 15(!!!) times difference in pass rates you need your head examining. The authors claim that there was no evidence of examiners favouring their own kind. There are two problems with this. It is not explained what proportion of IMGs are examiners and whether this sample is of a sufficient size for it be robust enough to conduct such analysis. A results of a sample size that is too small is not significant. Even if we ignore it is possible for IMG to discriminate – a basic fact the authors seem to have ignored.

    Even if you accept the findings of the study, there remains an issue that after five years of studying and working in the UK, passing all exams with no issues, having no issues raised by trainers or indeed anyone else, IMGs have a 15times failure rate versus white UK graduates. This tells us two very clear things. Either there are significant failures in the training of IMG’s in the UK (surely an outcome that cannot be beneficial to anyone including the RCGP and UK PLC) or the exam is designed to engineer a high failure rate in IMGs (for whatever reason).

    Unsuitable or offensive? Report this comment

  • This comment has been edited by the moderator.

  • What are white IMGs significantly more likely to pass than BME IMGs despite not coming from anglo-saxon type countries? Answer me that one then?

    And if you manage that then answer me why are BME UK graduates four times more likely to fail than their white counterparts if there is not discrimination going on?

    Unsuitable or offensive? Report this comment

  • 11.56am, not sure if the info you wanted is on Table 1, which shows the number of examiners from BME and the number of examiners who were IMGs.

    My reason for posting is actually another question to anyone who has looked at the methods for this and Prof Esmail's paper. I'm assuming that the datasets they used overlap significantly, but this set of researchers apparently only failed to assign ethnicity to candidates in 29 out of 4029 attempts, whereas Prof Esmail's group had data on 5094 candidates of which 489 did not provide sufficient data for categorisation and another 61 had data missing, so 550 were excluded from analysis. Maybe I'm missing something obvious in the methodology here, or were the datasets available to the two groups different? (or maybe the exclusions were done more rigourously by one group of researchers?)
    Agree completely with 11:56 that there is no benefit to RCGP or the UK (or anyone else or any other country of the world for that matter) if there are significant failings in the training programme for IMGs.

    Unsuitable or offensive? Report this comment

  • RCGP doing their own research led by Deputy chief Examiner !!! What a joke.
    Does RCGP think IMGs are blind!!?really didn't think they can stoop so low. Total Desperation...

    Unsuitable or offensive? Report this comment

  • Ruth- the datasets are different. The MRC data i think looked at first time takers and this study looks at all candidates. Also this study looks at csa cases , not exam result.

    Unsuitable or offensive? Report this comment

  • Just been a patient in the MRCP PACES exam. All of the nine candidates that came through the station I was in, bar one, were very inept, getting very obvious things wrong. The examiners were most surprised. All the candidates bar one were from abroad. It was all most disconcerting.

    Unsuitable or offensive? Report this comment

  • This is the height of desperation from RCGP.

    Unsuitable or offensive? Report this comment

  • why?

    Unsuitable or offensive? Report this comment

  • My BJGP goes straight into the bin. I passed CSA first attempt by studying BMJ study material and case videos. No body has time to read this s**t!!
    A useless journal of no professional standing internationally!

    Unsuitable or offensive? Report this comment

View results 10 results per page20 results per page50 results per page

Have your say