This site is intended for health professionals only

Babylon claims its in-app clinical advice is ‘on par’ with GPs

Private GP provider Babylon has claimed its app is able to provide clinical advice to patients that is ‘on par’ with doctors, sparking criticism from GP leaders.

Babylon, which also provides its GP at Hand service for NHS patients, said it had used diagnostic questions from trainee GP exams to test the artificial intelligence behind its app, which features a symptom checker and provides patients with medical information and triage advice.

The company said its AI scored 81% when it was tested using RCGP exam questions, whereas the average mark for real-life doctors was 72%, according to data from 2013 to 2018.

However, the RCGP said the claims were ‘dubious’. It highlighted Babylon had used its MRCGP exam preparation questions, which were for ‘revision purposes,’ and that these ‘are not necessarily representative of the full-range of questions and standard used in the actual MRCGP exam’.

The RCGP’s vice chair, Professor Martin Marshall, said: ‘To say that Babylon’s algorithm has performed better than the average MRCGP candidate is dubious.’

Babylon also carried out further tests by collaborating with the Royal College of Physicians, Stanford University’s chief of general primary care, Dr Megan Mahoney, and Yale New Haven Health’s chief population health officer, Dr Arnold DoRosario, which involved using 100 independently devised symptom sets.

According to Babylon, during this further testing, its AI scored 80% for accuracy, while the seven doctors it compared results with achieved an accuracy range of 64 – 94%.

Babylon also said the app’s safety score was found to be 97%, while the GPs were assessed as having an average of 93.1%.

Babylon launched its research on the app’s ability to provide accurate health information at the Royal College of Physicians in London tonight.

Ali Parsa, Babylon’s founder and CEO, said: ‘Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on par with top-rated practising clinicians.’

‘Tonight’s results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world.’

However the RCGP’s Professor Marshall said that while the ‘potential of technology to support doctors to deliver the best possible patient care is fantastic… at the end of the day, computers are computers, and GPs are highly-trained medical professionals’.

‘The two can’t be compared and the former may support, but will never replace, the latter,’ he said.

‘No app or algorithm will be able to do what a GP does…

‘An app might be able to pass an automated clinical knowledge test but the answer to a clinical scenario isn’t always cut and dried, there are many factors to take into account, a great deal of risk to manage, and the emotional impact a diagnosis might have on a patient to consider,’ he added.