What you need to know about the governance of medical transcription services

NHS England’s recent guidance on ambient voice technologies (AVT), which cover transcription tools used by GPs in consultations, caused major confusion for GP practices. Dr Dominic Pimenta, CEO and co-founder of British AI company TORTUS, explains the issues and practices’ requirements.
This is part of the Pulse Partners series. This article has been paid for by X-On Health, with editorial input by Pulse. The opinions in this article do not necessarily reflect the views of Pulse.
Ambient voice technology – AVT – is already revolutionising healthcare. Around 30% of organisations are already employing these medical transcription services in their operations. We are now moving beyond the initial ‘it’s magic’ phase, where clinicians are introduced to what the technology can do, and entering the ‘how do we adopt’ phase, particularly around the governance issues of data protection, cybersecurity, and clinical safety.
This issue hit the headlines last month following new guidance from NHS England, leaving many practices confused about their obligations. So we will look at the particular considerations when it comes to AI and governance.
Data handling
This refers to how data is stored, processed, retained and where is that happening geographically (literally where the physical servers are located).
GPs, as data controllers, are required to act within GDPR legislation. Under the GDPR, all data should be processed within the UK if possible and definitely within the EU. It also requires any data, especially patient data, is handled as minimally as possible to reduce the risk of breach or exposure, which means a clear justification for every data decision is needed. Why does the system store patient data for X days instead of Y days, for example?
In terms of AVT specifically, we need to identify the type of data the system handles. For ambient voice technology, there is the conversation audio data, any additional text data (eg, patient demographics, perhaps problems) and the user data. The first two are considered patient information and highly sensitive, the latter is not.
There are also two types of models of AI; inference, when the tool produces an output from an input (such as a summarised note from a transcript); and ‘machine learning’ or ‘training’, which is when data is used to make the model better.
It is the second point – training – where there may be an issue for suppliers, as technically the patient data used to train models can be stored inside the model, and studies have shown can actually be extracted again in some circumstances. This is a very new area, but may be perceived in future as ‘data retention’. It also changes the nature of who controls the data.
It is likely in the future new legislation will be needed around this to clarify it, but if an AI company is training models on patient data, then that should be very explicitly agreed to by the data controller (you), and ideally the patient as well. Therefore, it might be worth asking suppliers if they train models on patient data, and take advice from your Data Protection Officer (DPO), outsourced companies as a service, or the Information Commissioner’s Office (ICO) if you have worries.
Cybersecurity
So once the data is handled well, the next question is how secure the supplier’s system is from potential external attack. This can be accidental or on purpose. All software should have a penetration test (from an accredited supplier – CREST), which is essentially paying hackers to try and break in and expose any vulnerabilities. Other issues include: the handling of users and passwords; whether the company has independent certifications (such as CyberEssentials Plus); whether the suppliers have filled out a Data Security and Protection Toolkit; and whether the cloud services they use are secure from attack.
Sometimes the AI itself can be a weak point in a security set up – for example if the user can interact directly with it via a chatbot, then depending on what the model does or holds (eg, patient data) or system data (eg, the prompt or instruction itself), it can be attacked by unscrupulous individuals. This is known as ‘prompt injection’.
Again, your supplier should be able to supply all of this data needed as well – and certifications for CyberEssentials Plus and ISO, which are available directly from the auditing companies.
Clinical safety
For any clinical software, the NHS has a system called DTAC (Digital Technology Assessment Criteria) that covers a lot of the data and CSQ requirements, but also ensures that clinical risk is appropriately looked at. In my view, this is one of the hardest areas to understand.
Most IT systems should follow a system of recording risk and safety called DCB0129/160. Essentially, the supplier looks at their safety risks and how they’ve mitigated them, such as potential hazards, what impact they might have for the patient and how the system has tested and mitigated these risks. This is called DCB0129. As part of this process, they must involve clinical safety officers – clinically qualified individuals who have undertaken further training specifically in digital clinical safety. The buyer also needs to repeat a similar process, which is called DCB0160 in their case.
When there is a higher level of clinical safety requirements – for example, software that makes a diagnosis or automates a part of clinician-facing work – the software might be considered a medical device. This means a higher level of regulation under Medicines Health Regulatory Authority (MHRA), which includes more stringent testing and continuous safety monitoring..
NHS England guidance suggests that as the AVT is processing the data in some way (ie, to summarise the conversation) it can be defined as ‘high functionality’ and therefore is at least a class I medical device, and if it goes further being able to make diagnoses, for example, it should be at least a IIa.
From an AI perspective, there are a few specific considerations that are a bit different:
Models change: They change their output even given the same input, and they also change over time (for lots of reasons including training, updating the models, changing the hardware). It means that a safety assessment carried out on initiating use of a product may not be valid anymore over time. So monitoring and continuous assessment are a must for any system using AI, and definitely any system using large language models (AKA generative AI).
Hallucinations: Generative AI specifically is prone to creating content that looks complete and readable to a human – but this sometimes means it will add information that may not be true. This is called a ‘hallucination’. Monitoring and reducing these types of errors is a relatively new science, and needs careful monitoring.
Reliance bias: While a clinician may be able to be vigilant to errors initially, over time they become used to systems and trust them more, biasing to trust the outputs. This ‘reliance bias’ will become more and more of a problem in AI as we go mainstream, and is an important risk to be aware of.
Medical device regulation: NHS England guidance has issued specific terms for ambient voice technology, and the MHRA is periodically reviewing technologies. For medical device errors, the Yellow Card system (like for drugs) can be used to report errors that the MHRA can then investigate should they receive a lot of them.
Monitoring: Any medical device specifically needs to be continuously monitored for errors – how this is evidenced and provided by the vendor is really important, and whether it’s in real-time or periodically tested, the organisation needs to be comfortable it is being monitored for patient safety.
Other resources
That’s everything you need to know about AI and governance to start to deploy safely. There are several outsourced companies that can help with assurance, as well as free resources available to NHS organisations. Unfortunately, ignorance of the guidelines is no defence.
Here are some more resources to help with this – and rely on your organisation’s governance teams for support with specific questions as needed:
- NHS England Guidance on AI-enabled ambient scribing products
- MHRA Strategic Approach to AI
- NHS Data Security and Protection Toolkit
- DCB0129/160 Clinical Safety Standards
Please also feel free to reach out to us anytime if you have a question about our systems.
In May, TORTUS AI, whose Ambient Voice Technology (AVT) was described as a ‘game changer’ by Health Secretary Wes Streeting, confirmed a strategic partnership with X-on Health, the largest primary care telephony provider in the UK, serving over 3,500 GP surgeries.
Surgery Intellect, powered by TORTUS, is a voice-enabled AI assistant that uses ambient voice technology (AVT) to listen, transcribe and code consultations in real time. Find out more here > https://www.x-on.co.uk/surgery-intellect/