Use of AI ‘does not predict specific rating’ during inspections, says CQC
The CQC has clarified its position around providers using artificial intelligence (AI), adding that AI use will not be used to predict a specific rating during inspections.
The watchdog said that it has a role in ‘encouraging the use of AI’, but listed a series for risks providers should keep in mind before using the technology.
It acknowledged that AI is now used to ‘write up notes and consultations’ in GP settings, but warned that if things go wrong ‘accountability and liability can be unclear’.
The CQC is currently consulting on a framework for assessing GP practices, including expectations for providers to use artificial intelligence (AI).
The use of AI and other ‘innovative technology’ is incentivised in the draft framework, ‘to ensure people have timely access to care, treatment and support’.
To achieve an ‘outstanding’ rating, the document suggested that ‘staff identify, assess and implement new and innovative approaches to delivering care, for example artificial intelligence in clinical care pathways’.
However, the watchdog has now clarified in new guidance that the absence or presence of AI ‘does not predict a specific rating’.
It set out a list of principles that providers should follow (see box) and said that providers must carry out Data Protection Impact Assessment (DPIA) to assess and document risks to privacy and data subject rights to ‘understand and minimise’ any interference with people’s rights, to enable lawful use of AI.
It said: ‘Where providers use AI, we must know whether AI facilitates high-quality, equitable care. We are considering any implications on how we apply our new assessment frameworks and what additional guidance might be needed.
‘Our new assessment frameworks will be rolled out alongside supporting guidance for providers. As technology including AI evolves, we will keep requirements for AI-specific guidance and training under review.
‘We may develop AI guidance tailored to providers across different sectors and settings alongside building internal capability.’
What does it mean for providers?
Using AI to provide health and social care in line with the regulations means providers must follow the following principles:
- AI to support, not to replace: AI can enhance, but not replace human decision making.
- Human oversight: AI outputs and processes are continuously monitored and evaluated.
- Transparency and choice: People who use services have appropriate information to make informed decisions about their care, including the role of AI in care pathways. Non-digital routes to care are offered if needed, also considering digital skills and connectivity.
- Safety and reliability: Providers ensure that introducing and using AI provides safe care and equitable outcomes for people.
- Security: The technology is resilient to cyber-attacks and sensitive information is stored and processed securely to comply with the General Data Protection Regulation.
- Fairness and impartiality: Good AI accelerates equity of care in terms of access and outcomes. For example, it provides accurate results across geographies and population groups and any potential known biases are mitigated.
- AI readiness and training: Those using AI are sufficiently trained and confident in using the technology, and they assess whether it can be integrated into existing pathways.
- Effective governance: There are effective mechanisms such as risk assessments to ensure AI contributes to high-quality, equitable care, and established systems to recognise, report and investigate when something goes wrong so that lessons are learned and solutions implemented.
- Data Protection Impact Assessment (DPIA): A DPIA assesses and documents risks to privacy and data subject rights to understand and minimise any interference with people’s rights, to enable lawful use of AI.
- Accountability: There are clear mechanisms for addressing issues or harm caused by AI.
- Procurement: AI tools are procured in line with relevant regulatory standards.
Source: CQC
It also said that it is exploring the potential use of ambient voice technology to ‘support our inspection and regulatory activity’, and asked providers for their views on this.
It said: ‘AVT involves tools that automatically capture and transcribe spoken conversations, such as those between staff and people using services, or during inspection activity, to support documentation and evidence gathering.
‘These tools are increasingly used across health and social care, and we want to better understand providers’ views and experiences.’
The CQC will start piloting its new assessment framework specific to primary care next this month, with inspectors using ‘Word templates’ initially while new digital systems are built.
The watchdog is planning an ‘initial sample’ of 40 inspections beginning this month, and the pilot will continue over the summer and autumn.
