In July, Babylon Health released the results of their testing against the MRCGP (Member of the Royal College of General Practitioners) exam based on publicly available questions. As we reported at the time, its AI system passed the exam with a score of 81 percent. A separate test where Babylon worked with the Royal College of Physicians, Stanford University and Yale New Haven Health subjected Babylon and seven primary care physicians to 100 independently-devised symptom sets. Babylon passed with an 80 score.
Now these results are being questioned in a letter to The Lancet. The authors–a medical doctor and two medical informatics academics–argue that the methodology used was questionable. ‘Safety of patient-facing digital symptom checkers’ shows there ‘is a possibility that it [Babylon’s service] might perform significantly worse’. The symptom checking methodology was questioned for not being real world–that the data in the latter test was entered by doctors only, not by patients or other clinicians. While the authors commended Babylon for being open about their research, they felt there was an “urgent need” for improvements in evaluation methods. “Such guidelines should form the basis of a regulatory framework, as there is currently minimal regulatory oversight of these technologies.” Babylon promises a response and additional improvements, presumably from its $100 million investment in AI announced in September. DigitalHealth (UK), Mobihealthnews
Wed 13 December, 9.00am-4.30pm
Horizon Leeds, Kendall Street, Leeds
The King’s Fund is hosting a December conference in Leeds on the digital sharing of health and care records. Delivering the key benefits of coordinated care requires three things: the appropriate technology, the right governance structure and a culture of adoption. Attendees will learn more at this full-day event about:
- The direction of national programmes on interoperability and data sharing across and between local areas
- Case studies from around England where teams have developed ways to share health and care records locally
- The challenges involved in implementing data sharing across and between local areas and learn how others have overcome them
Keynote speakers include Will Smart (CIO, NHS England), Prof. Maureen Baker (Chair, Professional Record Standards Body), Andy Kinnear (Director of Digital Transformation, NHS South, Central and West Commissioning Support Unit and Chair, BCS Health), Nicola Quinn (Project Manager, Health Informatics Unit, Royal College of Physicians), and Jan Hoogewerf (Programme Manager, Health Informatics Unit, Royal College of Physicians).
For complete information, agenda, and to register, click on the sidebar advert or here. TTA is pleased to be a long-time supporter of The King’s Fund and a supporter of this event. Hat tip to Claire Taylor of The King’s Fund–if you are interested in supporting this conference, contact her here.
The Royal College of Physicians has just published app guidance that, according to EHI “doctors should only use medical apps with an official CE mark”. EHI goes on to clarify that the guidance “applies to medical apps that can be classed as medical devices, which are bound by EU law to carry the mark.”
The Digital Health & Care Alliance (DHACA), of which this reviewer is Managing Director, is extremely concerned that this advice may seriously impact on the beneficial use of medical apps in the UK as it places the onus of deciding whether an app is a medical device on individual clinicians, a decision that at times even experienced MHRA personnel can equivocate on.
As the original research done by this editor on the topic of medical app take-up demonstrated, clinicians (more…)