Last month, this Editor took note of the Twitterstorm around Babylon Health on the issues raised surrounding diagnosis of women’s cardiac symptoms. @DrMurphy11, who has been raising performance issues with the Babylon chatbot for the past three years, ran a test on the app. First using a male patient, then a woman, with identical cardiac symptoms, the app returned two different diagnoses: the man was advised to go to an ED on an emergency basis and given information on a heart attack, the woman to her GP in six hours and given information on a panic attack.
@DrMurphy11 came out earlier this week to BBC Two’s Newsnight’s Emma Barnett on a profile of ‘healthcare juggernaut’ Babylon as Dr. David Watkins, a consultant oncologist. You can see him on YouTube here (at the 1 minute and 3 min. 30 mark). He demonstrates the response of the chatbot, using as the patient an older male smoker with chest pains. The chatbot advises him that he might have either gastritis or ‘sickle cell crisis in chest’–and to go to his GP in 6 hours. What is far more likely than sickle cell with this history is, of course, a heart attack, as a consultant cardiologist, Dr. Amitava Banerjee confirmed on the program. Dr. Banerjee has also been critical of Babylon’s chatbot on cardiac diagnosis and Health Secretary Matt Hancock in his visible advocacy of Babylon in the NHS alone (at 6 min.) According to Dr. Watkins, he has been documenting chatbot problems to the MHRA and the CQC since 2017, and the problems haven’t been fixed.
Timed with the Newsnight piece, Babylon fired back with a press release labeling Dr. Watkins a “troll” and stating that only 100 of his 2,400 tests demonstrated any concerns with the chatbot. According to the release, Babylon’s staff “have attempted to start a positive conversation with this anonymous person. We have invited him in to start a dialogue, to test our AI, and to meet with the senior doctors who build our products” without response. Babylon has also cited that all of Dr. Watkins’ trials were theoretical tests and cites millions of real uses without a single report of harm, that it meets regulatory standards in five countries including use in the NHS, and that its real life users are highly satisfied (85 percent at 5 stars).
At 6:48 to 12:40 in the video, Newsnight’s Emily Maitlis grills both Babylon’s Dr. Keith Grimes and Dr. Watkins. She brings up that Babylon’s former head of regulatory affairs, Hugh Harvey, had stated that no one has assessed how well the app works. Dr. Watkins also counters Babylon’s non-contact claim that he contacted one of the Babylon leadership members back in 2018 on chatbot problems. Dr. Grimes responded to Ms. Maitlis’ remark that founder Ali Parsa is not a doctor that over 600 doctors work for Babylon. This Editor will leave it to Readers to decide what side won, or if it was a draw. Also Mobihealthnews global edition. (For US Readers, Newsnight and Ms. Maitlis conducted the exclusive, disastrous–for Prince Andrew–interview on his relationship with the late Jeffrey Epstein.)
Most Recent Comments