In 2012, a few months into the Health Secretary job and with no background in health whatsoever, Jeremy Hunt started doing half-day shifts at Watford General, washing beds between patients in A&E.
At the latest SomX Healthtech Talks event, he talked about a ninety-year-old lady with dementia, brought in unconscious from a care home after falling. Surprising to him was that nobody knew a thing about her. Not her history, not her medications, not her allergies. And they were about to start giving her drugs.
“Because she had dementia, even if she woke up, she wouldn’t have been able to tell them.”
At that point, there was just one hospital in England where A&E staff could read a GP’s digital record. The promise of a paperless NHS by 2018 followed, which is what I reminded him of at the top of the event. Maybe unkindly, but hey, a front-bench Tory MP’s faced way worse than me and I knew he’d have an answer for why he didn’t achieve it.
It’s not as if he wasn’t close. Fourteen years on, and every GP record in the country is electronic. Every A&E can pull a summary care record. Forty-two million NHS App accounts have been registered. The specific problem Hunt walked into in Watford is, broadly, solved.
But look at what hasn’t been solved… If that woman came through the doors tonight, there’s a world where an AI model could read her whole record, flag her interactions and propose a plan before the registrar had finished washing their hands. But it wouldn’t be permitted to act on any of it without a human signing underneath. Not because it can’t (arguably). But in part, because we haven’t decided who’s liable when it's wrong.
Is the human in the loop a scapegoat?
Picture what the human in the loop actually is. It’s not a committee, it’s a registrar at 2am, holding 14 patients in their head, looking what an AI has outputted and clicking something to make a decision.
The first is clinical. Someone trained looks at the output and decides whether it makes sense for this patient, tonight. That is the job everyone assumes is happening. Fine.
The second is legal. The click puts a name on the decision. It routes the consequence to a person who has an employer, a regulator, a defence organisation and a professional registration to lose. It turns a risk nobody can assign into one that lands somewhere specific.
What sucks for that registrar is that the second job is being done reliably and the first one is not at all.
“The reason that there’s a human in the loop at every decision is because the risk sits with the clinician in the hospital.”
Jack Needham, Founder of Sanctuary Health was right when he said this. The clinician is in the loop because the clinician is the only component of the system anyone has worked out how to insure.
Hunt, earlier in the evening, quoted someone telling him that every surgeon carries a private graveyard in their head, the patients they lost and still turn over. That’s grief. It’s also, coldly, a functioning liability mechanism. Behind it sits a century of case law, a claims process, an indemnity market and someone who can be sued. An algorithm has no graveyard, and nobody standing behind it.
Algorithmic autonomy
So… if you can solve the underwriting, does algorithmic autonomy follow?
