Diagnostic Subjectivity, Single vs Multi-Model Strategies

It is a well established fact that a single-person view is inherently subjective. We all have our blind spots, no matter how much of an expert somebody might be. The solution is logical: assemble a team of experts to cover as much perspective as possible and challenge each others conclusions and thought process. The only issue with that solution is the economic imperative - it’s too costly to have multiple people looking at the same problem.

The same problem has been widely documented when it comes to diagnosing diseases and performing different assessments. Indeed, it is exactly the selling proposition of many AIs that they can now finally overcome that subjectivity thanks to being trained on thousands of images and thus reaching a level of expertise few humans are capable of reaching. And yet, despite often beating human performance benchmarks in lab conditions, they struggle to do so in real life.

The reasons are many - from differences in data produced using device A vs device B, to edge cases, to hallucinations - but the problem remains the same: no single model can produce reliably objective diagnosis. The solution should then follow the same line of thought - add more AIs, that are trained on different datasets, to analyse the test, provide a second opinion and then also work out what the takeaway should actually be. The difference with a human team - it’s considerably cheaper.

This Multi-Agent Conversation (MAC) approach has been tested for rare diseases by certain labs already and it seems to work quite well. I’m surprised it hasn’t been more widely adopted but I very much expect it to become the new standard.