AI Agents On Premise vs In The Cloud, Part 1

Before AI came to dominate the picture, healthcare software was running almost exclusively on premise - on the hospitals’ own local servers. This was working, because the software was deterministic, meaning it followed clear rules. If this, then that; 2+2 equals 4. It’s basically a simple database lookup. The problem is it’s impossible to write down all the rules, which means you are limited in the problems that the software can solve and automate.

AI on the other hand is probabilistic and thus instead of using defined rules, it calculates the statistical probability of something being true. It does that for every word and pixel it reads and generates, which requires a massive amount of computing, in the range of 1 billion to 1 trillion times more than a deterministic software. This type of performance requires a different type of chip - a GPU - which can handle the workload and heat, which in terms require considerably more cooling and space, compared to a normal CPU. And to top it, the lifespan of GPU is 2-3 years (3x shorter than CPUs), which means you need to replace them pretty fast.

Tech companies have always benefitted from avoiding that on-premise build up and instead utilising the cloud and paying for the service of being able to flexibly use as much compute as they needed. The arrival of AI models and the need for GPUs made this even more obvious. 

So do hospitals also need to transition to the cloud in order to use and benefit from those AI tools? I just made the case how expensive that is, so now we need to understand in which cases it is worth it.

To begin with, certain hospitals already have on-premise GPUs and will continue to do so, for example university clinics with research institutes attached to them. Scientists get the GPUs for their research, use them to build a model and that model can then be deployed locally. There might be a need to expand the GPU cluster in order to accommodate for the extra inference workload coming from the clinic, but that’s a marginal cost in this case. More on that in a second (or tomorrow).

Most other hospitals, medical centres, specialised and family practices have neither the access to, nor the funding for, nor the space to install GPU clusters. Here the choice isn’t what setup to use for AI; it’s whether we want to use AI in the only setup economically possible - in the cloud.

In Germany there are 35 university hospitals, plus a few private ones, which are debating the cloud vs on-prem argument. The other 2,000 already know the answer.

To be continued to tomorrow.

UPDATE: Read Part 2 here. Read Part 3 here.