Reliability in the age of AI

Being on the fourth year that I am running my game development company, there is one thing that I am always looking out for when choosing a new service, product or even long term collaboration.

And that thing is.. reliability.

It's a super simple rule on itself, but given Today's AI usage around the world... I think it has become important to share here anyways.

If my business needs something to operate, let's say.. an email or a website, then it means that it should always be working the way we expect it to.

If a company has an history of raising prices, having predatory cancellation fees or just losing focus and becoming the "everything app" - doing all the things mediocrely instead of excelling at one - then it's unlikely we will pick their services or products at all.

Everything occasionally stops working and can cause interruptions. But as a director you need to know what tools you depend on and who or what to trust.

Which brings us to AI.

AI is a stochastic parrot. The outcome you get is different based on whatever state it is currently processing. Even if you send the same prompt, you might have extra memories or chats unexpectedly interfering with your context window. Or they [Open AI, Anthropic etc] could be running out of GPUs and could lower everyone's AI "thinking" so that it stops earlier and gives you an half baked reply. They could even release an update of their harness (it seems to be daily at this point) and you suddenly have eaten half your weekly usage, just because they have introduced another token-consumption bug.

How can you depend on something that you cannot control and cannot estimate the outcome?

We are already seeing companies max out their yearly budget and being left without their new "workforce" [example]. Chances are that these companies produced more things due to AI though (not necessarily good) and are now dependant on it to browse the documentation, work on whatever fix/thing, or keep their chatbot/support agent running.

This simple rule, reliability, is already a big warning sign for teams adopting AI. It's not a matter of "it will improve in the future!" neither. The way it works is intrinsically probabilistic, and maybe an open source model that runs locally is the only way that could improve things a bit (but even then, there are so many other issues and warnings that it will still be difficult to use reliably).

If you are someone depending on it, especially if you just started out and want to learn the craft... be careful!