Never attribute to malevolence that which can be explained by incompetence.

Including the end of humanity at the hands of the robots apparently

That reminds me of a fairly recent article about research around visualisation systems to aid with interpretable or explainable AI systems (XAI). The idea was that if we can make AI systems that explain their reasonings, then they can be a useful tool, especially in the hands of domain experts.

Turns out that actually, the fancy visualisations that made it easier to understand how the model had come to a conclusion actually made subject matter experts less accurate in catching errors. This surprised researchers and when they later tried to make sense of it, they realised that they had inadvertently dialled up people’s likelihood to trust the model because it looked legit.

One of my favourite aphorisms is “all models are wrong, some are useful.” Seems that the tricky part is figuring out how wrong and how useful.

It’s worse now than ever though, many managers have been steeped in tech optimism their whole working careers. The failures of “revolutionary new systems” have been forgotten about while the success of other things are lauded.

They’ve been primed to jump on any new “innovation” and at the same time B2B marketing has started adopting some of the most manipulative practices that used to be only used on consumers. They’ve crafted a narrative that shapes discourse so the main objections that appear are irrelevant to the actual issues managers might run in to.

Stuff like “but what if it is TOO good?!” and “what if the wrong people get their hands on this AMAZINGLY POWERFUL new tech?!”

Instead of “but does this actually understand anything or is it just giving output that looks correct?” or “ Wait, so, how was this training data obtained? Will there be legal issues from deliverables made with this?”

The average manager has been primed by the zeitgeist to ask the sales rep the kinds of questions they want to answer.

Seems to me that a lot of the world’s problems start with “well, the managers think…” They all seem extremely bad at the whole managing thing, good thing we don’t overpay them or anything like that.

It is quite terrifying that people think these unoriginal and inaccurate regurgitators of internet knowledge, with no concept of or heuristic for correctness… are somehow an authority on anything.

you fool

“these are chatgpt’s recommendations we just provided research to back them up and verify the ai’s work”

“It came up with more or less the same recommendations. Though it didn’t fully understand the specific target goals of your project, so our recommendations are more complete and actionable ready.”

I think this points to a large problem in our society is how we train and pick our managers. Oh wait we don’t. They pick us.

I mean, as long as you are the one prompting ChatGPT, you can probably get it to spit out the right recommendations. Works until they fire you because they are convinced AI made you obsolete.

There are some very impressive AI/ML technologies that are already in use as part of existing medical software systems (think: a model that highlights suspicious areas on an MRI, or even suggests differential diagnoses). Further, other models have been built and demonstrated to perform extremely well on sample datasets.

Funnily enough, those systems aren’t using language models 🙄

(There is Google’s Med-PaLM, but I suspect it wasn’t very useful in practice, which is why we haven’t heard anything since the original announcement.)

They are already used in medicine reliably. Often. Welcome to the future. Computers are pretty good tools for many things actually.

I read the same thing in Nevvsweeek.

Exactly. So the organisations creating and serving these models need to be clearer about the fact that they’re not general purpose intelligence, and are in fact contextual language generators.

I’ve seen demos of the models used as actual diagnostic aids, and they’re not LLMs (plus require a doctor to verify the result).

Fair enough