Read the Portuguese version here.
As he returns to his spacecraft, astronaut Dave Bowman gives a clear command to the system that controls all the equipment on board.
– Open the pod bay doors, HAL.
But HAL 9000 refuses to obey, arguing that Dave’s presence aboard the spacecraft would pose a threat to the mission it had been assigned.
The scene is from Stanley Kubrick’s 2001: A Space Odyssey (1968), and it brings to the surface anxieties and uncertainties that we are experiencing today.
We cannot really treat the film as a prediction of the future. Kubrick got the date badly wrong and overestimated the speed at which space exploration would advance (probably carried away by the space race that put the first man on the Moon the year after the film was released). But the film works very well as a metaphor for discussing the risks and concerns related to the advancement of artificial intelligence today.
Agentic AI is the term we now use for systems capable of assessing a context, creating a plan of action, and putting it into practice autonomously to accomplish a mission. HAL 9000 was a kind of agentic AI that made a decision consistent with what it interpreted as its mission: eliminate the human crew members who could put it at risk.
Needless to say, Dave did not agree with that reasoning.
Risk #1: AI can make the wrong decision
We are still getting used to dealing with the outputs of new artificial intelligence systems, which can produce results not only from deterministic rules but also by learning patterns from large volumes of data.
Until recently, if a system told me I could not withdraw money from an ATM, I assumed this was a rule established by my bank: a withdrawal limit for a certain time of day or insufficient funds. Rules deliberately inserted into the system by programmers.
This month:
- Netflix recommended a movie I hated;
- Gmail classified an important client’s message as SPAM;
- ChatGPT made a bad joke;
- Google Photos created a stylized picture of me in which I did not recognize myself.
These results were not necessarily produced through explicit rules and programmed deterministic logic (such as IF-THEN-ELSE). They come from systems that decide and act differently, also based on patterns learned from large volumes of data.
When there is no clear, declared rule determining the expected result, assessing whether decisions are right or wrong becomes more complicated. It depends on judgment based on context and perspective.
Errors and defects
A calculator that says 2 + 2 = 5 is certainly defective. The result is incompatible with what is expected.
But it is harder to say that HAL 9000 was defective in 2001. Was the decision not to open the doors for the astronaut aligned with what the system was expected to do in order to follow its directives and ensure the success of the mission?
If so, someone could argue that the AI worked perfectly: it decided in accordance with the model for which it had been trained and delivered what was expected of it, even though the result was catastrophic.
From a broader and more human perspective, judging right and wrong requires us to consider not only the output of the decision-making process, but also the impacts and consequences resulting from that decision: the outcomes.
Human beings also often make wrong decisions while following deterministic rules, without understanding a broader context or considering different perspectives.
Concerns for today’s analysts
Rule-based or pattern-based systems
As powerful as machine learning may be for assessing enormous volumes of information in seconds and making optimized decisions based on the available data, those who develop systems must take into account the potential impact of those decisions on people’s lives.
It is essential to distinguish decisions that need to be made deterministically, based on clear rules, from those for which probabilistic models may be appropriate.
Examples:
- Deterministic: determining whether my balance is sufficient to complete a transaction;
- Probabilistic: identifying a possible fraud situation and triggering an alert.
Hybrid models are likely to occupy an increasingly important place in today’s systems, combining different types of decisions within the same functionality to provide users with both security and convenience.
Agent autonomy
Even if an AI agent is capable of making decisions, should we give it the authority to execute actions based on those decisions?
I do not believe there is a single answer to that question. It depends on the context.
Whoever designs the agent will need to assess each type of authorization granted and define what kinds of review, approval, supervision, and monitoring are required.
In addition to the possibility of placing a human in the decision-making process (human in the loop) as a safeguard, it is necessary to keep records of all relevant decisions and actions so that the system’s behavior can be audited and monitored.
Continuous monitoring makes it possible not only to assess the quality of decisions against the impacts produced, but also to make adjustments. This may mean reviewing rules, limits, data, permissions, instructions, controls, supervision mechanisms, or even reconsidering the degree of autonomy granted to the AI.
Corrections
When necessary, it must be possible for humans to override automated decisions.
That is how astronaut Dave managed to shut HAL 9000 down and prevent the worst from happening. It could have been easier, of course, but then the movie would not have been nearly as interesting.
If you are creating an AI agent today, pay attention to these risks and make sure the safeguards are there to protect us in 2001. Or in whatever year Kubrick would have liked to have guessed correctly (2026?).
About this article
This is the first article in a series dedicated to the FABR Framework (Five AI Business Risks), in which I illustrate each risk with a movie classic.
I hope these articles inspire business analysts to take the lead in AI development within their organizations, bringing important discussions to the table, clarifying the changes underway for those involved, and adopting measures that can prevent potential harm to the businesses they serve, their customers, employees, and society at large.
Next week’s question addresses the risk of bias and discrimination:
Is AI fair?
References
- A call to leadership: #BA4AI – Business Analysis for Artificial Intelligence
- Learn more about the FABR Framework – The Five AI Business Risks
- A practical tool: FABR Framework Checklist
