Is AI responsible?

Who is accountable when an AI-supported decision causes harm? Explore why explainability and clear responsibilities are essential.

A scene of the movie EX_Machina, adapted by ChatGPT to discuss AI accountability.
A scene of the movie EX_Machina, adapted by ChatGPT

Read the Portuguese version here.

In his laboratory, reclusive billionaire and technology genius Nathan shows programmer Caleb a gel-like structure that he presents as Ava’s ‘brain.’

Ava is an android with an almost human appearance, whose behavior was learned from enormous volumes of data extracted from the Internet and information obtained through digital devices.

Although Caleb already knows in advance that Ava is an AI, Nathan invites him to take part in a kind of variation on the ‘Turing test’: knowing that he is facing a machine, will Caleb be able to recognize behavior that he considers genuinely intelligent?

The film Ex Machina (2015) draws us into a suspense story in which we spend much of the time trying to figure out who the villain is.

The beautiful young woman or the drunken billionaire?

The android describes the abuse she suffers at the hands of her creator. He warns the programmer that he may be being manipulated.

There is no way to know exactly what Ava intends or to predict with certainty how she will behave.

After all, her behavior was not programmed as a simple sequence of deterministic rules that could be traced through a chain of IF-THEN-ELSE commands. It emerges from a complex artificial intelligence capable of learning patterns from enormous volumes of data.

Although it is fiction, Ava is the ideal metaphor for the final risk explored in this series of articles about the FABR Framework.

Risk #5: Explainability and Accountability

We understand how machine-learning-based AI models work, but what is produced by a complex model cannot always be explained simply, nor are its results 100% predictable.

Why does the hand of a character in an image I generated with AI have six fingers when almost every human hand, with rare exceptions, has five?

I have no idea! AI generates strange things sometimes.

I can live with that answer for some of the things I do with AI.

But not knowing why a sixth finger appeared in an image is very different from not knowing why an AI-supported decision affected someone’s life.

The inability to provide a clear and appropriate justification can make a model unsuitable for certain uses in which that explanation is essential:

  • Supporting judicial decisions that affect a person’s freedom;
  • Prioritizing investments in a portfolio;
  • Approving credit at a bank;
  • Selecting candidates to fill a job opening.

Explaining does not necessarily mean presenting a mathematical formula or a probability matrix that generated the output.

What is expected is a justification in business language that the people involved can understand, indicating the relevant factors and criteria considered in the decision and allowing the result to be properly assessed.

The greater the impact of the decision on stakeholders, the greater the need tends to be for an appropriate explanation.

Who is accountable?

Ultimately, the simplest explanation for a black-box AI would be:

‘Because the AI says so.’

  • The AI says the defendant is guilty.
  • The AI tells you to invest all your money in Bitcoin.
  • The AI denied your loan application.
  • The AI considered you unfit for the job.

Why? I have no idea. I am just a user of the system.

When appropriate, people affected need to be able to understand, question, and challenge decisions that have a significant impact on them.

You cannot blame a bad decision on AI and use that as a way to avoid responsibility.

Using AI does not transfer moral or organizational responsibility to the machine.

A machine cannot be held accountable for business decisions.

When something goes wrong, a series of questions needs to be asked. And although AI may be the object of the analysis, it should not serve as the answer to the question, ‘Who was responsible?’

  • Did the person who used it provide the correct data and apply the system in the appropriate context?
  • Did the people who developed the model define the limits and controls required for the context in which it would be used?
  • Did the people who structured the process establish the appropriate points for human intervention, judgment, and approval of sensitive issues?
  • Did the person responsible for approval make a careful judgment based on the available information and seek additional information when necessary?
  • Did the people who developed the solution ensure that the model was implemented and integrated in accordance with the specifications?
  • Did the people who prepared the data and trained the model use information that was appropriate and representative of the intended context of use, and make its limitations known?
  • Did the people monitoring model performance identify deviations and take appropriate action to prevent or repair harm?

Concerns for the analyst

Maintaining audit information that makes it possible to reconstruct what happened when a relevant decision is questioned is essential.

But it is not enough.

Going out to look for someone to blame after an incident occurs is awful and very unproductive. I suggest avoiding this problem from that perspective.

A better approach is to mitigate these risks by defining responsibilities before implementation and ensuring that everyone involved clearly understands the objectives they are expected to pursue and the outcomes that should guide their work.

As I listed above, responsibility for the behavior of a complex system may be distributed among several roles involved in its conception, development, implementation, use, and operation.

But that distribution needs to be specific about what is expected from each role.

Responsibility can be distributed. It cannot be undefined.

Defining the roles associated with each automated business decision is itself essential. We need to know who approves, uses, monitors, stops, and, when necessary, corrects each decision.

In Ex Machina, we cannot simply assign Ava responsibility for everything that happens.

She was designed, developed, trained, and placed in that situation by humans.

Just like the systems you are helping your company develop and use today.

Be responsible!

About this article

This article on Ex Machina completes the sequence of the Five AI Business Risks in the FABR Framework illustrated through movies.

If you felt somewhat uncomfortable, with a certain weight on your shoulders while reading the articles in this series, then I achieved my goal.

Our society needs people like you: people who pay attention to what is being produced, understand the risks involved, and are willing to lead the transformation caused by AI in organizations.

If you missed any article in the series, here they all are:

Other references