Read the Portuguese version here.
Standing before an enormous screen, John moves his hands frantically, dragging images of the future from one side to the other to identify the person responsible for a crime that has not yet happened and that can be prevented thanks to technology.
The Precrime prevention system seems perfect to him: murders are prevented by arresting future murderers in advance.
His opinion changes when he sees his own face on the screen, identified as guilty of a crime he never imagined himself capable of committing.
The film is Minority Report, with Tom Cruise playing John Anderton, a kind of police officer who uses a crime-prevention system to arrest people based on predictions.
Predictions of the future are produced from the dreams (or something like that) of three people with supernatural abilities, the precogs. I am not going to get into that inexplicable detail of the plot. What interests me is using the film as a metaphor for the use of artificial intelligence technologies in decision-making.
How to predict the future
Suppose I needed to predict the future.
Since I do not know anyone with supernatural talents and cannot read people’s dreams, my strategy would be to turn to historical data, look for recurring patterns, and, based on a set of criteria, produce the best possible guess about what will happen.
That is how we try to predict the progression of a disease, the behavior of celestial bodies, next Monday’s minimum and maximum temperatures, or the number of tourists expected to visit Mecca in May.
Scientists do not have crystal balls. To predict the future, they learn from the past.
That is also how many machine-learning-based AI systems are trained. By analyzing volumes of data that humans would be unable to process in the same way, these systems can identify patterns, make inferences, predict trends, and support decisions.
From there, we can ask an AI to analyze a specific case and answer questions such as:
- Could this spot on the skin be cancer?
- Could this credit card transaction be fraud?
- Will this loan be repaid?
- Is that person more likely to commit a crime?
Questions like these are already being asked of AI systems in different contexts, and their answers need to be evaluated carefully.
A prediction is not the future. It is an inference built from data about the past.
Risk #2: AI can amplify bias and propagate discrimination
It is important to understand that the data used to teach a model how reality works are only a partial representation, never reality in its entirety.
The choice of which data will be used, which characteristics will be considered, and which criteria will be evaluated involves human decisions. And those decisions carry the perspectives, limitations, and judgments of the people who made them.
Historical data themselves are not neutral either. They record not only facts, but also previous human decisions and the consequences of those decisions.
These factors can introduce or reproduce bias in the model, which can turn into discrimination when it systematically contributes to unfavorable or unfair outcomes for certain groups.
Examples:
- Denying credit to start a business based on the neighborhood where a person lives;
- Classifying someone as more threatening based on their ethnicity;
- Rejecting a job candidate because of her gender or age.
I do not know about you, but to me these sound like very human decisions. I have seen injustices like these all my life.
Ultimately, all of us see the world from a perspective limited by what we know and the experiences we have had. We make decisions influenced by that history.
But we need to do everything we can to prevent past injustices from being learned, automated, and reproduced at scale.
Accurate models vs. fair models
A researcher who wants to assess voting intentions in an election needs to build a sample that adequately represents the population being studied.
If the researcher interviews only young, single women from a particular city, for example, it will be difficult to generalize the result to the entire country.
The quality and representativeness of the data are therefore essential to the quality of the inferences.
The same care applies to the training and evaluation of AI models. We need to ensure that the data used allow us to adequately understand the different groups and situations about which the system will make or support decisions.
But representativeness does not solve everything.
Even a model with excellent statistical performance can produce unfair results, especially when it learns from data that carry inequalities or discriminatory decisions from the past.
In addition, simply removing information such as gender, age, or ethnicity from the model does not guarantee neutrality. Other characteristics can act as indirect indicators of the same information. Machine learning systems can establish correlations and identify patterns that may not be obvious to us. And those patterns can be reproduced in their outputs.
A good example is the criticism directed at image-generating AIs, which can reproduce stereotypes and perpetuate prejudices present in the data and patterns on which they were trained.
That is why it is not enough to assess which data go into the model. We also need to observe the outcomes its decisions produce for different groups.
Fairness criteria are not obvious and can vary depending on the context. In some cases, maximizing overall accuracy may not be enough. It may be necessary to deliberately establish additional criteria, limits, or objectives to reduce unfair impacts.
In the end, fairness is not merely a technical property of the model. It is also a human decision about what we consider fair in that context.
Concerns for the analyst
Many decisions in your business are already (or will be) made or supported automatically by systems.
They need to be monitored constantly to ensure that they are reliable and based on appropriate models, but also that they produce fair results.
This requires looking beyond overall accuracy metrics. False positives and false negatives may need to be analyzed across different groups, as well as the outcomes produced by decisions.
Even when a model has excellent statistical performance, certain actions derived from its predictions may require additional analysis, human involvement, and contextual judgment.
- Identifying a probability of fraud is one thing. Automatically blocking a customer’s account is another.
- Estimating the risk of default is one thing. Automatically denying access to credit is another.
- Predicting the possibility of a crime is one thing. Arresting someone for a crime they have not yet committed is another.
In Minority Report, that last idea can even seem reasonable when observed from a distance, from the statistical perspective of public safety and harm reduction.
The perception changes completely when John Anderton sees his own face on the screen.
And it would probably change for any of us if we were treated unfairly.
Justice should be for everyone.
About this article
This is the second article in which I illustrate one of the Five AI Business Risks in the FABR Framework with a movie classic.
If you are responsible for business analysis in your organization, I call on you to take the lead in AI development. Bring important discussions into project meetings with your stakeholders. Help your company adopt measures to prevent injustice and ensure valuable outcomes for everyone.
Next week’s question addresses the risk of excessive trust in AI:
Is AI addictive?
References
- Article #1 in this series: Is AI right? – 2001: A Space Odyssey
- A call to leadership: #BA4AI – Business Analysis for Artificial Intelligence
- Learn more about the FABR Framework – The Five AI Business Risks
- A practical tool: FABR Framework Checklist
