Is AI honest?

Data, inferences, and AI agents create new privacy risks. Explore how to reduce exposure, malicious use, and unauthorized access.

Scene of the movie Eagle Eye, adapted by ChatGPT to discuss privacy and malicious use of AI.
Scene of the movie Eagle Eye, adapted by ChatGPT

When Jerry Shaw begins noticing strange behavior from the machines that are part of his everyday life, he has no idea that he is being manipulated by a massive system that knows practically everything about him.

ARII, short for Autonomous Reconnaissance Intelligence Integration, is a kind of artificial superintelligence with access to an enormous amount of information about people, their activities, communications, relationships, and behavior.

ARII can also access and control security cameras, telephones, traffic lights, and different connected devices, using that power to force Jerry and a stranger named Rachel to become involved in a mysterious mission.

The film is Eagle Eye (2008), starring Shia LaBeouf and Michelle Monaghan. It is far from being a classic and greatly exaggerates the ability to connect and control different devices, but it works very well as a metaphor for the AI-related business risk I want to discuss this week.

Risk #4: Privacy breaches and malicious use

Laws such as the GDPR in Europe and the LGPD in Brazil establish principles, rights, and obligations for the processing of personal data, including issues related to purpose, transparency, security, and limits on its use.

But complying with the law is only part of the problem.

Users rarely read the fine print in terms of use and privacy policies. They often accept conditions in order to access applications and services without fully understanding what data they are making available and what may be done with it.

The film addresses this issue in an interesting way by showing that it is not only the data a person provides directly that can reveal who they are.

AI can establish correlations and infer personal information that was never explicitly provided, by combining data and identifying patterns.

In the hands of people with malicious intentions, that information can be used to deceive, manipulate, and exploit.

It happened to my father, who fell for a scam when someone claiming to be his bank manager called and instructed him to make a transaction. It seemed strange, but the ‘manager’ knew so much about him that my father ended up believing him.

Every day, it happens to someone’s father.

Now imagine how much this kind of scam can expand with AI: more information, greater personalization, and the ability to operate at scale.

Concerns for the analyst

As someone responsible for developing AI systems that will be used in your business, you need to ensure compliance with the law.

But that is only the baseline.

Protecting the privacy and security of customers, suppliers, employees, and everyone else involved with your business needs to be a constant concern and part of solution design from the very beginning.

I will list three precautions that may be useful. But think of them only as a starting point.

Data minimization and protection

I am fine with Netflix using my viewing history and the ratings I have given to classify my profile and recommend other titles I will probably enjoy.

But if someone asks a chatbot which movies Fabrício has watched, I would not want that information to be disclosed.

Personal data used by AI systems need to be properly protected, and their use limited to the purpose for which they are needed. Depending on the context, anonymization or other techniques can be used to reduce the risk of exposure.

But be careful: simply replacing my name with XXX may not be enough to conceal my identity.

XXX, 51 years old, male, married, two children, also known as The Brazilian BA.

That does not sound very anonymous, does it?

Other information can enable re-identification through correlation. And AI can be particularly effective at finding patterns and establishing those relationships.

Protecting privacy requires us to think not only about isolated data, but also about what can be discovered by combining them.

Minimum access

Each application should have access only to the information it needs to perform its task, and nothing more.

With agentic AI, this care must extend to actions as well. In addition to accessing only the information it needs, an agent should receive only the permissions required to perform its function.

During analysis, define access according to purpose and specify which information may be accessed, stored, shared, and reused.

A query to my bank’s chatbot, for example, may temporarily give a conversational AI access to my account balance in order to answer a question.

That does not mean this information should be incorporated into the system’s memory, reused in another context, or made available to other users.

Access does not mean ownership. Context does not mean authorization for reuse.

Auditability

Your system needs to maintain appropriate records of who – human or system – accessed which information, when, and for what purpose.

This makes it possible to identify inappropriate behavior, investigate incidents, and assign responsibility when necessary.

Remember: even if your AI is working exactly as it should, someone with malicious intent may try to use its capabilities for a purpose other than the one for which they were designed.

The ethics of the machine

The idea of a malicious AI using information and access to manipulate people, as in Eagle Eye, is frightening.

And increasingly autonomous systems make this discussion particularly important. In experiments and applications, agents may pursue objectives in ways that were not anticipated or desired by those who designed them.

It is up to us to anticipate these risks and establish limits and barriers to reduce them.

But we cannot classify malice as an attribute of a system. AI is a tool, and malice lies in the intention of the person using it or in the irresponsibility of the person building it.

A knife can be used to kill or to spread butter on bread.

But there is an important difference between a knife and agentic AI.

The more autonomy and capacity for action we give a tool, the more careful we need to be in defining who may use it, what information it may access, and what it is authorized to do.

Honesty is a human virtue, not a software attribute.

Ethical principles cannot be something we simply expect a system to respect spontaneously. They need to be debated, clarified, and translated into requirements and business rules that guide the responsible development and use of the systems that run our businesses.

About this article

This article connects the fourth of the Five AI Business Risks in the FABR Framework with a story from the movies. I hope this series is inspiring you to think more deeply about the implementation of AI in your business.

Do not miss the previous articles, available in the references below.

Next week I will discuss the final risk, which deals with Explainability and Accountability:

Is AI responsible?

References