Pioneering AI Frontier: Integrated Knowledge Bases

Integrated Knowledge Bases – AI Midjourney generated image

In this second article in the series, I will explore a possible path for AI advancement that must have a profound impact on our society: access to integrated knowledge bases.

With the help of specialized applications, we will be able to enhance our analytical capacity on an unprecedented scale, predicting trends and taking action more quickly and accurately to address problems or explore opportunities.

In a previous article, I invited you to dream with me about the future of how Artificial Intelligence (AI) will serve as a natural language interface for all technological applications. Beyond answering questions like today’s ChatBots, AI-accessed applications will act as automated agents capable of performing tasks and recording transactions, coordinating various sequences of unprogrammed activities. AI itself will dynamically define the tasks needed to meet a user’s goal. This transformation will make technology more accessible and user-friendly, as well as much more powerful and productive.

Human-machine interaction using natural language conversation, natural style, super-8 color schema; Created by the author using Midjourney AI

Read the first article of the series in full at Pioneering AI Frontier: Unleashing Natural Language Interface.

I want to emphasize that in this series of articles, my aim is not merely to present the current state of technology. My goal is to envision the next stages of evolution to guide research and development investments. It’s not about predicting the future but rather designing it. Join me on this journey.

Integrated Databases

The trend of integrating databases has been happening independently of AI technologies and can be observed in various corporate initiatives. For example:

Governmental: aiming for integrated control of business and citizen taxes.
Banking sector: integrating credit information, accounts, and financial transactions.
Healthcare: sharing patient records and examination histories.
Commercial: Customer Relationship Management (CRM) systems that integrate customer data and preferences to identify greater business opportunities in personalized campaigns and actions.
Management: monitoring transactional data from business operations to generate indicators at various levels.

Integration provides higher data quality, avoids redundancy, and enables cross-analysis to identify deviations and trends.

Integrated Knowledge Bases

In the future, AI will have access not only to integrated databases but also to knowledge bases. A database and a knowledge base are two distinct things, even though both are used to store information.

an organized shelf of file folders next to a large library – IA Midjourney generated image

A database is a structured collection of organized information accessible for efficient use by applications and systems that need to access data. It is primarily designed to store raw data such as numbers, texts, dates, etc. Examples of databases include Oracle, SQL Server, MySQL, and Postgres.

On the other hand, a knowledge base is a collection of information that goes beyond raw data. It involves processed and contextualized information, including links, cross-references, and relationships between different parts of knowledge, allowing for a broader and more integrated view that is easily understandable and accessible. Knowledge bases typically include interpreted and related information, specialized knowledge, and insights.

Examples of knowledge bases include encyclopedias, wikis like Wikipedia, and knowledge management systems within companies. Business knowledge can be maintained in the form of documents, manuals, policies, process diagrams, procedures, reports, enterprise architecture models, system documentation, best practices, and analysis results, among others.

In most organizations, knowledge bases are not well-structured and are rarely updated. Much of the business knowledge is tacit and exists only fragmentarily in the memories of long-serving employees who have accumulated a wealth of experience. When such employees retire or move to another company, part of the business knowledge is lost, and much needs to be recreated or rediscovered exploratively in reverse engineering initiatives.

A broad and well-structured knowledge base could avoid rework, train new people, serve as a basis for gap analysis, and reduce the business’s dependence on individuals’ memory and goodwill. However, maintaining an updated knowledge base is a significant challenge. Structuring information in an organized and consistent manner requires modeling knowledge and recording it in expert systems capable of creating links between information and validating model consistency.

As these tasks are difficult and labor-intensive, and their benefits will only be realized in the future, many companies do not invest in knowledge management and only address the problem when it arises. This is not a good strategy. It’s like a person who knows they have high cholesterol waiting for a heart attack to change their diet and sedentary habits.

AI Maintaining Up-to-Date Integrated Knowledge Bases

With AI-based expert systems, the maintenance of knowledge bases will increasingly depend less on human dedication and become a more straightforward, accessible, and cost-effective process.

Data has been made available in enormous volumes through transactional system logs, sensors, and devices in an increasingly connected world. Consider how connected you are. Banking systems monitor all your financial transactions; your phone stores all the places you visit and all your messages and personal communications; your smartwatch tracks your steps and heart rate, while the bioimpedance scale in your bathroom monitors not only your weight but also your body fat and various other indicators. Databases are expanding. However, turning all this data into knowledge would be a humanly impossible task without computational assistance, and AI plays a vital role in this process.

Currently, it is possible, for example, to map business processes through process mining systems, transforming data into knowledge. These systems assess the logs of transactional applications and outline how process flows are occurring in practice. This information is valuable for identifying deviations and inconsistencies compared to what was originally designed.

AI-based expert systems can extract information from documents, emails, meeting dialogues, and any other data sources to automatically keep business knowledge bases up to date. Inconsistencies and areas of concern are automatically identified and brought to the attention of humans with decision-making authority over what should be considered correct for the knowledge base. Note that human involvement is reduced to decision-making only for what AI cannot do on its own. Since it can learn from these decisions, the volume of exceptions gradually decreases, and the need for human intervention in knowledge base maintenance becomes less necessary, resulting in more consistent knowledge bases that are easier and cheaper to maintain.

AI with Access to Various Integrated Knowledge Bases

One of the most commonly used capabilities in AI-based systems is to map correlations and identify patterns in vast amounts of information. With access to more comprehensive and consistent knowledge bases, expert systems can identify deviations and trends with much greater precision.

As artificial intelligence gains access to integrated knowledge bases containing government information, healthcare data, financial data, citizen data, customer data, business policies and processes, and transactional data from corporate management systems, it can perform a wide range of tasks and offer significant benefits in various areas.

Here are some examples of what AI is already doing at some level and will do in an amplified way in the future:

Strategic Monitoring: Offering insights and recommendations for strategic decision-making by recommending changes in policies, processes, and systems to achieve goals and objectives. This includes generating dashboards for key performance indicators and scenario simulations.
Fraud and Attack Detection: Identifying suspicious patterns and fraudulent activities in real-time, issuing alerts, and taking automatic actions to interrupt and block activities, protecting institutions and their customers.
Accurate Medical Diagnosis: Assisting doctors in analyzing patient data, comparing their health history to that of their ancestors, and comparing it to other patients with similar profiles to make precise diagnoses and suggest personalized treatments based on the latest medical literature.
Epidemic Prediction and Public Health: Predicting disease outbreaks and efficiently allocating healthcare resources through real-time monitoring of public health data crossed with other non-official channels, such as news media and social networks to generate additional warning signs.
Personal Butler: Providing highly personalized assistance to each individual with personal assistants answering questions and solving problems more effectively. Just as today, everyone has a personalized mobile phone with all their contacts and message history, each person will have a personal assistant that accompanies and assists them in all their activities.
Regulatory Governance with 100% Compliance: Ensuring compliance of all business processes with government regulations by analyzing policies and monitoring transactional data to identify, alert, and correct any violations. Some kind of digital surveillance.
Demand and Inventory Forecasting: Predicting future demand and optimizing inventory to reduce costs and improve logistics efficiency in various sectors.
Urban Planning Assistance: Aiding in city management by monitoring and optimizing traffic, managing resources, and improving quality of life.

These are just a few examples of what AI can achieve when fed with a broad and integrated knowledge base. As technology continues to evolve, AI’s ability to make more accurate decisions and automate complex tasks is expected to grow significantly, positively impacting various sectors and improving efficiency and quality of life. All of this can be accessed by users through natural language, conversing with AI in their preferred language.

Challenges for this Future

To make this future possible, we still need to overcome a series of structural and cultural barriers in the current landscape involving organizations and individuals.

Non-Integrated Systems: Many systems still generate and handle data in isolation. Check within your organization how many controls are still maintained in some sort of Excel spreadsheet. Even corporate systems are often isolated in silos. I receive frequent calls from my phone company offering a service I already have because their sales and service systems are not integrated. There is still much work to be done to achieve database integration between different systems and data sharing across different organizational silos.
Data Quality: Data scientists often complain that most of their efforts are not spent generating information for decision-making. Much of their work consists of cleaning and classifying data from poorly disciplined data sources. Those working in business operations rarely understand how the data will be used for decision-making, which means they are not committed to proper data classification in the systems. If “garbage” goes in, “garbage” comes out. Ensuring the presence of quality data is a prerequisite for good analysis and good machine learning.
Culture of Competition: Access to information is a competitive advantage, causing competing companies to avoid sharing information. Collaboration within an industry often depends on the guidance of a regulatory authority, such as the central bank, the health or tourism ministry, or on associations that generate standards and norms, such as ISO or W3C. In many sectors, getting organizations and individuals to act collaboratively still requires a shift to a cooperative paradigm that is not always well understood.
Tacit Knowledge: Most current business knowledge is still only available in the minds of people who conduct and manage the business. Structured and updated corporate knowledge bases are rare to serve as quality input for machine learning. So, even if we want to use AI as a tool to update knowledge bases in the future, we need a more robust starting point, with expert systems accessing explicit and structured information in understandable patterns where AI can learn before it can progress further. The first steps will still require a lot of human work until the machines are trained to move forward.
Machine Learning Limitations: AI does not follow a predefined algorithm to perform its tasks. It “learns” from recognizing patterns in historical knowledge shared with it and uses that learning discerningly for decision-making or generatively to create new content. This machine learning-based functioning carries the risk of propagating errors and biases that can perpetuate injustices and reinforce biases. It is impossible to guarantee that the training data used for AI is 100% correct and fair. In addition to the effort to clean training materials, mechanisms of control that implement deliberate policies to promote fairness and prevent bias need to be established.
Knowledge Governance: What is truly true or false? Who has the authority to validate knowledge? And how is this knowledge validated? To allow knowledge bases to develop reliably, a governance structure must be designed that establishes roles, responsibilities, and validation authority, crossing different organizations and sectors in knowledge management federations. A current example is the collaborative and decentralized coordination structure of Wikipedia pages. Such structures will need to be implemented on different scales for all types of knowledge bases.
Privacy and Confidentiality Assurance: Access to personal information in knowledge bases must respect individuals’ rights to privacy. Although we have sought to clarify the rules related to this in legislation such as GDPR in Europe or LGPD in Brazil, many organizations still do not comply with these laws. Enforcement and compliance with laws are necessary. In the future, we may identify situations of conflict between individual rights and collective benefits regarding the sharing of confidential information. These cases involve ethical issues that need to be discussed and regulated in the relevant forums. Respect for existing laws is crucial to ensure rights. To share knowledge on a global scale, we will need to discuss and develop a unified legislation agreement.
Biases: Even though unprecedented, AI-generated content is based on statistical models based on the knowledge bases used in its training. This behavior leads to the replication of patterns and can reinforce prejudices and injustices. For example, if in the historical knowledge base of a financial institution, credit was more frequently denied to people of a certain ethnicity, AI may take this as a judgment criterion and repeat that judgment. It’s not an intentional configuration made by someone with ill intentions, but rather a machine learning based on statistical models. With the feedback of these results into the knowledge bases, statistical trends become more pronounced, and biases tend to be exponentially reinforced. Mechanisms must be created to prevent this behavior.
Quality and Truth Assurance: It’s becoming increasingly difficult to discern what is true or not. Of course, generating false content through text has always been easy, and this challenge didn’t originate with AI. However, especially in the case of images, audio, and video, forgery used to rely on specialized knowledge and equipment. With the use of AI, anyone can create a “deep fake”, fabricating false situations with real people. Simply request it, and AI generates an image of “Pope Francis parading in a samba school with carnival dancers in Rio de Janeiro.” Technical mechanisms and punitive regulations need to be established to restrict the use of AI as a means for spreading false content.

Pope Francisco parading in a samba school during the Rio carnival – Image created by IA Midjourney

The challenges listed here are not easy to overcome, indicating a considerable distance before this dreamt future becomes a reality. However, these are mapped challenges, and I believe they can be overcome with shared coordination and interest among people who dream and work together for a better future.

Conclusion

Despite the challenges, the future with the use of AI is promising. With natural language access to Artificial Intelligence capable of acting as agents accessing integrated and consistent knowledge bases, we can solve problems, automate complex tasks, and explore opportunities with agility and predictability to an extent previously unimaginable.

The challenges need to be tackled head-on so that society can achieve the best outcome from this envisioned future and ensure an improved quality of life for individuals and society as a whole. Our role is to be aware of these challenges and lead actions to successfully navigate this journey.

Please leave your comments and share how you think AI will impact your day-to-day life.

The final article in this trilogy will address the last track of evolution that makes up this envisioned future: AI dynamically reconfigured processes and policies. See you there!