The Importance of Explainability in Artificial Intelligence (AI) in Governmental Accounting

by Preston Gilbert, Consulting Associate

Posted on May 22, 2024

AI is a buzzword that is being thrown around a lot these days. The term Artificial Intelligence was first coined in 1955 by John McCarthy, a Stanford University professor. He defined AI as “the science and engineering of making intelligent machines.” Thinking technology that can act, and problem-solve in ways that simulate human intelligence. A software that makes its own decisions. While artificial intelligence is unlikely to eliminate the need for flesh and blood accountants, it is likely to take a more integral part in data analysis and potentially replace the very manual process of reporting.

AI has come so far in such a short period of time. Below is a graph from ourworldindata.org that shows the progression of AI compared to humans in the last 25 years.

As of 2023, AI is consistently testing higher than humans in reading comprehension, image recognition, language understanding, and more. The now famous ChatGPT, a simple app that can be downloaded on our phones, that can write complete articles and research papers with our mannerisms in just minutes. All of this is amazing but can also be scary. Thus, the need for explainable artificial intelligence (XAI)

IBM defines XAI as “a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning (ML) algorithms, deep learning, and neural networks.” ML uses the data inputs it receives to make a prediction and if inaccurate then an engineer will step in to help shift its response. Deep learning can learn through its own method of computing using its own neural network. AI can run using both tools, but the more complicated the problem set the more likely a need for a deep learning network.

The picture above is a simplified model of deep learning. The hidden nodes are the “black box” part of the process. Techtarget.com defines a black box as any artificial intelligence system whose inputs and operations aren’t visible to the user or another interested party. A black box, in a general sense, is an impenetrable system. Black box AI models arrive at conclusions or decisions without providing any explanations as to how they were reached.“ Since these nodes cannot be decoded this is where XAI comes in.

XAI processes shed light on how the AI system, in the case above a deep learning model, reached its decision without knowing the internal step it took. Most XAI are additional ML models used to look at the strengths and weaknesses of the program, the criteria employed by the program, trustworthiness, and more. Common XAI models are decision trees, explanation graphs, and local explanations.

Explanation Graphs: the explanation graph shows the user’s history to represent why the model might have chosen a specific recommendation.

Decision Trees: uses a tree-like structure to show how factors might have influenced the decisions.

Local Explanations: if a model is used for recommending products, then local explanations show the products under consideration and the different reasons for selecting them.

Although XAI can shed light on the process as to why the model may have worked it cannot guarantee the deep learning model’s process. Different types of companies will have different requirements. For example, an audit firm will need a higher level of explainability compared to an internet streaming company. Therefore, it’s up to each organization to decide if they feel AI is right for them, using XAI to help in the decision-making process.

Whether it’s to help drive key financial planning or decision-making processes, AI’s continued growth will allow for more automation in governmental accounting. Long term, the industry will likely see greater efficiencies and less manual time on repetitive tasks. The current work with XAI is a good start to making sure checks and balances continue to be in place, but its growth is crucial for accessing the true potential of AI in governmental accounting.