Receive our
Monthly JGA Advisor

In our recent article, AI Governance Belongs in the Boardroom, Not the Server Room, we explored why firm leadership, not technology teams alone, must take ownership of AI governance. Governance establishes accountability. However, accountability alone does not prevent quality deficiencies. As firms increasingly deploy AI-enabled tools across audit execution and quality management processes, a new challenge is emerging. The very technology intended to improve consistency, efficiency, and audit quality may introduce new risks if governance, validation, and monitoring practices fail to keep pace. For Managing Partners, Chief Quality Officers, and SQMS leaders, the question is no longer whether AI should be adopted. The question is whether the firm’s system of quality management is prepared to govern its use. In this article, we examine a practical question that follows naturally from that discussion: What happens when governance exists, but the firm’s quality management processes fail to keep pace with technology adoption? Governance is Only the Beginning The governance discussion often focuses on who is responsible for AI. Equally important is how firms integrate AI into their systems of quality management. When firms deploy AI-enabled tools to support risk assessment, testing, supervision, or documentation, those tools become part of the firm’s quality response. Technology-related issues rarely present themselves as technology problems. More often, they appear as deficiencies in audit execution, supervision, documentation, or quality management. By the time those deficiencies become visible, the underlying technology considerations may have already affected multiple engagements. As firms evaluate the role of AI within their quality management, one governance question deserves particular attention: Who is accountable when the tool gets it wrong? While technology teams may support implementation, responsibility for how AI-enabled tools influence audit quality resides with firm leadership and the system of quality management. Leadership should evaluate whether AI-enabled tools align with firm methodology, support professional judgement, and introduce risks that require additional oversight. Firms create unnecessary quality risk when they treat AI primarily as an innovation or IT initiative rather than a quality management consideration. How AI Creates Quality Risks The use of AI does not change the auditor’s responsibilities. Requirements relating to audit evidence, professional skepticism, supervision, review, and documentation continue to apply. What changes is the way those risks may manifest. AI can accelerate processes, but it can also accelerate the consequences of weak controls, insufficient oversight, or flawed assumptions. The very technology implemented to improve audit quality may become the source of future inspection findings. AI introduces several audit quality risks, including: Over-reliance on automated outputs Reduced professional skepticism Inconsistent application across engagements Limited transparency around how conclusions are generated Insufficient documentation of judgment Unlike traditional technology risks, these issues may not be immediately visible. Deficiencies often emerge only after engagement teams have relied upon the technology across multiple audits. Firms may use AI-enabled tools to identify unusual journal entries or summarize large data populations. However, when engagement teams rely on AI-generated outputs without sufficiently applying professional judgment, skepticism, and client-specific knowledge, important risk indicators may be overlooked or insufficiently documented. This distinction is important because technology-related issues rarely present themselves as technology problems during an inspection, internal review, or remediation effort. More often, they appear as deficiencies in audit execution, supervision, documentation, or quality management. Through our work supporting firms with inspections, remediation initiatives, and quality management programs, we have observed that the underlying technology considerations are often identified only after broader quality concerns begin to emerge. Case Study: Accelerated Technology and AI Implementation Across our work with firms of varying sizes, we are observing a consistent pattern. Leadership focuses heavily on tool selection and implementation timelines, while significantly less attention is devoted to validation, monitoring, and ongoing evaluation. As a result, firms are discovering quality concerns only after the technology has already been deployed broadly across engagements. Consider a firm that adopted an AI-enabled risk assessment tool as part of its response to inspection findings related to audit execution and documentation. Leadership viewed the implementation as part of its remediation strategy and expected the technology to improve consistency across engagements. However, because validation, methodology updates, training, and monitoring failed to keep pace with implementation, engagement teams began relying on outputs that had not been sufficiently evaluated. Several challenges emerged. The firm had not fully validated the tool’s audit functionality, methodology updates were incomplete, training was limited, and accountability for oversight had not been clearly established. Subsequent post-issuance reviews identified engagement deficiencies directly tied to improper reliance on the tool’s outputs. By that stage, the tool had already been deployed across multiple engagements, amplifying the impact of those deficiencies. The lesson extends beyond implementation. Firms often devote significant effort to deploying new technology but considerably less attention to evaluating outcomes after deployment. Leadership should periodically ask a simple question: Is the tool improving quality? Without ongoing evaluation, firms may assume technology is achieving its intended objectives while quality risks continue to develop beneath the surface. Trusting AI Requires Validation Effective governance requires more than approving technology investments. At its core, validation is about answering a fundamental question: How do we know the output can be trusted? Leaders must understand how the firm validates AI-generated outputs and demonstrates that those outputs support audit objectives. How would the firm demonstrate to an inspector, peer reviewer, or internal reviewer that the tool was appropriately validated and monitored? Before deploying AI-enabled tools, firm leadership should be able to answer: How does this technology support the firm’s audit methodology? What quality risks does it introduce? How will outputs be validated? How will use be monitored across engagements? Final Thoughts Governance establishes accountability, but accountability alone does not ensure audit quality. Firms create risk when they treat AI implementation as a technology project instead of a quality response. The most significant AI risk facing firms today may not be the technology itself. It may be the assumption that implementation alone is sufficient. As firms continue adopting AI-enabled tools, leadership should consider a simple question: If this technology contributes to an engagement deficiency next year, can we demonstrate that we appropriately governed, validated, implemented, and evaluated its use? At Johnson Global Advisory, our perspective is informed by work performed across inspections, remediation efforts, technology risk assessments, and quality management initiatives. As firms continue integrating AI into audit execution and quality management processes, understanding how these areas intersect may become just as important as the technology itself.

As discussed in our prior articles, What Regulators Expect to See When AI is Used and AI Governance Belongs in the Boardroom, Not the Server Room, firms increasingly recognize that AI governance belongs within the system of quality management. However, inspection experience shows that even well-designed governance frameworks do not eliminate risk. Significant failures occur not only at the policy level, but also at the engagement level, where AI outputs are relied upon as audit evidence without sufficient validation. This article focuses on that execution gap. Specifically, it examines why validation of AI is emerging as one of the most significant audit evidence risks facing public company auditors today. For public company auditors, AI validation is no longer a technical exercise. It is an audit quality issue — and increasingly, an inspection issue. In the eyes of regulators, AI does not reduce evidentiary requirements; it changes how evidence must be evaluated, corroborated, and defended . How AI Changes Audit Evidence—and Raises the Validation Stakes PCAOB auditing standards governing audit evidence have not been rewritten for AI. The fundamental requirement remains the same: auditors must obtain sufficient appropriate audit evidence to support their opinion. What has changed is the evidence pipeline: when AI is used, outputs are often indirect (generated through models rather than procedures alone), abstracted (summaries, risk flags, or scores rather than raw data), and less intuitive to evaluate using traditional audit instincts. This creates a new risk: auditors may rely on AI assisted outputs without fully validating how those outputs were produced, what they mean, or whether they are reliable. From an inspection perspective, AI introduces a simple but critical question: How does the auditor know the AI result is reliable enough to rely on as audit evidence? Inspectors are increasingly focused on whether the engagement team can demonstrate the completeness and accuracy of inputs, the reasonableness of assumptions/logic (including prompts), the consistency and explainability of outputs, and the auditor’s independent evaluation and corroboration. A common misconception is equating firm tool approval (vendor diligence, IT review, or risk assessment) with audit evidence validation. Approval is necessary, but it is not sufficient: validation must occur at the engagement level, in the context of the specific audit objectives, data, and risks. Where AI Validation Commonly Breaks Down In practice, AI validation risk often arises in predictable ways: