AI Validation Is the New Audit Evidence Risk

In our recent article, AI Governance Belongs in the Boardroom, Not the Server Room, we explored why firm leadership, not technology teams alone, must take ownership of AI governance. Governance establishes accountability. However, accountability alone does not prevent quality deficiencies. As firms increasingly deploy AI-enabled tools across audit execution and quality management processes, a new challenge is emerging. The very technology intended to improve consistency, efficiency, and audit quality may introduce new risks if governance, validation, and monitoring practices fail to keep pace. For Managing Partners, Chief Quality Officers, and SQMS leaders, the question is no longer whether AI should be adopted. The question is whether the firm’s system of quality management is prepared to govern its use. In this article, we examine a practical question that follows naturally from that discussion: What happens when governance exists, but the firm’s quality management processes fail to keep pace with technology adoption? Governance is Only the Beginning The governance discussion often focuses on who is responsible for AI. Equally important is how firms integrate AI into their systems of quality management. When firms deploy AI-enabled tools to support risk assessment, testing, supervision, or documentation, those tools become part of the firm’s quality response. Technology-related issues rarely present themselves as technology problems. More often, they appear as deficiencies in audit execution, supervision, documentation, or quality management. By the time those deficiencies become visible, the underlying technology considerations may have already affected multiple engagements. As firms evaluate the role of AI within their quality management, one governance question deserves particular attention: Who is accountable when the tool gets it wrong? While technology teams may support implementation, responsibility for how AI-enabled tools influence audit quality resides with firm leadership and the system of quality management. Leadership should evaluate whether AI-enabled tools align with firm methodology, support professional judgement, and introduce risks that require additional oversight. Firms create unnecessary quality risk when they treat AI primarily as an innovation or IT initiative rather than a quality management consideration. How AI Creates Quality Risks The use of AI does not change the auditor’s responsibilities. Requirements relating to audit evidence, professional skepticism, supervision, review, and documentation continue to apply. What changes is the way those risks may manifest. AI can accelerate processes, but it can also accelerate the consequences of weak controls, insufficient oversight, or flawed assumptions. The very technology implemented to improve audit quality may become the source of future inspection findings. AI introduces several audit quality risks, including: Over-reliance on automated outputs Reduced professional skepticism Inconsistent application across engagements Limited transparency around how conclusions are generated Insufficient documentation of judgment Unlike traditional technology risks, these issues may not be immediately visible. Deficiencies often emerge only after engagement teams have relied upon the technology across multiple audits. Firms may use AI-enabled tools to identify unusual journal entries or summarize large data populations. However, when engagement teams rely on AI-generated outputs without sufficiently applying professional judgment, skepticism, and client-specific knowledge, important risk indicators may be overlooked or insufficiently documented. This distinction is important because technology-related issues rarely present themselves as technology problems during an inspection, internal review, or remediation effort. More often, they appear as deficiencies in audit execution, supervision, documentation, or quality management. Through our work supporting firms with inspections, remediation initiatives, and quality management programs, we have observed that the underlying technology considerations are often identified only after broader quality concerns begin to emerge. Case Study: Accelerated Technology and AI Implementation Across our work with firms of varying sizes, we are observing a consistent pattern. Leadership focuses heavily on tool selection and implementation timelines, while significantly less attention is devoted to validation, monitoring, and ongoing evaluation. As a result, firms are discovering quality concerns only after the technology has already been deployed broadly across engagements. Consider a firm that adopted an AI-enabled risk assessment tool as part of its response to inspection findings related to audit execution and documentation. Leadership viewed the implementation as part of its remediation strategy and expected the technology to improve consistency across engagements. However, because validation, methodology updates, training, and monitoring failed to keep pace with implementation, engagement teams began relying on outputs that had not been sufficiently evaluated. Several challenges emerged. The firm had not fully validated the tool’s audit functionality, methodology updates were incomplete, training was limited, and accountability for oversight had not been clearly established. Subsequent post-issuance reviews identified engagement deficiencies directly tied to improper reliance on the tool’s outputs. By that stage, the tool had already been deployed across multiple engagements, amplifying the impact of those deficiencies. The lesson extends beyond implementation. Firms often devote significant effort to deploying new technology but considerably less attention to evaluating outcomes after deployment. Leadership should periodically ask a simple question: Is the tool improving quality? Without ongoing evaluation, firms may assume technology is achieving its intended objectives while quality risks continue to develop beneath the surface. Trusting AI Requires Validation Effective governance requires more than approving technology investments. At its core, validation is about answering a fundamental question: How do we know the output can be trusted? Leaders must understand how the firm validates AI-generated outputs and demonstrates that those outputs support audit objectives. How would the firm demonstrate to an inspector, peer reviewer, or internal reviewer that the tool was appropriately validated and monitored? Before deploying AI-enabled tools, firm leadership should be able to answer: How does this technology support the firm’s audit methodology? What quality risks does it introduce? How will outputs be validated? How will use be monitored across engagements? Final Thoughts Governance establishes accountability, but accountability alone does not ensure audit quality. Firms create risk when they treat AI implementation as a technology project instead of a quality response. The most significant AI risk facing firms today may not be the technology itself. It may be the assumption that implementation alone is sufficient. As firms continue adopting AI-enabled tools, leadership should consider a simple question: If this technology contributes to an engagement deficiency next year, can we demonstrate that we appropriately governed, validated, implemented, and evaluated its use? At Johnson Global Advisory, our perspective is informed by work performed across inspections, remediation efforts, technology risk assessments, and quality management initiatives. As firms continue integrating AI into audit execution and quality management processes, understanding how these areas intersect may become just as important as the technology itself.

WASHINGTON, D.C.: Johnson Global Advisory is proud to support Santa Monica College through a donation to its STEM Program—investing in educational opportunities that prepare students for careers in science, technology, engineering, and mathematics. Santa Monica College’s STEM Program provides students with access to high-quality academic resources, hands-on learning experiences, and pathways to transfer to four-year institutions and enter in-demand fields. By fostering critical thinking, innovation, and technical skills, the program helps equip students with the tools they need to succeed in an evolving workforce. Katherine Moe writes, “We are deeply grateful to Johnson Global Advisory for its sponsorship of Santa Monica College’s Launch the Future campaign and its investment in the next generation of STEM leaders and innovators. This support expands access to hands-on research, industry-standard technology, scholarships, mentorship, and professional connections—ensuring financial barriers do not stand in the way of talented students pursuing careers that will shape the future of science, healthcare, technology, and innovation.” "My connection to California makes this especially meaningful, " said Jackson Johnson, JGA President. "Supporting the Santa Monica College STEM program reflects our broader commitment to education, access, and we’re proud to invest in opportunities that help shape the next generation of leaders." About Johnson Global Advisory Johnson Global partners with leadership of public accounting firms, driving change to achieve the highest level of audit quality. Led by former PCAOB and SEC staff, JGA professionals are passionate and practical in their support to firms in their audit quality journey. We accelerate the opportunities to improve quality through policies, practices, and controls throughout the firm. This innovative approach harnesses technology to transform audit quality. Our team is designed to maintain a close pulse on regulatory environments around the world and incorporate solutions which navigate those standards. JGA is committed to helping the profession in amplifying quality worldwide. Visit www.johnson-global.com to learn more about Johnson Global.

Johnson Global Advisory is pleased to announce that Jackson Johnson, CPA, President, has been appointed to serve on the AICPA & NASBA International Qualifications Appraisal Board (IQAB). The IQAB is responsible for evaluating international accounting qualifications and facilitating mutual recognition agreements between the United States and other countries, helping to support global mobility and consistency in professional standards. “It’s an honor to serve on the IQAB and contribute to efforts that strengthen the global accounting profession,” said Johnson. “As the profession continues to evolve, collaboration across jurisdictions is critical to maintaining high standards and enabling greater mobility for accounting professionals worldwide.”

Few technologies have generated as much excitement—and as much promise—for accounting firms as artificial intelligence (“AI”). The potential to streamline audit execution, reduce hours, and enhance firm profitability is real and already being realized. However, AI does not simply change how audits are performed; it fundamentally alters how firms must think about oversight, responsibility, and quality management. As regulators sharpen their focus on AI‑enabled audits, firm leadership must move beyond adoption and address a more complex challenge: establishing clear and scalable AI governance. This article outlines why AI governance is now a strategic imperative for accounting firm leadership. As discussed in JGA’s article What Regulators Expect to See When AI is Used , inspectors do not evaluate AI tools in isolation. They evaluate whether the engagement team obtained sufficient appropriate audit evidence, exercised professional skepticism, and applied appropriate supervision and review when AI was used. Those expectations are grounded in existing auditing standards and apply regardless of whether AI was used for risk assessment, testing, or documentation support. Against that backdrop, AI governance is not simply about approving tools or managing technology risk. It is about ensuring the firm’s system of quality management supports consistent, supervised, and well-documented use of AI that aligns with audit objectives and withstands inspection scrutiny. When firms treat AI as an IT matter, governance discussions tend to center on 1) Data security, 2) System access, 3) Vendor due diligence, and 4) Infrastructure controls. Those topics matter—but they are only the baseline. Inspectors do not evaluate whether AI systems are well engineered; they evaluate whether AI enabled audit work complies with standards, supports professional judgment, and is governed within the firm’s system of quality management. In short, AI governance is a firmwide audit quality issue, not a back office technology function. Using AI does not change the auditor’s responsibilities. Requirements still apply when AI is used for 1) Audit evidence, 2) Professional skepticism, 3) Supervision and review, 4) Engagement partner accountability and 5) Firm level quality controls. From an inspection standpoint, AI introduces new audit quality risks, including: Over reliance on automated outputs Reduced professional skepticism (automation bias) Inconsistent application across engagements Insufficient documentation of judgment Lack of transparency around how conclusions were reached These are not IT risks—they are audit quality risks. AI Touches Nearly Every Component of a QC System Under modern quality management frameworks (including PCAOB QC 1000 , AICPA SQMS No. 1, IAASB ISQM 1), AI affects nearly every component of a firm’s QC system, not just technology or data governance. 

Johnson Global Advisory ("JGA") is proud to announce that Joe Lynch, Shareholder, will be speaking on a panel at the 41st Midyear SEC Reporting & FASB Forum . Joe will deliver the PCAOB update on June 5, with attendance available both in person and virtually. This panel will summarize the activities of the PCAOB including: Recite new requirements for the lead auditor’s use of other auditors Anticipate the new standard, “The Auditor’s Use of Confirmation” Enumerate the new requirements of QC 1000, “A Firm’s System of Quality Control” Recall the guidance of the new auditing standard “General Responsibilities of the Auditor in Conducting an Audit” Understand the amendments addressing aspects of audit procedures that involve technology-assisted analysis of information in electronic form Learn about the proposal to replace existing auditing standards related to an auditor’s use of substantive analytical procedures Anticipate other Standard-Setting and Research Projects Summarize PCAOB inspection findings and enforcement activities Understand recent PCAOB publications, including: Spotlight Publications Audit Focus Publications Data Points Publications Click here to register and learn more. Johnson Global partners with leadership of public accounting firms, driving change to achieve the highest level of audit quality. Led by former PCAOB staff, JGA professionals are passionate and practical in their support to firms in their audit quality journey. We accelerate the opportunities to improve quality through policies, practices, and controls throughout the firm. This innovative approach harnesses technology to transform audit quality. Our team is designed to maintain a close pulse on regulatory environments around the world and incorporates solutions which navigates those standards. JGA is committed to helping the profession in amplifying quality worldwide. 

Johnson Global Advisory (JGA) has submitted its response to the PCAOB’s request for input on its 2026–2030 strategic priorities. Drawing on extensive experience supporting firms subject to PCAOB oversight, JGA’s comments emphasize a more modern, risk-based approach to regulation focused on audit quality, scalability, and transparency. View JGA's comments here. Johnson Global partners with leadership of public accounting firms, driving change to achieve the highest level of audit quality. Led by former PCAOB staff, JGA professionals are passionate and practical in their support to firms in their audit quality journey. We accelerate the opportunities to improve quality through policies, practices, and controls throughout the firm. This innovative approach harnesses technology to transform audit quality. Our team is designed to maintain a close pulse on regulatory environments around the world and incorporates solutions which navigates those standards. JGA is committed to helping the profession in amplifying quality worldwide.

In our work with firms, we have seen a clear shift in how monitoring and remediation are viewed under modern quality management frameworks. They are no longer treated as retrospective compliance exercises. Instead, engagement deficiencies are increasingly used as meaningful inputs into an ongoing, risk-based system designed to identify issues early, address them thoughtfully, and reduce the likelihood of recurrence. Regulatory messaging reinforces this evolution. Oversight bodies are signaling a shift in focus from isolated engagement outcomes and more on whether firms have a system of quality management that consistently detects quality risks, responds appropriately, and demonstrates that remediation is working in practice. Based on our experience, while individual engagement deficiencies remain important, the more critical question is becoming how firms analyze, respond to, and learn from those issues over time. Engagement Deficiencies Are Signals, Not Endpoints Engagement deficiencies can surface through many channels, including pre-issuance reviews, internal inspections, post-issuance reviews, peer reviews, and regulatory inspections. Regardless of source, firms benefit most when these findings are evaluated through a consistent quality management lens. In practice, we encourage firms to look beyond whether a single engagement fell short . The more meaningful consideration is whether the deficiency points to potential weaknesses in governance, methodology, training, supervision, resourcing, or monitoring activities. We often observe that when issues are quickly labeled as engagement-specific, without assessing whether they reflect broader quality risks, valuable insight is lost. Modern quality management frameworks are designed to use these signals to strengthen the system, not simply close individual findings. What Effective Monitoring and Remediation Looks Like in Practice Firms that navigate this environment effectively tend to apply a disciplined and repeatable approach when deficiencies are identified. Based on our experience supporting firms across a range of practice areas, several elements consistently make a difference: Assess whether the issue may be systemic Recurring observations across engagements, service lines, or time periods often indicate system-level risk. Similar documentation gaps, inconsistent application of methodology, or supervision challenges rarely arise in isolation. Perform meaningful root cause analysis Effective root cause analysis typically moves beyond surface explanations. Firms benefit from evaluating whether policies and procedures were designed appropriately, implemented as intended, and supported by sufficient training, time, and resources. Design remediation that directly responds to the quality risk Remediation is most effective when it is clearly linked to the underlying risk. Depending on the circumstances, this may include enhancements to methodology, targeted training, revised review requirements, or changes to engagement acceptance, staffing, or oversight processes. Validate remediation through timely monitoring Implementing corrective actions is only part of the process. In our experience, firms are most successful when they also confirm that remediation operates as intended. Follow-up monitoring performed early enough to prevent recurrence is a critical component of this step. Failure to validate remediation remains one of the most common and consequential weaknesses we observe across firms. Case Study: When Remediation Is Not Validated In one situation we encountered, a firm identified engagement deficiencies through post-issuance reviews. The issues mirrored observations that had previously been noted during peer review and were communicated as having been addressed by the group responsible for report issuance. However, responsibility for validation was not clearly assigned, and no follow-up procedures were performed to evaluate whether the revised processes were effective. Subsequent post-issuance reviews, triggered by an organizational change, revealed that similar and additional deficiencies had re-emerged. From a quality management perspective, this was not an engagement execution failure. It reflected a breakdown in monitoring and remediation. The firm had information indicating quality risk but did not adjust its monitoring activities to confirm that remediation was working. Viewed through a system lens, this represents a system-level deficiency rather than an isolated engagement issue. Quality Management Applies Across All Engagement Types Modern quality management frameworks apply across a firm’s assurance and attestation practice, including private company audits, public company audits, SOC engagements, nonprofit audits, and other services. Deficiencies identified in any practice area may signal broader weaknesses in: Governance and leadership Methodology and training Monitoring activities Remediation processes In our experience, firms struggle to maintain an effective system of quality management when certain practices are treated as exempt from system-level evaluation. Key Takeaways Engagement deficiencies are inputs into the system, not endpoints. Recurring issues often indicate systemic quality risk. Remediation should be validated, not assumed. Monitoring activities should evolve as risks emerge. Quality management applies across all engagement types. Firms that treat monitoring and remediation as a continuous feedback loop, rather than a periodic exercise, are typically better positioned to improve engagement quality and respond to evolving regulatory expectations. Looking for an independent perspective on whether engagement deficiencies have been fully addressed? Based on our experience working with firms across assurance and attestation practices, Johnson Global Advisory supports clients by performing independent reviews, validating remediation efforts, and strengthening monitoring processes. If you would like support refining policies, training, workflows, or documentation standards, or would benefit from an objective assessment ahead of regulatory, peer, or internal inspections, contact your JGA audit quality advisor to discuss your needs.

Artificial intelligence (“AI”) is no longer experimental in public company audits. From risk assessment and scoping decisions to population testing, anomaly detection, and documentation support, AI enabled tools are increasingly embedded in audit execution and workflow. As use expands, the auditor’s core obligations do not shift to the technology, they remain with the engagement team. If AI is used to inform judgments, influence the nature, timing, or extent of procedures, or summarize and interpret information, auditors must still demonstrate that they obtained sufficient appropriate audit evidence and applied professional skepticism throughout. In practice, auditors must understand what the tool is doing, confirm that inputs are complete and accurate, and evaluate whether the outputs are reliable and fit for purpose in the specific audit context. While the auditing standard devoted solely to AI have not been issued, our experience is that inspectors have been increasingly direct—through staff publications, questions from inspectors in the field, and public remarks—about what they expect to see when AI is used. The expectations are grounded in existing standards and longstanding inspection focus areas: audit evidence, supervision and review, professional skepticism, and firm quality control (now quality management). In other words, AI does not create a “new” audit; it amplifies the need to show your work. Firms that treat AI as a “shortcut”, rely on outputs that cannot be explained or reproduced, or fail to govern and document how tools were selected, configured, and monitored are inviting new risks to support their audit conclusions. Conversely, firms that can clearly articulate the purpose of the tool, how it aligns to audit objectives, how inputs and outputs were validated, and how experienced personnel supervised and challenged the results will be far better positioned during inspection. The table below summarizes what inspectors typically expect to see documented when AI is used in a public company audit. Firms can use these themes to evaluate whether their engagement documentation tells a complete story that an experienced auditor (and an inspector) can follow from objective, to procedure, to results, to conclusion. 

In a previous article, Back to Basics: Audit Documentation Failures Have Become Dangerous Low Hanging Fruit , we highlighted how audit documentation had quietly re-emerged as a source of regulatory risk after years of relative deprioritization. While PCAOB Auditing Standard 1215, Audit Documentation (AS 1215), has historically been cited less frequently than other standards, our direct experience from recent inspection activity, enforcement actions, and internal inspection results, demonstrate that documentation failures are increasingly treated as indicators of deeper execution, supervision, and quality management breakdowns. In today’s environment, audit documentation is no longer merely a record of work performed. It is the primary evidence inspectors rely on to evaluate whether an engagement was properly planned, executed, and supported at the time the auditor’s report was issued. What has been low-hanging fruit now requires firms to close these gaps and transform them into a load-bearing foundation for audit quality. From Rare Enforcement to Systemic Inspection Risk AS 1215 establishes clear requirements regarding what must be documented, when documentation must be completed, and how engagement files must be assembled and retained. As discussed in our prior article, failures to comply with these requirements were historically viewed as technical or secondary issues, often resulting in inspection comments rather than enforcement action. That distinction is no longer meaningful. Recent enforcement actions involving backdating, improper (both intentionally, and inadvertent) modification of workpapers, and failure to timely assemble a complete audit file reflect an evolving regulatory view. Documentation failures do not simply violate procedural requirements; they call into question the credibility of the audit opinion itself. More importantly, beyond enforcement, documentation deficiencies are increasingly cited as core inspection findings. Inspectors are challenging situations where engagement teams assert that work was performed but cannot demonstrate that work within the archived file. In these cases, the absence of timely, complete, and clear documentation is no longer treated as a formality. It is treated as evidence that the engagement may not have been properly executed, supervised, or supported in accordance with PCAOB standards. This represents a fundamental shift. Documentation is no longer “low-hanging fruit.” It is a systemic inspection risk that cuts across execution, supervision, and firm-level quality management. From Misconduct to Execution Failures Pervasive documentation failures that do not involve intentional misconduct but still result in non-compliance are increasingly observed. For example, reviewer signoffs occurring near the documentation completion date, rather than contemporaneously with the performance of audit procedures, raise questions about whether effective supervision occurred during the audit or was deferred to meeting archiving deadlines. Similarly, engagement teams may assert that key judgments can be explained verbally, even when those judgments are not clearly documented in the audit file. In today’s environment, the distinction between “we can explain it” and “it is clearly documented” is critical. If procedures, judgments, and conclusions are not evident in the documentation itself, inspectors increasingly conclude that the work was not performed in accordance with PCAOB standards. The issue is not whether the engagement team can explain what they did after the fact. The issue is whether the archived documentation allows an experienced auditor, with no prior connection to the engagement, to understand the procedures performed, evidence obtained, and conclusions reached at the time of the auditor’s report. When documentation fails to reach that standard, inspectors are increasingly concluding that the audit itself was not properly executed, regardless of intent. This reflects an important shift. Documentation failures are no longer viewed primarily as misconduct. They are viewed as symptoms of execution breakdowns, including delayed supervision, compressed review cycles, and audit workflows that defer documentation until the end of the engagement. As a result, AS 1215 has become a direct proxy for how audits are actually performed in practice. How the 14-Day Documentation Completion Requirement Changes the Risk Profile The execution risks are further amplified by the PCAOB’s shortened documentation completion timeline. Recent amendments to AS 1215 reduce the timeframe to assemble a complete and final audit file from 45 days to 14 days after the report release date. While this change may appear procedural, its implications are operational. Under this accelerated timeline, engagement teams no longer have a meaningful post-issuance window to resolve review notes, complete documentation, or finalize supervisory evidence. What were once viewed as “clean-up” activities are now more likely to result in timing violations and non-compliance. This shift places increased emphasis on: Contemporaneous documentation Real-time supervision Realistic workload and staffing models Audit Documentation as a Cornerstone of Audit Quality Audit documentation has long been described as low-hanging fruit in the inspection process. That characterization no longer reflects its role in today’s regulatory environment. Documentation now serves as the primary lens through which regulators assess whether an engagement was properly executed, supervised, and supported. With shortened timelines, expanded quality management expectations, and increased regulatory scrutiny, firms can no longer treat documentation as a downstream activity. It must be embedded into how engagements are planned, staffed, reviewed, and completed. In an environment where inspection conclusions are driven by what is, and what is not, in the audit file, strong documentation is not merely defensive. It is foundational to audit quality. At Johnson Global Advisory , we support firms in selecting, implementing, and optimizing these tools to meet their unique needs. For more insights, visit our blog or contact us to learn how we can help your firm AmplifyQuality®. For more information, please contact your JGA audit quality expert .