What to Measure
A common failure mode is to measure activity rather than outcomes — to count the number of ethics reviews conducted rather than the number of bad outcomes prevented. Activity measures are easy to gather but rarely answer the question that matters: is the ethics program making the AI program safer, fairer, or more trustworthy than it would otherwise be?
A working measurement framework includes four categories.
Coverage indicators measure whether the ethics program is reaching all the consequential AI activity in the organization. Examples: percentage of the AI estate that has gone through formal review; percentage of new AI projects that complete intake; percentage of vendor-supplied AI systems that complete enhanced due diligence; coverage of high-risk use cases (Article 9) by enhanced review.
Process indicators measure how the ethics program operates internally. Examples: median time from intake to design review; median time from design review to sign-off; rate of conditional approvals; rate of approvals overturned by escalation; rate of conditions verified versus conditions outstanding at sign-off.
Outcome indicators measure what the ethics program produces in the world. Examples: number of high-risk use cases declined or restructured before deployment; number of bias incidents detected pre-deployment versus post-deployment; number of public ethics incidents and the median time to remediation; user-facing measures such as complaint volume, override rates, and adverse-action contests.
Maturity indicators measure progress along the COMPEL D15 maturity scale (introduced in Article 1) over time. The maturity assessment is typically conducted annually and tracked as a trend.
The OECD AI Principles framework treats accountability as a core principle, and measurement is the operational expression of accountability; see https://oecd.ai/en/ai-principles. The NIST AI Risk Management Framework provides extensive guidance on measurement under its Measure function; see https://www.nist.gov/itl/ai-risk-management-framework.
Leading Versus Lagging Indicators
A mature program tracks both leading indicators (predictive of future outcomes) and lagging indicators (descriptive of past outcomes).
Leading indicators include coverage rates, training participation, audit findings open against the program, and time to remediate identified gaps. Leading indicators tell the program where it is likely to fail before the failure occurs.
Lagging indicators include the number and severity of public incidents, regulatory actions, customer complaints, and litigation. Lagging indicators tell the program what has already gone wrong and provide the strongest basis for organizational learning.
Programs that report only lagging indicators learn slowly and at high cost. Programs that report only leading indicators may report well while drifting toward outcomes the leading indicators do not anticipate. The combination is what supports both proactive improvement and accountability.
Audits
Audits provide the periodic deep examination that ongoing measurement cannot. Three audit types are standard.
Internal audits are conducted by the organization’s internal audit function (typically reporting to the audit committee of the corporate board) against the documented ethics policies and procedures. Internal audits address whether the program is doing what it says it is doing — completeness of records, adherence to defined process, accuracy of self-reported metrics. Internal audits should occur at least annually and should produce findings with required management responses.
External audits are conducted by independent third parties — typically professional services firms with AI ethics practices, academic teams under contract, or specialist auditors. External audits provide credibility that internal audits cannot, particularly for external stakeholders. The frequency depends on the organization’s regulatory environment and reputational exposure but typically ranges from annual to triennial. The IEEE 7000 family of standards is increasingly used as the audit reference; see https://standards.ieee.org/ieee/7000/6781/.
Algorithmic audits are technical examinations of specific AI systems, conducted against defined criteria (fairness metrics, robustness, explainability quality). Algorithmic audits may be conducted internally or externally. They typically focus on the highest-stakes systems and are increasingly required by regulation. The proposed Algorithmic Accountability Act would require impact assessments that include algorithmic audit elements; see https://www.congress.gov/bill/118th-congress/house-bill/5628.
The audit findings should be tracked to closure with named owners and target dates, the same as any other risk function findings. Findings that remain open past their target dates should escalate.
Reporting Internally
Internal reporting closes the loop between the ethics program and the rest of the organization. Three reporting cadences are useful.
Operational reporting is monthly or quarterly, addressed to the executive team and to operating function leaders. The operational report covers coverage, process, and outcome metrics, identifies emerging trends, and surfaces decisions that require executive attention.
Board reporting is quarterly to the corporate board’s audit committee or risk committee, and at least annually to the full board. Board reporting should include the maturity trend, the audit findings, the top three to five risks, and any incidents requiring board awareness. The board’s engagement with ethics is itself an important signal that travels down through the organization.
Incident reporting is event-driven. Material ethics incidents — public bias findings, regulatory actions, significant complaints — require immediate communication to the executive team and to the board’s audit chair, with a defined cadence of follow-up reports through resolution.
Internal reporting that is consistent, candid, and timely becomes the basis on which executive sponsorship is sustained. Internal reporting that is sporadic or rosy erodes credibility and ultimately erodes the program’s authority.
Public Reporting and External Disclosure
Public reporting of ethics indicators is the most consequential maturity step. It moves the program from an internal exercise to a public commitment that creates external accountability.
The depth of public reporting varies. A baseline public commitment includes publication of the organization’s AI ethics principles and an annual report describing the ethics program’s activity at a high level. A more substantial commitment includes publication of impact assessments for major AI systems, disclosure of incident counts and remediation, and disclosure of the organization’s positions on contested ethics questions in its industry. The most advanced commitment includes participation in third-party transparency indices and disclosure of audit findings.
Public reporting carries risk: published commitments can be measured against, published metrics can be compared with peers’ metrics, published incidents become reputational events. The risk is precisely the source of the value. A program that has nothing to fear from public reporting is a program that has earned the credibility it claims.
Several organizations have begun publishing detailed AI ethics reports. The Partnership on AI maintains a public-facing knowledge base and member transparency commitments; see https://partnershiponai.org/. The UNESCO Recommendation on the Ethics of AI calls for transparent reporting on AI ethics implementation; see https://www.unesco.org/en/artificial-intelligence/recommendation-ethics. The World Economic Forum has documented emerging public reporting practices; see https://www.weforum.org/topics/artificial-intelligence-and-machine-learning.
Reporting to Regulators
In an increasing number of jurisdictions, AI ethics reporting is required by law. The EU AI Act requires conformity assessments for high-risk systems and post-market monitoring reports. Several US states require bias audit disclosures for hiring AI. The proposed federal Algorithmic Accountability Act would create broad impact assessment reporting requirements; see https://www.congress.gov/bill/118th-congress/house-bill/5628.
Regulatory reporting and voluntary public reporting should be coordinated. Reports produced for regulators are typically detailed and structured; reports produced for the public are typically narrative and accessible. The two should tell consistent stories. Inconsistencies between regulatory disclosures and public communications create both legal and reputational exposure.
The Singapore IMDA Model AI Governance Framework provides guidance on regulatory reporting structure that is increasingly cited as a reference; see https://www.pdpc.gov.sg/help-and-resources/2020/01/model-ai-governance-framework.
Benchmarking
Benchmarking compares the organization’s ethics program against peers and against industry standards. Useful benchmarks include the COMPEL D15 maturity rubric, sector-specific maturity models (financial services has several, healthcare is developing them), third-party indices (Stanford’s Foundation Model Transparency Index, the Responsible AI Index from various publishers), and direct peer comparison through trade associations.
Benchmarking serves two purposes. It calibrates the program’s self-assessment against external reference points, reducing the risk of internal complacency. It provides a basis for prioritization — gaps relative to peers indicate where investment will be most visible to external stakeholders.
The risk of benchmarking is the temptation to optimize for the benchmark rather than for substantive outcomes. Benchmarks are imperfect proxies; programs that game them produce metrics improvement without meaningful change. The mitigation is to use multiple benchmarks, to be explicit about their limitations, and to maintain a focus on outcome indicators that benchmarks may not capture.
Maturity Indicators (For the Measurement Function Itself)
- Level 1: Ethics program activity is unmeasured.
- Level 2: Some metrics are tracked internally but inconsistently; no audit function.
- Level 3: Coverage, process, and outcome metrics tracked; annual internal audit; quarterly executive reporting; annual maturity assessment.
- Level 4: External audit on a defined cadence; incident tracking and root-cause analysis; metrics tied to objectives at executive and board levels; some public reporting.
- Level 5: Comprehensive public reporting; participation in third-party transparency indices; audit findings publicly disclosed; benchmarking against peers and standards; the organization’s measurement framework is shared with the broader community.
Practical Application
Three first actions. First, define the metrics — coverage, process, outcome, maturity — and instrument the existing process to capture them. The instrumentation does not need to be sophisticated; a shared spreadsheet with monthly updates is sufficient to begin. Second, commission a single internal audit against the documented ethics policies; the findings will identify gaps in the policies themselves as well as gaps in adherence. Third, set a public reporting target — typically a first annual report two to three years out — and use the target to focus attention on the substantive program improvements that will be defensible when the report is published.
The Asilomar AI Principles include the commitment that AI’s benefits should be widely shared and that decisions affecting the public should be transparent; see https://futureoflife.org/open-letter/ai-principles/. The Montreal Declaration for Responsible AI similarly calls for transparency in AI development; see https://montrealdeclaration-responsibleai.com/. The EU HLEG Trustworthy AI requirements include accountability as a core requirement, with measurement as the operational expression; see https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.
Closing the Module
Module 1.11 began with foundations (Article 1), built through the substantive ethics topics — fairness, bias, explainability, oversight, transparency, governance, stakeholders, high-stakes domains, privacy, workforce, generative AI, and cultural difference — and closes with the operational disciplines of process (Article 14) and measurement (this article).
A practitioner who completes the module should be able to: articulate the principles of responsible AI in language a business audience can engage with; design and operate an ethics review process; lead the substantive work of fairness analysis, explainability design, and human oversight specification; engage affected stakeholders meaningfully; navigate the regulatory landscape across major jurisdictions; and demonstrate the program’s effectiveness through credible measurement and reporting.
The work is open-ended. The principles converge but the operational practice continues to develop, the technology continues to change, and the social context in which AI operates continues to evolve. A mature ethics function is not a finished thing but a continuing practice — and the practitioners who carry it forward are the audience this module was written for.
© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.