Building a Harmonized Compliance Evidence Portfolio

COMPEL Certification Body of Knowledge — Module 3.4: Enterprise Governance Architecture Article 23 of 23

Compliance is ultimately demonstrated through evidence. A governance program without evidence is an aspiration; a governance program with comprehensive, current, and well-organized evidence is a demonstrable capability. For organizations managing multi-framework AI governance compliance, the evidence portfolio is the most operationally critical component — it is what auditors review, what regulators inspect, what customers examine during due diligence, and what boards use to assess governance effectiveness.

This article provides a practical guide to building a harmonized evidence portfolio: one collection of governance artifacts that serves all applicable frameworks simultaneously. It covers evidence types, the evidence lifecycle, quality assurance for compliance evidence, automation strategies, and maintaining evidence currency as frameworks and AI systems evolve.

Evidence Types That Serve Multiple Frameworks

The Evidence Taxonomy

Governance evidence falls into six categories, each serving different compliance functions:

1. Policy Documents: Organizational policies that define governance commitments, principles, and boundaries. Examples include the AI policy, data governance policy, risk management policy, acceptable use policy, and incident response policy. Policy documents serve virtually every framework — the EU AI Act requires documented quality management systems (Article 17), ISO 42001 requires an AI policy (Clause 5.2), NIST requires that trustworthy AI characteristics be integrated into policies (GOVERN 1.2), and every other framework has comparable policy expectations.

The harmonization insight: write one AI policy that addresses all applicable framework requirements. Structure it so that each framework’s specific policy requirements are covered, even if they use different terminology. The EU AI Act’s “quality management system” documentation and ISO 42001’s “AI policy” can be satisfied by the same document if it is comprehensive enough.

2. Assessment Records: Documentation of evaluations conducted — risk assessments, impact assessments, fairness assessments, security assessments, data quality assessments, and vendor assessments. These are the evidentiary backbone of multi-framework compliance because every framework requires some form of assessment activity.

A single risk assessment, structured to cover the dimensions required by all applicable frameworks, generates evidence for EU AI Act Article 9, NIST MAP and MEASURE functions, ISO 42001 Clause 6.1.2, and every other framework’s risk assessment requirement. The key is structuring the assessment to address all required dimensions — technical risk, ethical risk, rights impact, societal impact, environmental impact — rather than limiting it to only the dimensions of one framework.

3. Process Documentation: Records of how governance processes work — procedures, workflows, decision trees, escalation paths, and operational playbooks. Process documentation demonstrates that governance is systematic rather than ad hoc. ISO 42001 places particular emphasis on documented processes (Clause 7.5), but all frameworks benefit from process evidence because it shows organizational capability, not just individual compliance.

4. Activity Records: Logs of governance activities performed — meeting minutes, review records, approval decisions, training attendance, communication logs, and stakeholder engagement records. Activity records prove that documented processes are actually executed. This is the evidence category that most frequently distinguishes genuine governance from paper compliance. An auditor reviewing ISO 42001 Clause 9.3 (Management Review) will examine not just the management review procedure but the actual meeting minutes, attendee records, and documented decisions.

5. Technical Artifacts: System-level evidence including model cards, data cards, test results, monitoring dashboards, log samples, architecture diagrams, and deployment records. Technical artifacts demonstrate that governance requirements have been implemented in AI systems, not just documented in policies. The EU AI Act’s Annex IV technical documentation requirements are the most prescriptive specification for technical artifacts, and satisfying Annex IV creates technical evidence that serves all other frameworks.

6. Improvement Records: Evidence of the governance program’s evolution — corrective action reports, audit findings and remediation, lessons learned documentation, and governance maturity assessments. Improvement records demonstrate the continual improvement commitment required by ISO 42001 Clause 10.1, NIST MANAGE 4.2, and the continuous learning embedded in the COMPEL Learn stage.

Evidence Reusability Matrix

The six evidence categories have different reusability profiles across frameworks:

In practice, approximately 75-80% of evidence items in a well-structured portfolio serve three or more frameworks without modification. An additional 15% serve multiple frameworks with minor framing adjustments. Only 5-10% are framework-specific.

Evidence Lifecycle Management

Stage 1: Evidence Planning

Before generating evidence, plan what evidence is needed, when it will be generated, and who is responsible. The evidence plan maps directly to the harmonization matrix:

For each requirement (or convergence cluster of requirements), define:

• Evidence type required
• Evidence owner (the role responsible for generation)
• Generation trigger (COMPEL stage gate, calendar interval, event-driven)
• Review frequency
• Retention period (driven by the longest applicable regulatory requirement)

The evidence plan is a governance artifact itself — ISO 42001 Clause 7.5 specifically requires planning for documented information.

Stage 2: Evidence Generation

Evidence generation should be embedded in governance activities, not treated as a separate documentation task. When the governance team conducts a risk assessment, the risk assessment report is evidence. When the AI ethics committee meets, the meeting minutes are evidence. When the monitoring system detects model drift, the alert log and response record are evidence.

The key practice is structured evidence capture: ensure that governance activities produce their outputs in a format that satisfies the evidence requirements of all applicable frameworks. This means:

• Risk assessment reports should address technical, ethical, legal, and societal dimensions (satisfying all frameworks) rather than only technical risk (satisfying only one)
• Test results should cover performance, fairness, robustness, and security (satisfying EU AI Act Article 15, NIST MEASURE 2.1-2.7, and ISO 42001 Annex A.10.5) rather than only performance
• Meeting minutes should capture attendees, decisions, action items, and rationale (satisfying ISO 42001 Clause 9.3 management review outputs) rather than just action items

Stage 3: Evidence Cataloging

Every evidence item is cataloged with metadata that enables multi-framework retrieval:

Required metadata fields:

• Evidence ID (unique identifier)
• Title and description
• Evidence type (policy, assessment, process, activity, technical, improvement)
• COMPEL stage(s)
• COMPEL domain(s)
• AI system(s) covered
• Framework requirements satisfied (with specific article/clause/subcategory references)
• Date generated
• Author/owner
• Review date (next scheduled review)
• Status (current, under review, superseded, archived)
• Version number
• Retention expiry date

Why metadata matters: An auditor asking “show me your evidence for ISO 42001 Clause 9.2” should be able to query the catalog and receive a list of internal audit reports, sorted by date, for the relevant scope. A regulator asking “demonstrate your compliance with EU AI Act Article 14 for AI system X” should receive human oversight procedures, operator training records, and override mechanism test results — all retrieved through metadata queries.

Stage 4: Evidence Review

Evidence items are reviewed at defined intervals:

• Scheduled review: Policy documents (annually), assessment records (with each COMPEL cycle), process documentation (annually or when processes change), technical artifacts (with each system update)
• Triggered review: After incidents, after regulatory changes, after significant system modifications, after audit findings
• Continuous review: Activity records and monitoring logs are reviewed as part of ongoing governance operations

Review assesses three dimensions:

1. Currency: Is the evidence still accurate and reflective of current practice?
2. Completeness: Does the evidence fully satisfy all mapped framework requirements?
3. Quality: Does the evidence meet the quality standards defined in the evidence quality framework?

Stage 5: Evidence Refresh

When review identifies that evidence is stale, incomplete, or below quality standards, trigger a refresh. Evidence refresh is not a documentation exercise — it is a governance exercise. Refreshing a risk assessment means conducting a new risk assessment, not updating the date on the old one.

Stage 6: Evidence Retirement

When evidence is superseded by refreshed versions, or when the AI system or framework it relates to is no longer applicable, evidence is retired to archive. Retirement does not mean deletion — regulatory retention requirements may mandate preservation for specific periods. The EU AI Act requires documentation retention for 10 years after the AI system is placed on the market.

Quality Assurance for Compliance Evidence

The Evidence Quality Framework

Not all evidence is equal. Poor-quality evidence creates compliance risk because auditors and regulators may conclude that the underlying governance activity was also poor. Define quality standards for each evidence type:

Accuracy: Evidence must correctly represent the governance activity it documents. Risk assessment reports must reflect actual assessment methodology and findings, not hypothetical or aspirational statements.

Completeness: Evidence must address all dimensions required by applicable frameworks. A bias assessment that tests only one protected characteristic when the framework requires testing across multiple characteristics is incomplete.

Timeliness: Evidence must be current. A risk assessment from two years ago does not demonstrate current risk management capability, even if the methodology was sound.

Traceability: Evidence must be traceable to the governance activity that produced it. Who conducted the assessment? When? What data was used? What methodology was applied? Traceability enables auditors to verify evidence integrity.

Consistency: Evidence across multiple AI systems should follow consistent formats, methodologies, and quality standards. Inconsistency suggests ad hoc rather than systematic governance.

Quality Assurance Processes

Peer review: Assessment records and technical artifacts should be reviewed by a second qualified person before being cataloged as compliance evidence.

Calibration: Ensure that different teams producing similar evidence types (e.g., risk assessments for different AI systems) are applying consistent methodologies and quality standards. Calibration sessions where teams compare approaches help identify and correct inconsistencies.

Sampling: Periodically sample evidence items and assess them against quality standards. Track quality trends over time. If evidence quality is declining, investigate root causes (team capacity? methodology gaps? tool limitations?).

Audit findings integration: When internal or external audits identify evidence quality issues, treat them as corrective action triggers. Update quality standards, training, or processes to prevent recurrence.

Automation Strategies for Evidence Collection

Automated Evidence Sources

Many evidence items can be generated automatically from existing systems:

System logs and monitoring data: AI system activity logs, performance metrics, drift detection alerts, and security events are generated automatically. Configure monitoring systems to export evidence-formatted reports at defined intervals.

CI/CD pipeline outputs: Automated testing results, code review records, deployment approval records, and release notes are produced by the development pipeline. Configure pipelines to archive these outputs as compliance evidence.

Governance workflow outputs: If governance activities are managed through workflow tools (approval workflows, review workflows, incident management workflows), the workflow system produces activity records automatically.

Training management systems: Training completion records, competency assessments, and certification status can be exported from learning management systems.

Calendar and communication systems: Meeting records, attendee lists, and communication logs can supplement governance activity evidence.

Semi-Automated Evidence Generation

Some evidence requires human judgment but benefits from automation in structure and formatting:

Template-based assessments: Risk assessments, impact assessments, and fairness evaluations use standardized templates that ensure all required dimensions are addressed. The template provides structure; the assessor provides analysis.

Pre-populated reports: Monitoring dashboards can pre-populate periodic review reports with quantitative data, leaving analysts to add interpretation and recommendations.

Evidence tagging assistance: When evidence is created, automated tagging can suggest framework requirement mappings based on evidence type and content, with human review and confirmation.

Fully Automated Evidence Workflows

At enterprise maturity, implement end-to-end automated evidence workflows:

1. Governance activity occurs (e.g., model performance evaluation in the COMPEL Evaluate stage)
2. Evidence is automatically generated (test results report)
3. Evidence is automatically cataloged with metadata (framework requirements, AI system, date)
4. Evidence quality is automatically assessed against standards (completeness check, format validation)
5. Evidence is routed for human review if quality checks flag issues
6. Evidence is indexed in the portfolio and available for framework-specific reporting

Maintaining Evidence Currency

The Currency Challenge

AI systems change. Frameworks evolve. Organizations restructure. Evidence that was current six months ago may no longer reflect reality. The currency challenge is particularly acute for multi-framework compliance because different frameworks have different currency expectations:

• EU AI Act: Technical documentation must be “kept up to date” (Article 11) — no specific frequency, but the expectation is that documentation reflects the current state of the system
• ISO 42001: Evidence must be current at the time of surveillance audits (typically annual)
• NIST: The AI RMF Playbook recommends continuous review and update

Currency Management Strategies

Event-driven refresh triggers: Define events that automatically trigger evidence refresh: system updates, model retraining, significant performance changes, incident reports, regulatory changes, organizational restructuring.

Calendar-driven refresh cycles: Establish minimum refresh frequencies for evidence types not captured by event triggers: policy documents (annual), risk assessments (semi-annual or with each COMPEL cycle), process documentation (annual), improvement records (continuous).

Staleness alerts: Implement automated alerts when evidence items approach or exceed their review dates. Dashboard visibility of evidence currency status enables proactive management.

COMPEL cycle alignment: Align evidence refresh with the COMPEL lifecycle. Each complete COMPEL cycle (Calibrate through Learn) should produce a full set of refreshed evidence for the AI systems in scope. If the organization runs quarterly COMPEL cycles, evidence is refreshed quarterly.

Currency for Legacy AI Systems

A particular challenge is maintaining evidence currency for AI systems that are in production but not undergoing active development. These systems still require monitoring evidence, performance evidence, and risk assessment evidence — but the governance team’s attention naturally focuses on newer systems. Define minimum evidence currency requirements for legacy systems and ensure they are included in governance review cycles.

Evidence Portfolio Architecture at Scale

Portfolio Organization

At enterprise scale with dozens of AI systems and multiple frameworks, the evidence portfolio must be organized for efficient access. The recommended structure uses three dimensions:

Dimension 1 — By AI System: Each AI system has a complete evidence folder containing all evidence items relevant to that system. This is the primary access path for system-level audits and due diligence.

Dimension 2 — By COMPEL Stage: Evidence is also accessible by the COMPEL stage that produced it. This supports internal governance reviews and lifecycle management.

Dimension 3 — By Framework Requirement: Evidence is tagged and retrievable by framework requirement. This is the primary access path for framework-specific audits, regulatory reporting, and certification assessments.

These are not three separate copies of evidence — they are three access paths into the same evidence repository, enabled by metadata tagging.

Portfolio Metrics

Track portfolio health through four metrics:

1. Coverage: What percentage of applicable framework requirements have current evidence? Target: 100% for mandatory frameworks.
2. Currency: What percentage of evidence items are within their review period? Target: 95% or higher.
3. Reusability: What percentage of evidence items serve multiple frameworks? Target: 75% or higher.
4. Quality: What percentage of evidence items meet quality standards on review? Target: 90% or higher.

Report these metrics to the governance committee quarterly and to the board annually.

Key Takeaways

The harmonized evidence portfolio is the operational heart of multi-framework compliance. It transforms governance activities into demonstrable compliance through structured evidence generation, rigorous lifecycle management, and multi-framework tagging. Six evidence types — policies, assessments, processes, activities, technical artifacts, and improvement records — combine to demonstrate governance capability to any audience.

Quality matters more than quantity. A smaller portfolio of high-quality, current, traceable evidence items is far more valuable than a large portfolio of stale, incomplete, or poorly organized artifacts. Quality assurance processes — peer review, calibration, sampling, and audit integration — maintain evidence integrity over time.

Automation is not optional at enterprise scale. Automated evidence generation from system logs, CI/CD pipelines, and governance workflows reduces the manual burden and ensures consistent evidence production. Semi-automated template-based assessments ensure completeness while preserving human judgment. Fully automated evidence workflows represent the maturity target for enterprise governance programs.

The evidence portfolio does not exist in isolation — it is the tangible output of the COMPEL governance lifecycle, the input to multi-framework reporting, and the foundation for audit readiness. Organizations that invest in building a well-structured, well-maintained evidence portfolio find that framework-specific compliance becomes a reporting exercise rather than a governance exercise. The governance is already done; the evidence already exists; the report simply presents it in the framework’s expected structure.