AITM-ECI: AI Experimentation Associate — Body of Knowledge Artifact Template

How to use this template

This template is the companion artifact to Article 14. An experiment brief is authored before the experiment starts, reviewed before execution, and committed to the experiment tracking system as the pre-registration record. The brief is typically one to two pages when rendered in a normal document style; the sections below map directly onto the eight-section structure in Article 14.

Every field is required. Empty fields are rejected by the review gate. Fields for which the answer is “not applicable” are filled with “not applicable” and a one-sentence rationale.

Copy the template, rename it with the experiment’s name, and fill in each section. Leave the frontmatter intact so that the brief is recognized by the documentation extractor.

Experiment Brief — [Experiment Name]

1. Identification

2. Hypothesis

Using the four-element structure from Article 2: subject, predicted effect, measurement, threshold.

• Subject. [What is the change? Which model, prompt, retrieval policy, feature, threshold, configuration, or combination is the object of the experiment?]
• Predicted effect. [What observable quantity does the change alter, and in which direction?]
• Measurement. [How will the effect be observed? On what population? Over what window? By what instrument?]
• Threshold. [What size of effect, detected with what confidence, triggers what decision?]

One sentence stating the hypothesis in falsifiable form:

[Hypothesis sentence, e.g., “Changing the retrieval reranker from cross-encoder A to cross-encoder B will increase answer-acceptance rate on English-language sessions in the last 14 days of the experiment window by at least 1.5 percentage points at p < 0.01 two-sided.”]

3. Experiment mode

Using the vocabulary from Article 1. Select all that apply and specify sequence if multiple.

• Offline evaluation
• Shadow deployment
• Online A/B test
• Online multi-armed bandit
• Canary deployment with guardrail-only evaluation
• Adversarial / red-team experiment
• Other (specify)

Sequence and gating (if multi-mode): [e.g., “Offline first; if offline metrics meet threshold, shadow for 14 days; if shadow divergence within tolerance, canary at 1% then ramp.”]

Rationale for the mode choice in one paragraph:

[Why these modes and not others; what gap each mode closes that the previous mode could not.]

4. Metrics

Primary metric

Secondary metrics (short list)

Guardrail metrics (short list)

Adversarial review of the primary metric

One paragraph per Article 2: how might the primary metric move without the feature actually being better?

[Adversarial-review paragraph; 3–6 sentences.]

5. Evaluation protocol

Population and scope

• Included population: [description; e.g., “English-language sessions from organizations on paid plans”]
• Excluded population: [description with rationale]
• Randomization unit: [user / session / account / request / other]
• Stratification (if any): [strata and rationale]

Sample size and duration

Slice evaluation

Slices to be reported separately (from Article 3):

• [Slice 1, e.g., geography]
• [Slice 2, e.g., segment]
• [Slice 3, e.g., temporal cohort]
• [Slice 4]

Minimum per-slice sample size to render the per-slice metric reliable: [number]

Shadow or canary prerequisites (if mode includes them)

• [Description of shadow duration and comparison protocol]
• [Description of canary ramp steps and minimum time per step]

6. Budget

From Article 12. All three categories are required.

Compute cost

Human-review cost

Triggers

• Cost trigger (escalates for additional approval): [amount; e.g., “if aggregate spend exceeds 1.3× estimate”]
• Stopping cost (halts experiment regardless of results): [amount; e.g., “if aggregate spend exceeds 2× estimate”]

7. Decision rule and stopping rule

Ship rule

The experiment supports a ship decision if and only if:

1. [Primary metric condition, e.g., “primary metric improves by at least 1.5 percentage points at p < 0.01”]
2. [Guardrail condition, e.g., “no guardrail metric breached at any point in the experiment window”]
3. [Slice condition, e.g., “no per-slice metric degrades by more than 3% relative to baseline”]
4. [Any other conditions]

Rollback rule

Beyond the guardrail triggers in §4, the experiment triggers full rollback if:

1. [Condition; e.g., “primary metric degrades by more than 0.5 percentage points at p < 0.01”]
2. [Condition; e.g., “safety-signal rate exceeds X per thousand requests for more than 1 hour”]
3. [Condition]

Extension rule

The experiment is extended past planned duration if:

1. [Condition; e.g., “result is inconclusive (neither ship nor rollback) and duration extension remains within stopping-cost budget”]
2. [Condition]

Peeking policy

• No peeking allowed before planned end.
• Interim peeks allowed under the sequential-testing framework named in §5; peek results must be recorded in the tracking system.
• Other (specify): […]

8. Reproducibility and governance

Tracking

Regulatory mapping

Which regulatory anchors does this experiment produce evidence for? (From Article 13.)

• EU AI Act Annex IV §§2, 3, 4, 5
• EU AI Act Article 12 record-keeping
• ISO/IEC 42001 Clause 9.1 monitoring records
• NIST AI RMF MEASURE subcategories (list which)
• Other (specify)
• None (non-regulated system)

Retention

Review and sign-off

Execution is not authorized until all required rows show “Approved” and the brief is committed to the tracking system. Modifications after authorization require re-approval; the modification, the rationale, and the re-approval are recorded in the tracking system with the original brief.

Notes on use

When to use this template. Every experiment on a deployed or soon-to-be-deployed AI feature, regardless of risk tier. High-risk systems must use it; low-risk systems benefit from its discipline.

When a lighter template is acceptable. A very small offline experiment (a single training run for a personal hypothesis, run in under a day, with no production implications) can use a shortened version that omits §§5–8 and fits on half a page. The full template returns as soon as the experiment has any production pathway.

Common errors in first-time use. Vague predicted effect (“will improve X”); missing guardrails; no decision rule; no stopping cost; no regulatory mapping for regulated systems. A peer reviewer catches these; a governance reviewer treats them as blocking.

What follows. The companion artifact is the experiment report, authored at conclusion (Article 14 §2). Every brief has a report; every report references the brief.

© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.