Architect in Evaluate and Learn Stages for Agentic Systems

Many architecture organisations drop the ball here. Calibrate and Organize attract architect attention because they are upstream and consequential; Evaluate and Learn feel downstream and routine. That is backwards. Evaluate and Learn is where architectural learning compounds — where the platform gets better because every agent’s post-mortem updates the reference architecture.

Evaluate — what the architect contributes

Evaluate is the post-launch review stage, typically running on a monthly or quarterly cadence per agent. The architect has four contributions.

Evaluate contribution 1 — Production metrics review

The architect reviews the agent’s production metrics against the commitments made at Organize:

• Goal-achievement rate — is it within target? Trending?
• HITL intervention rate — in line with expectation? Spiking (indicating a design problem) or falling (possibly indicating complacency)?
• Safety classifier pass/fail rates — are classifiers detecting what they should? Are false-positive rates acceptable?
• Latency SLOs — p50 / p95 / p99 within bounds? Trending?
• Cost per interaction — within target? Outliers investigated?
• Error rates by tool — any tool disproportionately failing?
• Memory-write volume and provenance — healthy? Unprovenanced writes trending to zero?
• Policy-engine denials — are deny rates reflecting appropriate tightness or excess friction?
• User feedback / satisfaction — tracking? Comments patterns?
• Article 50 disclosure coverage — is disclosure happening on every applicable output?

Findings turn into action items the architect tracks across quarters.

Evaluate contribution 2 — Incident post-mortems review

Every incident produces a post-mortem (Article 25). The architect reads each post-mortem through an architectural lens:

• Was this a design gap or a situational failure?
• Is the mitigation scoped only to this agent, or should it be a platform-level change?
• Does the incident reveal a pattern we should catalog (so the next team avoids it)?
• Are the action items appropriately owned?

The architect aggregates incidents quarterly. Patterns (e.g., three incidents in the past quarter relating to retrieved-content injection) trigger platform-level work.

Evaluate contribution 3 — Evaluation-harness refresh

The evaluation harness (Article 17) ages. New attack patterns emerge; new regressions appear; user behaviour shifts. The architect schedules quarterly evaluation-harness refreshes:

• New golden tasks added (for use cases now visible in production traffic).
• Obsolete golden tasks retired.
• Adversarial battery updated with newly-public attack patterns.
• Calibration scored on fresh traffic.
• Fairness metrics re-computed on fresh data.

Evaluate contribution 4 — Autonomy-expansion review

The most consequential Evaluate conversation is “should this agent do more?” Common expansion proposals:

• Raise the HITL threshold (e.g., from $500 to $2000).
• Add new tools (e.g., email-send tool where currently draft-only).
• Expand user base (e.g., from internal to external customers).
• Expand language coverage.
• Expand to new geographies with different regulatory envelopes.

The architect brings the decision back to the autonomy framework (Article 2) and to evidence:

• What does current performance tell us? Goal-achievement and HITL-override rates at the current scope.
• What is the incremental blast radius? At the new scope, what is worst-case harm?
• What controls would need to change? New HITL thresholds, new runbooks, new evaluation coverage.
• What is the rollback plan if expansion shows harm? Specific metrics, specific thresholds.

Evaluate gate-review conversation

Evaluate gate reviews are typically quarterly per agent. Agenda:

1. Production metrics readout.
2. Incident review and post-mortem themes.
3. Evaluation-harness refresh findings.
4. Autonomy-expansion or contraction proposals (if any).
5. Platform-level findings (what should propagate to other agents).
6. Action items and owners.

Learn — what the architect contributes

Learn is about compounding knowledge across the portfolio. The architect has three contributions.

Learn contribution 1 — Platform evolution

The architect translates Evaluate findings into platform roadmap items:

• A new safety classifier that would have caught three incidents.
• A new observability signal multiple teams requested.
• A new policy-engine feature that would eliminate a class of authorization gaps.
• A new registry field that captures lineage information currently missing.

Platform evolution proposals are sized, prioritised, and funded against the COE’s platform budget.

Learn contribution 2 — Pattern library updates

The architect curates the pattern library with new patterns and anti-patterns learned:

• New anti-patterns from incidents (“goal-hijack via JSON-formatted user input — here is the mitigation”).
• New patterns for autonomy expansion (“three-step expansion with gradual HITL-threshold change — here is the template”).
• New sector-specific patterns as new domains join the portfolio.

Pattern-library updates are broadcast across product teams in a regular rhythm (monthly architect-guild meeting, published bulletin).

Learn contribution 3 — Retirement planning

Agents eventually retire. Causes:

• Use case no longer valuable.
• Better alternative available.
• Unsustainable operating cost.
• Persistent quality problems.
• Regulatory changes that make the use case impractical.

The architect produces retirement plans:

• Sunset schedule with notice to users.
• Migration path where applicable (hand to alternative agent, hand to human workflow).
• Archive plan for audit trails (Article 28 retention applies).
• Lessons-learned post-mortem for the retirement decision (what led here, what to watch in the next agent).

Timeline of a single agent’s lifecycle