This pillar covers how initiatives are piloted, evaluated, scaled, monitored, and retired across the four application domains. It addresses two problems: promising pilots that never scale, and deployed systems that go unmonitored and degrade. AI is an operational capability with a lifecycle, not a project with an end date.
It is the most concrete pillar: pilots with success criteria set in advance, scale-or-stop gates, production monitoring for performance, drift, equity, and incidents, and feedback that returns lessons to the Compass. The six building blocks below organize the work. The biggest risk is rarely the technology. It's underestimating the human and operational work of making AI useful and trustworthy at scale.
Why it matters
Strategy that never reaches sustained operations is just intention; this is where value is realized or lost. Monitoring and feedback also close the loop: they catch a system going wrong before it harms someone, and make each initiative faster, safer, and cheaper than the last.
Connects to GRC (security & privacy by design) · Readiness
Monitoring, maintenance & optimization
Performance, accuracy, bias, drift, incidents, and cost in live systems.
Monitoring dashboards & routine AI health checks
Incident-response playbooks for AI failures
Regular model/prompt reviews; cost tracking
Connects to Compass Track & Reflect · ongoing GRC audits
Scaling, sustainability & improvement
Growing what works responsibly, and planning for long-term viability.
Clear scaling criteria & roadmaps
Decommissioning protocols & a lessons-learned repository
Sustainability reviews, energy, cost, equity
Connects to Maturity & Readiness · future Compass evaluations
Operational governance & accountability
Day-to-day ownership, service expectations, and escalation paths.
An AI service catalog of what's approved & supported
Operational SLAs & clear central/unit handoffs
Integration with the central AI governance body
Connects to GRC · Roles & Responsibilities
AI impact & evidence
The Strategic Compass judges an initiative before it starts. Impact is the after-the-fact counterpart: the documented, mission-aligned effect on individuals, communities, institutional performance, and the public good. It's how a campus shows an initiative worked, responsibly, and for whom.
Select with the Strategic Compass (before), prove with impact evidence (after). The two are a pair.
Four dimensions of impact
Outcome impact
What measurably changed?
Changes in completion, quality, access, speed, safety, or productivity, measured against a baseline, with comparison groups where feasible.
Equity impact
For whom, and who was left out?
How benefits and harms fall across populations, especially historically marginalized groups. Report disaggregated outcomes, not just totals.
Governance impact
Was trust strengthened or weakened?
Whether transparency, accountability, oversight, and compliance improved or degraded. Efficiency gained through opacity or bias fails the test.
Societal impact
What changed beyond the institution?
Contribution to the communities served: workforce, healthcare, climate resilience, public service.
Three kinds of evidence, shown together
A credible evidence package combines three categories; any one alone is incomplete.
Innovation evidence
It solves a meaningful problem in a new or better way: the baseline, what AI made possible, and why AI was the right approach.
Impact evidence
Outcomes changed and it matters: before-and-after measures, broken out by population, plus qualitative evidence where effects are experiential.
Governance evidence
It was reviewed, documented, monitored, and adjusted: risk and privacy assessments, oversight plans, incident logs, user notices, approvals.
Innovation without impact is theater; impact without governance is risk; and governance without either is paperwork.
The one-page AI Use Case Evidence Profile
One comparable record per AI initiative: light enough to adopt, structured enough for inventories, governance reviews, and annual reports. Impact data fills it over time.
How you can tell
When it's working
Pilots have explicit success criteria and a fixed evaluation window.
Scale-or-stop gates resolve pilots instead of letting them linger.
AI is resourced as ongoing operations (multi-year funding, named staff, compute), not one-time pilot money.
Change management is explicit: role-based training, communication, and a champions network.
AI is integrated into existing systems and workflows (SIS, LMS, CRM, ERP) with human-in-the-loop by design.
Production systems are monitored for performance, drift, equity, and incidents.
Every production AI service has a named owner, an SLA, an escalation path, and a service-catalog entry.
Lessons feed back into the Compass and governance.
When it's missing
Pilots multiply but none scale or are formally stopped.
Success is declared at launch and never measured.
Pilots are funded once, then starve in production with no owner for cost or staffing.
The tool ships but nothing else changes: no training or adoption plan, so it goes unused or worked around.
AI is bolted on beside existing systems, duplicating data and breaking workflows.
Systems run unmonitored; drift and bias surface only when someone complains.
Each initiative starts from scratch because nothing was captured from the last.
Each practice is something you can build, assign, and evidence, with a first move, an owner, and what proof looks like. Move one practice up one level at a time.
The five levels: 1 Initial · 2 Emerging · 3 Defined · 4 Managed · 5 Optimizing. Each practice describes levels 1, 3, and 5. Level 2, Emerging: some level-3 behavior exists in pockets, but informally. Level 4, Managed: level-3 behavior is applied consistently, measured, and reviewed on a set cadence.
1
Initiative pipeline & portfolio management
AI initiatives move from idea to deployment through a managed, transparent pipeline.
4ManagedThe practice is applied consistently, measured, and reviewed on a defined cadence.
5OptimizingThe cycle visibly changes course (retired policies, rebalanced budgets) and the board sees the evidence.
Next moves
Put the annual cycle on the governance calendar: reassess maturity, review policy adequacy, reflect on incidents.
Include enablement adequacy, equity resourcing, and AI financial sustainability (licenses, tokens, TCO) in the review.
Deliver an annual AI governance report to cabinet and board.
Who owns it
AI governance body chair
Evidence it exists
Governance calendar
Annual report
Documented course corrections
In practice
Worked example
Closing the loop
An institution runs a dozen AI pilots. A year later all are still pilots, and leadership can't say which worked. Under this pillar each gets success criteria and a 90-day decision gate: two scale, three are stopped, the rest iterate. Scaled systems get drift and equity monitoring, and a quarterly review feeds lessons into the Compass, so the next round is screened against what the institution learned, not what it hoped.
Watch for
Endless pilots with no decision gate.
Treating implementation as a technology rollout and underestimating the change-management work, the most common failure.
Declaring victory at deployment and never checking outcomes.
No operational owner or SLA, so no one is accountable when a live service breaks.
No monitoring, so a drifting or biased system runs until it causes harm.
Readiness & maturity questions
Readiness asks: are we prepared? Use these to surface blind spots before you build. Every “no” shows where the work is. Representative prompts for reflection, not a scored test. For the scored version, use the Maturity Assessment.
Organizational Change & Adoption
Is there a change-management strategy for AI?
Are staff and faculty trained on adopting AI?
Do we track readiness to change?
Is communication transparent and anticipatory?
Are user concerns proactively addressed?
Are early adopters supported as champions?
Are resistance patterns analyzed?
Are adoption metrics defined?
Is there alignment between tech and behavior change?
Are staff consulted pre- and post-implementation?
Resourcing & Infrastructure
Are budgets allocated for AI planning and deployment?
Is there capacity for infrastructure scale?
Are staffing plans AI-inclusive?
Is there funding flexibility for rapid AI evolution?
Are existing systems being reviewed for AI compatibility?
Are cloud/data contracts AI-aware?
Are procurement teams familiar with AI-specific needs?
Are support teams equipped to manage AI operations?
Is equity considered in resource allocation?
Are long-term maintenance needs planned?
Systems & Processes
Have business processes been mapped for AI insertion?
Are systems documented and interoperable?
Are data pipelines reliable and secured?
Is automation already part of the operational mindset?