Part 3: “Treat model choice as a strategic decision. Closed frontier models where you need the edge, open local models where control matters the most.”

Watch part 3 of our 3-part series ⬇️

‍

Episode summary

‍

1. Governance must scale with AI maturity

Liz challenges Jake on a practical issue: many mid-sized New Zealand organisations do not have dedicated AI risk teams. They may only have one or two overloaded IT staff, so asking them to write and maintain dozens of AI use-case rules may be unrealistic.

Jake’s response is that governance should scale with the organisation’s AI maturity.

Organisations do not need Level 5 governance when they are still at Level 1. Governance should grow as AI becomes more embedded and autonomous.

He maps governance to the AI maturity ladder:

Level 1: Personal productivity: Basic acceptable use, personal accountability, simple guidance

Levels 2–3: Team workflow and organisational infrastructure: Team and organisational controls, shared processes, controlled blast radius

Levels 4–5: Self-learning and self-driving organisation: Governance becomes part of the AI system itself through harnesses, evals and automated controls

The key message is that governance should not be over-engineered too early. It should match the level of AI risk, organisational reach, and autonomy.

2. At higher maturity, governance becomes part of the system

Jake makes an important point: at Levels 4 and 5, governance should stop being just a document. It should become part of the working AI system.

That means the controls that guide AI performance should also support safety and governance.

In Jake’s words, the same controls that help AI work well are often the controls that help make it safe.

This means organisations can get dual value from the same investment:

• Better AI quality
• Stronger governance
• Better auditability
• Clearer accountability
• Reduced operational risk

3. “Show me the evals”

Liz asks what her first governance question should be when a vendor claims their agent is “fully autonomous”.

Jake’s answer is simple:

> Show me the evals.

By evals, he means evaluation criteria: the tests and acceptance criteria written before the AI starts work.

Good evals define:

• What “done” looks like
• What must be true for the output to be acceptable
• How quality will be checked
• What criteria the AI must satisfy
• What rules or constraints apply

Jake argues that if a vendor cannot show clear evals, then “autonomous” may really mean “unsupervised”. Those are not the same thing.

His standard is direct: if an organisation cannot write down what a successful output looks like, it is not ready to hand that task to an AI agent.

Liz connects this to ITIL-style change enablement, where acceptance criteria, risk assessment and success measures are defined before implementation.

4. AI sovereignty has become a governance issue

Liz then raises a new challenge: AI sovereignty.

She refers to a recent situation where the US government placed export controls on a frontier AI model. According to the discussion, access to that model was restricted for foreign nationals, and because the AI lab could not verify nationality in real time, the model was switched off globally for nearly three weeks for many users outside approved organisations, including users in New Zealand.

Jake says this incident shows that AI dependency risk is no longer theoretical. If a business process depends on a model controlled overseas, that process can stop because of a foreign policy decision, not just because of a cyber attack or technical outage.

This brings sovereignty into the AI governance conversation.

The key question organisations should ask is:

> What is our fallback if access, pricing or policy changes overnight?

5. Open models as a realistic fallback

Liz asks whether the answer is for New Zealand businesses to build their own models.

Jake says not necessarily. Instead, organisations now have more choice because two things are changing:

1. Open models are catching up

  Openly available AI models are becoming more capable and are narrowing the gap with the largest closed frontier models.

2. Open models are becoming cheaper and easier to run 

  These models can increasingly be deployed on infrastructure an organisation controls.

Jake says he would not previously have considered open models viable for serious commercial use, but his view has changed. He now sees them becoming a real option within 12 months or sooner, at least as a fallback for US frontier models.

6. Healthcare as an early use case

Jake says this is already happening in New Zealand. He is working with a healthcare client to implement a locally run model on infrastructure the client controls.

Healthcare is a natural early adopter because it involves:

• Highly sensitive personal health data
• Strict rules about where data can go
• Critical services that cannot simply stop for weeks
• Strong operational requirements around sovereignty and control

For healthcare, Jake says sovereignty is not just a philosophical concern. It is an operational requirement.

7. Portfolio thinking for AI models

Jake does not recommend abandoning closed frontier models. Instead, he recommends **portfolio thinking**.

Organisations should use different model types for different purposes:

• Closed frontier models: Tasks needing the strongest reasoning, intelligence and capability
• Open/local models: Sensitive, secure or sovereignty-critical work
• Tested fallback models: Continuity if access, pricing or policy changes suddenly

Jake frames model choice as a Level 3 AI maturity decision because it becomes part of organisational infrastructure.

His key point is that model choice is no longer just a technical preference. It is now a strategic decision.

8. What executives should do this quarter

Liz asks what a chief executive should actually do in the next quarter.

Jake gives three practical recommendations.

• Place yourself honestly on the AI maturity ladder

Most organisations are probably at Level 1, and that is fine. The risk is not being at Level 1. The risk is not knowing where you are.

Organisations should identify their current level and deliberately choose the next step, usually moving from Level 1 to Level 2.

• Match governance to your maturity level before climbing

Organisations should use their existing IT and data governance structures, but write specific rules for specific use cases.

They should also keep a named human accountable for outcomes.

As more work is automated, checks and safeguards should be built into the system from the start, not added afterwards. The tests that help the AI perform well are also the controls that help keep it safe.

• Treat model choice as strategic

Organisations should decide deliberately when to use closed frontier models, when to use open local models, and what fallback options they have.

Jake emphasises that any fallback must be tested, because recent events show that access can be switched off from the other side of the world.

9. Why Jake remains optimistic

Liz asks Jake why he remains optimistic, given that he has seen his own AI work made obsolete multiple times in recent years.

Jake shares an example from his first week at Theta. Developers were already using coding agents such as GitHub Copilot and Claude Code, but project documents such as requirements and design documents were still being written by hand.

Jake used the same coding agents to generate those documents, producing first drafts in hours rather than weeks.

For him, the lesson was not that the technology was new. It was that the technology was already available, but people had not yet imagined using it that way.

This is why he remains optimistic. He does not believe AI will replace everything people think and do, but he believes organisations can use it to do more than they currently imagine.

He says the gap between Level 1 and Level 3 is smaller than it looks for organisations that start deliberately.

10. The episode’s one-sentence takeaway

Jake summarises the episode in one sentence:

> Know your level, match your governance, and remember that tokenmaxxing is a result, not a goal.

Liz’s key security and governance takeaway is similar: as organisations automate more work, they need to build checks and safeguards into the system at the same time. The tests that help AI do a good job are also the tests that help keep it safe.

11. Key takeaways

• AI governance should match the organisation’s maturity level.
• Level 1 organisations need simple, practical guidance, not complex enterprise frameworks.
• As AI becomes more autonomous, governance must become embedded in the system itself.
• Vendors claiming autonomy should be asked to show their evals.
• If success criteria cannot be written down, the task should not be handed to an agent.
• AI sovereignty is now a practical business continuity and governance concern.
• Closed frontier models remain useful, but open/local models are becoming important for sensitive work and fallback planning.
• Model choice is now a strategic decision, not just a technical preference.
• Organisations should know their maturity level, move up deliberately, and build governance into AI systems from the start.

‍

Chapters:

00:00 Trusting AI: The Accountability Dilemma

00:47 Matching governance to organisational maturity levels

02:29 Portfolio thinking: Closed frontier models vs open models

06:58 Final thoughts for executives: actions to take this quarter