Part 2: “The problem is there is no shared AI mental picture, a map.”

Watch part 2 of our 3-part series ⬇️

‍

‍

Episode summary

‍

1. “Can you trust AI?”

Jake opens with a direct question he received from an executive: how do you trust AI?

His answer is blunt: you can’t fully trust AI.

Even if agentic AI can act autonomously, plan work, and review its own outputs, humans still need to validate the final result because accountability remains with people. AI may support the work, but it does not carry organisational or legal responsibility.

An executive challenged this by pointing out that organisations do not have the resources to “babysit” AI at every step. Jake’s answer was that AI should be used to validate and improve its own work before a human reviews the outcome.

2. AI loops mirror how humans already work

Jake explains that expecting AI to get everything right first time is unrealistic, because humans do not work that way either.

In normal software delivery:

• Product owners and developers refine requirements together.
• Developers and testers work through issues iteratively.
• Front-end and back-end teams align on data and APIs through repeated discussion.
• Each cycle improves understanding and quality.

Jake’s point is that work already happens through loops of checking, correcting, and improving. AI makes those loops more visible and can run them much faster.

Instead of a human checking every intermediate AI output, AI can run several self-checking and improvement cycles first, then present a more mature result for human review.

3. “Who checks the checker?”

Liz raises the key cyber security concern: if AI checks AI, who checks the checker? Is that just AI marking its own homework?

Jake gives two responses.

First, AI self-checking should replace human review of intermediate steps, not the final review. A human still validates the final outcome. He compares this to validating the destination rather than every turn of the wheel.

Second, the AI is not checking against its own opinion. Humans define the desired outcome and rules in advance. The AI checks its work against those human-defined standards. These predefined standards are called evaluation criteria, or evals.

Jake also notes that organisations can use a different model to review the output, reducing the risk of the same AI system simply reinforcing its own mistakes.

4. What tokenmaxxing means

Liz then returns to the earlier term tokenmaxxing.

Jake explains that a token is the unit AI models process. It is roughly equivalent to a word or part of a word, and users typically pay according to token usage. In practical terms, token use represents how much AI “work” or “thinking” is being purchased.

Tokenmaxxing emerged as a status game where engineers or teams competed to burn large numbers of tokens. Jake refers to examples involving leaderboards at major technology companies and employees allegedly running agents on meaningless tasks to keep token-use numbers high.

By May, Fortune had declared that tokenmaxxing was dead because companies were not seeing the expected return.

5. Tokenmaxxing is half wrong, half important

Jake agrees with the backlash in part. He says tokenmaxxing is wrong when token use becomes the goal.

However, he argues that high token use can be valuable when it is the result of well-engineered AI loops. The purpose of using more tokens should be to reduce constant human supervision by allowing the AI to:

• Draft
• Check against a clear checklist
• Fix mistakes
• Iterate
• Present a better final result to a human

This matters because small errors compound across long chains of work. Jake gives the example of a 1,000-step process where each step is 99.9% reliable. Even with that high per-step reliability, the end-to-end success rate drops significantly. His point is that self-correction after each step is not optional in complex AI workflows; it is essential.

6. Where tokenmaxxers went wrong

Jake says the mistake was turning token count into the target rather than treating it as a by-product of good engineering.

He references Goodhart’s Law: when a measure becomes a target, it stops being a good measure.

High token usage is useful when it is supported by:

• Clear specifications
• Defined evaluation criteria
• Guardrails
• Harnesses
• Structured loops
• Clear limits

Without those, heavy token use simply becomes inefficient automation with a large invoice.

His bottom line is:

Tokenmaxxing should be a result, not a goal.

A large token bill is acceptable if it reflects structured, engineered work that produces better outcomes with less human babysitting. It is wasteful if it comes from leaderboards, vague tasks, or open-ended agent activity.

7. What happens to people?

Liz asks what happens to humans when AI writes code and loops run overnight.

Jake says the human role changes rather than disappears. For developers, the role shifts from coder to orchestrator.

People will spend less time manually typing code and more time:

• Setting goals
• Designing systems
• Writing acceptance criteria
• Defining evals
• Directing agents
• Reviewing final outcomes

Jake also identifies emerging roles, including:

• Deployed engineers: people embedded in customer environments who deliver AI solutions quickly.
• Agent managers: people who direct and govern fleets of AI agents.

His message to executives is that organisations do not necessarily need to replace people. But people who learn to direct agents will move ahead of those who try to compete with agents.

8. A five-level AI maturity map

Jake then introduces a five-level model to help organisations understand where they are and where to go next.

Level 1: Personal productivity

Individuals use tools such as Copilot, Claude, or ChatGPT with their own context and habits.

This creates individual productivity gains, but it does not transform the organisation. If those individuals leave, the knowledge and AI capability may leave with them.

Level 2: Team workflow

AI is embedded into shared team processes.

The team uses shared context, shared tooling, and defined workflows rather than relying on individual habits.

Level 3: Organisational infrastructure

AI is connected to the organisation’s data, systems, processes, and context.

At this stage, AI becomes part of the organisation’s operating infrastructure, similar to a network or core business system.

Level 4: Self-learning organisation

The organisation uses harness engineering at enterprise level.

Rules, feedback loops, and guardrails are built around business processes, allowing the organisation to improve continuously inside controlled boundaries.

Level 5: Self-driving organisation

This is the loop-engineering level.

Agents run whole processes through automated loops, while humans set direction and validate outcomes.

Jake is clear that most organisations are not close to this point yet.

9. Where most organisations are now

Jake says most companies in New Zealand and elsewhere are currently at Level 1. Some are working towards Level 2, particularly through workflow automation.

He says frontier AI companies such as OpenAI and Anthropic are likely around Level 3, while nobody is genuinely at Level 5.

Importantly, he emphasises that Level 1 is not a failure. It is a starting point. The real risk is not knowing that the maturity ladder exists.

10. Moving from Level 1 to Level 2

Jake gives a practical example of the Level 1 to Level 2 jump.

At Level 1, analysts may each use Copilot individually.

At Level 2, a monthly report could be produced through a shared AI-enabled workflow:

An agent drafts the report using approved data.

The agent checks the draft against quality evals.

A named person reviews and signs it off.

The tools may be similar, but the process is now designed, repeatable, shared, and governed.

11. Governance and security concerns

Liz raises the major security issue: every level of AI maturity gives AI more autonomy and access to more of the organisation. A Level 5 self-driving organisation could create serious visibility, control, and incident risks.

Jake agrees and says many organisations’ AI governance is stuck at the level of broad principles, such as “do this” or “don’t do that”. These principles are often too high-level to guide real decisions.

He identifies three reasons governance fails:

• Leaders do not understand AI deeply enough to write specific rules.
• AI changes so quickly that specific rules are hard to create and maintain.
• Enforcement breaks because broad rules push too many judgement calls back onto individuals.

In short, governance fails when every decision depends on individuals who cannot keep up with the volume and pace of change.

12. How to govern AI in practice

Jake recommends that AI governance should not become a separate empire. It should sit inside existing IT and data governance structures.

That means using the same:

• Accountable owners
• Governance forums
• Escalation paths
• Risk management mechanisms

He also states one rule that should always apply:

A named human is accountable for validating AI output.

Within that existing governance structure, Jake says rules should be written per tool and per use case.

For example, an organisation should have different rules for:

• Coding agents
• Customer-facing chatbots
• Finance workflows
• Internal productivity tools

This prevents both extremes: uncontrolled AI sprawl on one side, and vague “poster on the wall” governance on the other.

13. Key takeaway

The central message of this episode section is that organisations should not blindly trust AI, but they also cannot afford to manually supervise every step. The practical answer is to design AI systems with clear specifications, evals, guardrails, loops, and human accountability at the final decision point.

High AI usage, including high token usage, is not automatically bad. It is valuable when it reflects structured, governed, well-engineered loops that improve quality and reduce unnecessary human babysitting. But without clear goals and governance, it becomes waste.

For organisations, the next step is to move deliberately from individual AI productivity towards team-level workflows, while embedding governance into existing IT and data structures.

‍

Chapters

00:00 Trusting AI: The Accountability Dilemma

02:37 Tokenmaxxing: The Status Game and Its Implications

05:01 The Evolution of Developer Roles in AI

07:28 Navigating AI Maturity Levels

10:39 Governance in AI: Building a Framework

13:04 Sovereignty and AI: The New Challenges

14:14 Strategic Model Choices in AI

16:10 The Future of AI: Balancing Autonomy and Control

19:20 Final Thoughts: Embracing AI with Caution