From AI Demo to Production: 5 Lessons on AI Agents, Governance and Security

My reflections from AI Engineering: From Agents and Harnesses to Production | Melbourne, October 2026

We've been hearing a lot about AI agents lately, especially how they can automate workflows, interact with different systems and complete tasks that would normally require human involvement. With the tools and frameworks available today, building an AI agent that can perform an impressive demonstration is becoming increasingly accessible.

But what happens when we want to take that same agent into a real business environment, where it needs to work with sensitive data, follow business rules, interact with existing systems and potentially make decisions that affect customers or employees?

That's where things start to get a lot more complicated.

On 7 October, I attended AI Engineering: From Agents and Harnesses to Production, held at Amazon's Melbourne office. The evening brought together people working across AI engineering to explore topics including agent governance, architecture, context management, evaluation and security.

As someone working in enterprise Data & AI, I found the discussions particularly interesting because they connected with many of the questions that come up in my day-to-day work. A big part of my role involves understanding business challenges, exploring where AI could create value and working with different stakeholders and technical teams to turn those opportunities into practical solutions.

And while there's understandably a lot of excitement around what AI can do, I've increasingly found myself thinking about what it actually takes to make these solutions work in a real organisation.

Here are a few things from the evening that stayed with me.

Building an AI agent is one thing. Deciding what it should be allowed to do is another.

One of the presentations introduced Agenova, an open-source approach to managing AI agent permissions at the task level. What I found interesting wasn't just the technology itself, but the thinking behind it: rather than giving an agent broad access to different systems, we should be much more intentional about what information and actions it needs for a particular task.

The example shared during the presentation was quite straightforward. Imagine asking an AI agent to analyse your company's quarterly sales performance. To complete that task, it might need read-only access to an approved sales dataset, but that doesn't mean it should also be able to modify sales records, access unrelated confidential information or connect to external websites.

The approach demonstrated how access could be requested for a specific piece of work, checked against organisational policies and enforced through infrastructure, with permissions ending when the work is completed or expires. It sounds like a technical security consideration, but I actually see a strong connection to business analysis and product discovery here.

When we're defining requirements for an AI solution, we naturally spend a lot of time understanding what users want the system to do. But with AI agents becoming more autonomous, I think we also need to spend more time defining what the system shouldn't be allowed to do, which decisions require human approval and what should happen when an agent encounters something outside its authorised scope.

These are questions that need input from business stakeholders, product teams, architects and security specialists, rather than being left entirely to engineering at the implementation stage. It also made me think about how we define a successful AI solution. Just because an agent can complete more actions independently doesn't necessarily mean we've designed a better product. Sometimes, having clear limitations is exactly what makes a solution more reliable and appropriate for the business.

The AI model is only part of the solution

Another topic that caught my attention was the concept of an agent harness, which refers to the systems and mechanisms surrounding an AI model that help manage how an agent actually operates.

Before diving deeper into AI agents, it's easy to think that much of their capability comes down to the underlying model. But an agent operating in a real environment needs much more than the ability to understand instructions and generate responses.

It needs to maintain the right context, interact with tools and APIs, manage multi-step workflows, handle unexpected situations and operate within appropriate boundaries. The discussions also covered concepts such as agent loops, context management, evaluations and Model Context Protocol (MCP), which supports standardised connections between AI applications and external tools or data sources.

What made this particularly relevant to me was thinking about the difference between a solution that works well during a proof of concept and one that's ready to be used by hundreds or thousands of employees.

Imagine an AI agent supporting customer service. It might be able to understand a customer's question, retrieve their order details and suggest an appropriate resolution. In a controlled demonstration, everything could work perfectly.

But what happens when the order management system becomes unavailable, the customer provides incomplete information or the agent encounters conflicting data? What if it attempts to initiate a refund that exceeds the company's approval threshold?

The language model might be capable of understanding the request, but that alone doesn't guarantee the overall solution will behave correctly. This is where the surrounding architecture, business rules, exception handling and human intervention processes become important.

From my experience working across enterprise AI opportunities, I think this is also why the discovery and requirements stages matter so much. It's relatively easy to get excited about a potential use case when we focus on the ideal scenario, but the real complexity often emerges when we start exploring the exceptions, dependencies and operational constraints.

The event was a good reminder that a capable AI model doesn't automatically translate into a reliable AI solution, and that production readiness needs to be considered across the entire system.

We need to think differently about how we measure AI performance

One of the broader themes throughout the evening was the importance of evaluating AI agents, particularly as they become capable of taking actions rather than simply generating information.

When we talk about AI performance, accuracy is often one of the first things that comes to mind. Did the model provide the correct answer? Was the response relevant? Did it complete the task? Those questions are still important, but I think AI agents introduce another layer of complexity because the final output doesn't necessarily tell us whether the process was appropriate.

For example, an AI agent could generate an accurate financial report while accessing information it wasn't authorised to use. It might complete a task successfully most of the time but occasionally fail in ways that are difficult to predict or detect. From the user's perspective, the result might look perfectly reasonable, even though something went wrong behind the scenes.

This made me reflect on how we should approach evaluation from both a technical and business perspective.

Beyond checking whether an agent produces the right answer, we need to understand whether it follows the expected process, uses the correct tools, respects permissions and handles failures appropriately. We also need to think about how consistently it performs across different scenarios, particularly when the consequences of an error could be significant.

And then there's another question that I think is sometimes overlooked: even if the AI solution performs well technically, is it actually delivering the business outcome we wanted?

If we're introducing an AI agent to improve customer service, for example, we might measure response accuracy and successful task completion, but we'd also want to understand whether it reduces resolution time, improves customer satisfaction or helps employees work more effectively without introducing unacceptable risks.

This is something I find particularly relevant in enterprise AI adoption, where there can be a long list of exciting opportunities competing for attention and investment. Ultimately, the goal isn't simply to implement AI because the technology is available. It's to understand which problems are worth solving, what success looks like and whether the solution creates enough value to justify the investment.

For me, this is where technical evaluation and business outcome measurement need to come together, ideally from the beginning rather than after a solution has already been built.

More autonomous AI means we need to be more deliberate about security

The final technical discussion explored the security challenges associated with AI agents interacting with infrastructure, which was probably one of the more engineering-heavy parts of the evening. One idea I found particularly interesting was the tension between the probabilistic nature of AI models and the deterministic controls expected in enterprise systems.

AI models can respond differently depending on the context, instructions and information they receive. That's part of what makes them flexible and useful, but it also introduces challenges when we allow them to interact with systems where actions need to be predictable, authorised and auditable.

The presentation explored approaches to enforcing security boundaries through infrastructure and verification mechanisms, rather than relying solely on the AI model to decide what is appropriate.

To put that into a practical business context, imagine an AI agent supporting IT operations. It might be useful for the agent to investigate incidents, analyse logs and recommend potential fixes, but allowing it to make changes directly to production infrastructure introduces a different level of risk.

Should the agent be able to restart a service automatically? Under what circumstances would that be acceptable? When should a human review or approve the action?

I don't think there's necessarily one answer that applies to every organisation or use case. It depends on the potential impact, the safeguards available and the level of risk the business is prepared to accept.

What matters is that these decisions are made intentionally.

As AI agents become more capable, I think we'll need to become more thoughtful about the boundaries between what an agent can recommend, what it can execute independently and what still requires human oversight. And importantly, these decisions shouldn't happen in isolation from the people who understand the business processes and consequences involved.

What this means for people working at the intersection of business and technology

One of my biggest reflections from the evening was actually about the people involved in making enterprise AI successful.

Listening to engineers discuss agent architecture, context management, permissions and infrastructure security gave me a deeper appreciation for how much technical work sits behind solutions that might appear relatively simple to an end user.

At the same time, it reinforced why I think collaboration between business and technical teams is becoming increasingly important.

In my role, I often work with stakeholders from different business functions to understand their challenges and explore opportunities where data and AI might help. The initial idea might sound straightforward, but once we start unpacking the problem, there are usually many more questions to consider.

Do we understand the business process well enough? Is the required data available and reliable? How will the solution fit into existing workflows? What dependencies or constraints do we need to account for? How will we know whether the solution has actually improved anything?

And as we introduce more autonomous AI capabilities, we're adding further considerations around permissions, evaluation, security and operational accountability.

This is why I believe technical fluency is becoming increasingly valuable for Business Analysts, Product Owners and other professionals working in AI delivery. We don't necessarily need to be the people building every agent framework or implementing every infrastructure control, but we do need enough understanding of how these systems work to ask meaningful questions, challenge assumptions and collaborate effectively with engineering teams.

At the same time, I don't think the emergence of AI makes traditional business analysis and product skills any less relevant. If anything, understanding the problem, managing stakeholder expectations, defining requirements, prioritising opportunities and measuring outcomes become even more important when the technology itself is evolving so quickly.

The challenge is bringing these different perspectives together so we're not simply building something technically impressive, but creating solutions that are genuinely useful, responsible and sustainable.

Final thoughts

I always enjoy attending events that give me an opportunity to learn from people working in different parts of the technology landscape, particularly when the discussions go deeper into areas I don't necessarily work with hands-on every day.

This evening was a good reminder that while AI agents are becoming increasingly capable, there is still a lot of work involved in making them reliable enough for real enterprise environments.

It's one thing to demonstrate what an agent can do. It's another to ensure that it can operate securely, handle unexpected situations, respect organisational boundaries and consistently deliver value.

And I think that's where some of the most interesting challenges in enterprise AI are emerging.

As organisations move beyond experimenting with AI and start introducing more autonomous capabilities into their operations, the conversation will increasingly need to shift from what can we build? to what can we confidently deploy, operate and scale?

For me, that's also what makes working at the intersection of business, product and technology so interesting. There's always something new to learn, but the goal remains the same: understanding real problems and finding better ways to solve them.

I'd be interested to hear from others working in this space. What have you found most challenging when moving AI solutions from proof of concept into production?

And if you're exploring a career in Data & AI or working through similar challenges, feel free to reach out. I'm always happy to exchange ideas.