The next phase of enterprise AI adoption will depend less on the raw capabilities of AI models and more on organisations’ ability to govern, secure and control increasingly autonomous agents, according to Microsoft.
As AI tools evolve from assistants into agents capable of autonomously executing complex tasks and managing operational processes, organisations must find ways to maintain human oversight, protect organisational knowledge and ensure the systems remain safe, trustworthy and aligned with business objectives so they deliver measurable value.
Satya Nadella, chairman and CEO of Microsoft, outlined how organisations can achieve this goal during his keynote at the 2026 Microsoft Frontier Tour in London, UK.
Nadella identified three fundamental challenges for organisations adopting agentic AI: ensuring humans remain in control, retaining ownership of organisational knowledge and establishing trust in increasingly capable AI systems.
When designing and implementing AI systems, organisations have to ensure people remain in control. This requires them to determine which responsibilities can be delegated to agents, where human judgement remains essential and how employees can intervene when necessary.
“We have to recapture that sense of empowerment,” said Nadella. “You still have agency, you have control; it’s your ambition that’s driving these systems as opposed to the other way around.”
Unlike traditional software, AI systems are learning systems that continuously improve through their interactions with users and organisational data. Nadella described this as a “reverse information paradox”, arguing that businesses must ensure they retain ownership of the knowledge generated through these interactions.
Noting that the goal for organisations is to “capture that knowledge and keep it locally” for themselves, Nadella said: “You have to own the knowledge that gets created, both with the human capital that you have, as well as the token capital that you generate.”
To achieve this aim, organisations should separate organisational data, memory and actions from the underlying AI models.
“The key is to separate out the harness from the models so that you can keep the action space, as well as the context and memory, outside of a model,” said Nadella.
Taking this approach gives organisations greater flexibility and control as the AI landscape evolves, reducing their dependence on individual providers and allowing them to swap models without disrupting business processes or losing valuable organisational knowledge.
Nadella proposed what he called a “simple acid test” for organisations to follow when building enterprise AI architecture.
“Can you remove any model and yet still hold the line on any evaluation of your agentic system?” he asked. “If not, that means you now have a lock-in of a given model.”
However, retaining control of organisational knowledge and AI architecture is only part of the challenge. As AI agents become increasingly capable of accessing information and taking action on behalf of employees, organisations must also address the security and governance risks associated with their deployment.
“Our scaling laws are producing even more powerful models that are less and less trustworthy,” warned Nadella.
For Nadella, establishing trust in these systems is essential to enabling businesses to adopt AI at scale. He advised organisations to “treat these models as a new form of insider risk” and establish a zero-trust architecture around them. “Avoid nested black boxes so that you can have real trust in the systems that you‘re deploying,” he said.
Businesses should not view AI agents as software tools, but as operational entities that require strict monitoring and policy controls.
“You have to be able to ID every one of these agents,” said Nadella. “You need to be able to log every one of the activities of the agent. If it has abnormal behaviour, you need to be able to block it and contain it.”
For businesses, the priority is to build governance into AI architecture from the outset, with clear accountability for the decisions and actions agents undertake. “Ultimately we individually have to be accountable for the outcomes of these systems,” said Nadella.
Microsoft is developing a broader technology platform to help organisations build, deploy and govern their own agentic AI systems. During live demonstrations, Microsoft executives showcased how organisations can use Copilot to build applications, manage workflows and automate tasks. They also explained how Microsoft IQ provides the underlying architecture for connecting organisational data, memory, tools and models, while keeping these components separate from the AI models themselves.
The company also highlighted Agent 365, which provides a central control plane for managing agent identities, permissions, policies, security operations and financial oversight. Microsoft’s Sovereign Control Panel, meanwhile, gives organisations visibility into data residency, confidentiality and business continuity, helping them maintain control over where and how their AI workloads operate.
Efe Abugo, senior product marketing manager for Microsoft Copilot, demonstrated how Copilot could be used to pull together a product launch (Photo: Technology Record/Rebecca Gibson)
Marlene Mhangami, senior developer advocate at Microsoft, demonstrated how organisations could use Copilot to automate tasks and coordinate workflows (Photo: Technology Record/Rebecca Gibson)
Microsoft’s message is clear: becoming a frontier organisation requires more than adopting increasingly capable AI models. Businesses must establish the governance frameworks, underlying architecture and operational controls needed to ensure autonomous agents deliver value without compromising security, organisational knowledge or human accountability.
For Nadella, Microsoft’s ultimate goal is to provide a platform that enables organisations to “build systems that are empowering people in your organisations” while creating the “trust architecture around these sophisticated agentic systems”.
Read more: New Microsoft survey predicts UK could be a world-leading AI economy within five years