Agentic AI will break security models built for human users

Agentic AI changes the security problem because it changes the actor.

Traditional security models were designed around humans logging in, applications requesting access, and service accounts performing defined tasks. Agents blur those lines.

They can reason over a goal, call tools, access data, trigger workflows, delegate steps, and act across systems with limited human involvement. That creates a sharper risk than generative AI chat interfaces ever did.

The models built for human users assume judgement sits between permission and action.

Agentic AI weakens that assumption because the system can use the permission, decide the next step, and execute through connected tools before a human reviews the outcome.

For ANZ security leaders, the exposure now concentrates around five control gaps:

1. Identity: agents acting through user or service accounts without clear attribution
2. Access: permissions granted for static roles being used for dynamic actions
3. Data: poor provenance turning weak inputs into automated decisions
4. Testing: single agent assurance missing multi agent failure paths
5. Governance: policy sitting outside the workflows where agents operate

ADAPT found that 75% of CISOs say they are unprepared to securely adopt AI, while more than 50% of participating organsations remain below Level Two Essential Eight maturity.

At the same time, agentic AI is moving into operational automation, IT and security workflows, and data driven decision making, even though many use cases remain in pilots or narrow production deployments.

Security leaders now have to close the gap between autonomous productivity and controls still built around human sessions, static access, manual approval, and after the fact audit.

Treat agents as identities before they become shadow actors

The first security failure will come from treating agents as extensions of users or applications.

That assumption collapses once agents begin acting across workflows, tools, and data stores.

CoSAI’s agentic identity guidance states that traditional IAM assumes long lived principals, coarse roles, and a “trust for the session” model, which does not hold for agents whose behaviour and security context change rapidly.

It recommends treating agents as first class identities, separating agent permissions from on behalf of user permissions, eliminating standing privilege, and enforcing authorisation at every hop.

A human identity tells security teams who logged in. An agent identity must also show what code, model, toolset, delegation path, and runtime context produced the action.

Without that separation, investigators may only see a user or service account while the real actor remains hidden.

ADAPT’s research reinforces why this matters locally.

Many Australian organisations are still struggling with identity and access controls, legacy environments, governance standards, and outdated systems.

CareSuper Chief Technology Officer Simon Reiter points to identity management as one of the essentials that decides whether AI reaches production safely, alongside data quality and integration capability.

Security leaders should establish an agent identity register before scale.

Each agent should have an owner, purpose, risk tier, approved tools, data access boundaries, model or code version, and revocation path.

Recommended action: Register agents as distinct identities with named owners, scoped permissions, approved tools, delegated authority, and revocation controls before they enter production.

Back to top

Rebuild access control around action, intent, and context

Human access control usually answers who can access what.

Agentic AI forces a harder question: what is this agent trying to do, under whose authority, through which tools, and with what downstream effect?

That matters because agents can chain legitimate actions into risky outcomes.

A support agent with CRM access, knowledge base access, and email access may stay within its granted permissions while still aggregating and sending sensitive information externally.

A finance agent may inherit broad write access and alter supplier records if prompted through compromised inputs.

A role assigned to a person assumes the person interprets context before acting.

A role assigned to an agent can become an execution path, where permission, reasoning, tool use, and workflow change happen inside the same loop.

CoSAI’s guidance calls for short lived credentials, on behalf of tokens that preserve both the agent and the user, adaptive access policies, and continuous re evaluation of permissions based on identity, intent, telemetry, risk indicators, and environment.

OWASP’s Agentic AI, Threats and Mitigations extends this concern from access design into threat modelling.

It identifies tool misuse, privilege compromise, identity spoofing, agent communication poisoning, rogue agents, and insecure inter agent protocol abuse as agentic threats that emerge when systems can reason, remember, delegate, and invoke tools.

The report also warns that agents can chain tools in unexpected ways, allowing sensitive data to move through authorised APIs and appear in user visible responses, even when individual tool permissions seem valid.

This is where many current AI programmes are weakest. ADAPT found that 50% of AI pilots and deployments are not covered by a formal governance framework, while only 7% of Australian CIOs report enterprise wide AI governance with board involvement.

Security teams should require step up approval for irreversible actions, payment changes, administrative changes, production changes, sensitive data export, and agent to agent delegation.

The control point should sit at the tool and API layer, not only in the interface where the user interacts.

Recommended action takeaway: Replace static role based access with task scoped, short lived credentials, contextual authorisation, and step up approval for high impact actions.

Back to top

Secure data provenance before agents automate decisions

Agentic AI raises the consequence of poor data discipline. A human may question a suspicious report. An agent may turn the same data into an action.

ASD’s ACSC, alongside international cyber agencies, warns that data used during AI development, testing, deployment, and operation must be protected because manipulated data can manipulate system logic.

Its AI data security guidance highlights data supply chain risk, poisoned data, and data drift, and recommends provenance tracking, encryption, digital signatures, secure storage, trust infrastructure, and ongoing risk assessment across the AI lifecycle.

Human centred controls often rely on review, escalation, and professional judgement to catch data quality issues before a decision is made.

Agentic workflows reduce that buffer because bad data can become an automated recommendation, transaction, alert, or workflow trigger.

ADAPT’s evidence shows this is an operating constraint, not a technical detail.

40% of Australian CIOs identify data foundations as the number one constraint to scaling agentic AI, while only 8% say their organisation is optimised for AI data readiness.

Mission Australia Chief Information Officer Peter Smith makes the same point from an ERP lens: cleaner data, better integration, and measured AI matter more than upgrades alone.

Security leaders should classify agent accessible data by sensitivity, source trust, usage rights, and allowable action. Agents should be prevented from using low trust data for high impact decisions unless validation, provenance, and human review thresholds are met.

Recommended action takeaway: Map the data agents can access, validate its lineage and trust level, and block high impact automation where provenance, sensitivity, or usage rights are unclear.

Back to top

Test multi agent systems as systems, not safe components

A safe agent does not guarantee a safe agent network.

The Department of Industry, Science and Resources highlights emerging risks when multiple large language models operate together in multi agent systems.

The identified failure modes include inconsistent performance derailing complex processes, cascading communication breakdowns, shared blind spots, repeated mistakes, groupthink dynamics, and coordination failures.

Traditional single agent testing does not capture these risks, especially for critical infrastructure and essential services.

This is where human user security models break most visibly.

Human workflows usually have natural friction between teams, systems, and approvals.

Multi-agent systems can remove that friction, allowing one flawed output, poisoned context, or delegated task to propagate across connected agents.

That warning aligns with ADAPT’s market evidence.

Agentic AI adoption in Australia is concentrated in operational automation, IT and security workflows, and data driven decision making, which are exactly the environments where one agent’s decision can trigger downstream workflows.

At The Cheesecake Shop, Chief Information Officer Brad Dight is rebuilding the 34 year old retailer around cleaner data, modernised systems, and AI ready operations, showing how agentic AI pressure is already reaching core enterprise workflows.

Security leaders should test agent systems under realistic interaction patterns.

That means simulating poisoned inputs, conflicting agent outputs, delegation loops, privilege escalation attempts, tool misuse, and alert overload.

Controls should measure the behaviour of the system, not only the safety of each component.

Recommended action takeaway: Red team agentic workflows end to end, testing inter agent communication, delegated authority, tool chaining, failure escalation, and human review overload.

Back to top

Move governance from policy documents into operating rhythm

The governance gap is now a security exposure.

ADAPT’s 2026 research shows many organisations are still AI adjacent: experimentation is happening, copilots exist, and isolated use cases are running, yet the operating model has barely moved.

That is dangerous for agentic AI because autonomy magnifies weak ownership.

If no one owns the workflow, no one owns the failure path.

Gabby Fredkin, Head of Analytics and Insights at ADAPT, gives this shift a sharper operating lens.

In his CIO Edge presentation on scaling AI agents safely, he highlighted that 77% of organisations are investing in AI agents, but the real lifecycle is less about “pilot, deploy, scale” and more like buy it, use it, break it, fix it.

That pattern exposes why governance has to sit inside live operating rhythms rather than waiting for a mature enterprise rollout.

ADAPT Executive Advisors Claudine Ogilvie, Mark Cameron, and Brett Raven similarly argue that leaders need clearer risk appetite, operating discipline, and accountability before AI can scale.

Security leaders should embed agentic AI into existing risk forums, architecture review boards, change processes, and incident response.

Every agent should have a business owner, security owner, review cadence, kill switch, and evidence trail that can show what it did, why it acted, and who approved the access.

Recommended action takeaway: Embed agentic AI controls into architecture review, change approval, risk acceptance, access recertification, monitoring, and incident response, with evidence trails for every production agent.

Back to top

The control model must catch up to the actor

Agentic AI will expose every weak assumption in security models built for human users: standing access, shared accounts, static roles, incomplete logging, unclear ownership, and governance that appears after deployment.

The ANZ security task is to narrow the gap between experimentation and accountable autonomy. Before agents become embedded in core workflows, security leaders need to prove five things:

• Which agents exist
• What each agent can access
• Whose authority each agent acts under
• Which data, tools, and systems shape each action
• How quickly access can be reduced, revoked, or investigated

That means building agent identity, contextual authorisation, data provenance, multi agent testing, and governance evidence into the operating model before scale. Progress will depend on proving control at the same speed the business wants agents to act.

Back to top