AI security

Zero Trust AI (ZTAI)

Secure the data, models and workflows of AI, from development to controlled execution and output release.

An AI core connected to data and services through several security boundaries and access gates
About this collection

Start with the zero trust principles, then follow the operational foundations, maturity stages and practical controls. Then examine indirect access through code, outputs and infrastructure administration, and how to keep autonomous agents within enforceable authority boundaries.

1

From Zero Trust Architecture to Zero Trust AI

Apply zero trust to data, models and the AI lifecycle.

Read chapter: From Zero Trust Architecture to Zero Trust AI

The rapid adoption of AI has brought sensitive data, valuable intellectual property and consequential decisions into systems that are difficult to secure with perimeter defenses alone. An AI service is rarely just a model behind an API. It includes data preparation, training code, external dependencies, model artifacts, deployment infrastructure and people with different kinds of access.

Traditional security designs often treated the internal network as a relatively trusted space. That assumption is especially weak in distributed AI workflows. Alongside familiar cybersecurity threats, these workflows face data poisoning, model inversion and adversarial inputs. An attacker may undermine a decision without taking over a server, or extract information through an otherwise legitimate model interface.

In this collection, Zero Trust AI (ZTAI) means applying zero trust principles to the specific engineering problems of AI. It is the architectural approach developed here, rather than the name of a separate, approved security standard. Its scope includes the data pipeline, model lifecycle and operational workflow, as well as identities and access.

Why perimeter security fell short

Earlier network architectures commonly divided the world into a trusted inside and an untrusted outside. Isolated networks and controls at their entry and exit points formed the main defensive boundary. Web application firewalls, intrusion detection and prevention systems, and data loss prevention tools added further layers.

A perimeter-oriented network divided into LAN, DMZ and internet segments
A conventional network layout built around internal and external boundaries.

Those layers did not eliminate information leaks or compromised servers. A staff member could disclose data; an apparently legitimate connection could carry an attack. Cloud infrastructure, remote work and personal devices also made the distinction between inside and outside increasingly unreliable.

John Kindervag introduced the zero trust model at Forrester in 2010. Its familiar shorthand is “never trust, always verify”: a user, device or application should not receive access simply because it is inside an organizational network. Identity must be established, the requested action authorized, and access limited to what the task needs. Authentication is the beginning of that decision, not permission to use every resource.

Strong identity controls, least privilege, segmentation, device posture assessment and ongoing monitoring support this approach. Decisions need to account for changes in the user, device, workload and operating conditions throughout a session.

The NIST architecture separates policy decisions from their enforcement. A policy engine evaluates access; a policy administrator establishes or terminates the communication path; and a policy enforcement point applies that decision. Identity information, asset state, threat intelligence, access policies and activity records inform the process.

Zero trust logical components: policy engine, policy administrator and policy enforcement point, with supporting identity and security information
Logical components of zero trust architecture. See NIST SP 800-207.

NIST already treats resources, services and workflows as part of zero trust. The AI-specific discussion here makes the controls concrete for a model’s lifecycle; it does not imply that zero trust was originally limited to protecting network traffic.

What AI changes

The first difference is the workflow. Developing and operating a machine learning service involves several teams, frameworks and technologies. A weakness can enter through a dataset, a notebook, a training dependency, a model registry or a serving configuration. Securing the application endpoint leaves much of that chain untouched.

The second difference is the data. A model can consume large quantities of sensitive information during collection, preprocessing, training and inference. Protection must continue while that information is being transformed. A processed dataset, an embedding index or a trained model is not automatically safe to share merely because it no longer looks like the original records.

AI development is also unevenly integrated with software engineering and DevSecOps. Outsourced projects can expose gaps in ownership, review and release responsibilities. Open-source packages are essential to the workflow, but their rapid evolution makes dependency risks harder to track. Security tools designed for conventional applications may not inspect model artifacts or data transformations adequately.

A model is not an ordinary software artifact. Teams need to understand how it was produced, how it is loaded and what execution capabilities its format or dependencies require. Operations teams unfamiliar with that chain can introduce vulnerabilities through otherwise routine deployment choices.

Finally, model behavior depends on data and is often probabilistic. A successful build and an authenticated caller do not establish that an output is correct, appropriate or free of confidential information. Those are additional properties that require their own evidence.

Threats specific to AI

Data poisoning manipulates data used to train or update a model, changing its behavior or introducing a targeted failure. Model inversion attempts to infer sensitive information about the data behind a model from its behavior or outputs. The information recovered depends on the attack and model; it should not be assumed to reproduce every training record.

Model theft includes unauthorized acquisition or replication of a trained model. Adversarial examples are inputs constructed to mislead a model, sometimes with changes that are difficult for a person to notice.

A poisoned update can introduce malicious code or alter a model’s behavior through a compromised update path. Information leakage occurs when an output exposes sensitive input or training information. These risks make the integrity of data, models and processes an ongoing concern, rather than a one-time check at network admission.

Applying zero trust to the AI lifecycle

AreaGeneral zero trust concernAI-specific application in this collection
Core principleNo implicit trust based on location or ownershipDo not assume data, models or outputs are trustworthy because they came from an internal pipeline
Protected resourcesAssets, services, identities and workflowsDatasets, training jobs, model artifacts, retrieval stores and inference services
AuthorizationGive a verified identity only the access neededBind a workload to an approved dataset, operation, execution environment and output destination
Trust assumptionsReassess access as context changesReassess changed data, dependencies, models and deployment conditions
Supporting controlsIdentity, segmentation, least privilege and monitoringAdd data provenance, model integrity checks, adversarial evaluation and output review appropriate to risk
ThreatsUnauthorized access, lateral movement and insider misuseAlso consider poisoning, extraction, adversarial inputs, unsafe updates, disclosure and unapproved AI use
Operational lifecycleApply policy to the resource and its useIntegrate controls into collection, training, evaluation, deployment, monitoring and retraining

The practical question is not only who may call a model. It is also who may change its processing, what information it can use and what its outputs can reveal. MLOps as the foundation for Zero Trust AI explains how a traceable lifecycle makes those controls implementable. The complete reading sequence is available in the ZTAI technical collection.

Open article in a new tab
2

MLOps as the foundation for Zero Trust AI

Make data, code, experiments and releases traceable.

Read chapter: MLOps as the foundation for Zero Trust AI

Zero Trust AI requires more than security tools surrounding a model. If data, code, experiments, model artifacts and deployments cannot be traced and controlled, continuous verification remains a policy statement. MLOps provides the engineering processes through which that policy can operate.

Machine learning operations, or MLOps, brings data science, software engineering and operations together to build and maintain reliable production models. It covers the entire model lifecycle, including the changes that happen after deployment.

Modern machine learning capabilities, including asset protection, version control, distributed training, scalable inference, registries and CI/CD/CT
The operational requirements extend beyond training a model: artifacts, infrastructure, access and ongoing delivery all need management.

How MLOps relates to DevOps

DevOps automates and monitors software development from coding and building through testing, release, deployment and operation. MLOps applies those practices to additional stages: collecting and exploring data, preparing it, training a model, evaluating it and monitoring its behavior in use.

Both aim to improve delivery speed, quality and operational reliability. Machine learning adds dependencies that are less prominent in conventional software. A model’s behavior depends on its training data; changing data may require retraining; and the resulting model artifacts need management alongside the application code.

Data and machine learning practices combined with development and operations in an MLOps lifecycle
MLOps connects data and machine learning work with the software delivery process.

Why the operational foundation matters

The first benefit is a shorter path from research to production. Automation reduces the manual work needed to move a model from an experiment into a service. Standardized steps also reduce avoidable mistakes, while monitoring makes problems easier to detect. Faster release is useful only when the organization can understand and manage what it is releasing.

The second benefit is scale. Without a shared process, every model becomes a separate project dependent on particular people and their local tools. A common lifecycle makes training, deployment and maintenance repeatable across models and teams.

The third benefit is governance. The organization needs to reconstruct which data, code and configuration produced a model, which evaluation supported its release and which version is serving a given request. This evidence supports security and privacy reviews as well as operational investigation. It must continue through retraining and replacement, rather than ending at the first deployment.

The main parts of the lifecycle

Data management includes collection, cleaning, feature engineering, versioning and a catalog. Code and configuration management tracks model code, training scripts, dependencies and settings. Training and validation organize experiments, hyperparameter selection, evaluation and approval into a process that can be repeated and increasingly automated.

The trained model must then be stored, versioned and associated with its supporting evidence. A registry can connect an artifact with its source code, dataset references, experiment results and release status. Deployment controls govern how that artifact reaches a server, an API service or an edge device.

After deployment, monitoring checks performance, changes in the input distribution and indications that the model no longer fits its intended use. Feedback brings new operational evidence into the next development or training cycle. A retraining trigger should lead through the required validation and release controls; it is not permission to replace a production model without them.

Security and governance run through all these activities. Access policies, retention rules, artifact integrity and approval requirements cannot be deferred to a final security review after the pipeline has already exposed sensitive data.

Where MLOps supports ZTAI

MLOps is not itself zero trust. It creates an observable, structured lifecycle in which security decisions can be enforced.

Versioned datasets, code and experiments make provenance and integrity assessable. A defined release path provides a place to admit or reject an artifact. Monitoring creates evidence for reassessing a deployment after its initial approval. Together, these capabilities move verification beyond login and into the operation of the service.

They also make limits visible. Recording a model’s origin does not prove that its training data is benign. Automating a pipeline does not remove the privileges of everyone who can change its code or inspect its outputs. Those decisions require the principles and controls of ZTAI to be designed into the workflow.

Most organizations build this capability gradually. The ZTAI maturity model describes a progression from isolated manual work to automated operations with development and sensitive production processing separated. Its final stage adds an architectural goal: routine work should not require people to handle raw sensitive data. Whether that goal is actually achieved depends on controlling indirect access as well as direct access.

Continue with the five-part ZTAI collection.

Open article in a new tab
3

A maturity model for Zero Trust AI

Move from manual work toward controlled automation.

Read chapter: A maturity model for Zero Trust AI

An organization cannot move from scattered notebooks and manually copied datasets to a controlled AI lifecycle in a single step. It first needs to know how models are built and released, then make that process repeatable, and finally enforce who can use sensitive information at each stage.

The model below describes that progression through five levels, from 0 to 4. Its operational foundation is MLOps. The additional ZTAI objective is to redesign routine processing so that people can define a task and evaluate its results without repeatedly seeing, moving or manipulating raw sensitive data.

This is the architectural interpretation used in this collection. It draws on the staged thinking of the CISA Zero Trust Maturity Model and on Microsoft’s MLOps maturity model. The five ZTAI stages below are not an official CISA scale or a certification scheme. In particular, the data-access restrictions proposed at level 4 are an additional security design objective, not something that an MLOps maturity rating guarantees.

Levels 0 through 3 establish progressively better operational control. They are useful foundations for zero trust, but none proves that the system already enforces it. An organization may also have capabilities from several levels at once: automated training in one team, manual deployment in another and an uncontrolled maintenance path across both.

Level 0: isolated, manual work

At level 0, data scientists, data engineers and software engineers work largely in isolation. Collaboration takes the form of files and instructions passed from one person to another. There is no dependable shared process connecting data preparation to the production application.

Data is collected manually, and the compute environment may be a workstation or an unmanaged server. Experiments are not consistently recorded. A successful experiment produces a model file, often without a complete account of its data, parameters, dependencies and environment.

Release is also manual. A scoring script may be written after experimentation and remain outside version control. A single person can become responsible for deciding that the model is ready, handing it over and explaining how to run it. The application team depends heavily on that person’s knowledge.

Level 0: manually loaded data, preprocessing and training in a manually created environment, followed by manual deployment
Level 0: an experiment can produce a model, but the lifecycle depends on manual work and individual knowledge.

Reproducing results is difficult. Different tools and undocumented settings make comparisons unreliable, and changes in data may not be distinguishable from changes in code. Sensitive data can spread across personal environments without a clear record. Shared GPUs and other scarce resources are also hard to coordinate.

The immediate improvement is to make the work visible: version code, record inputs and experiments, define a repeatable handover and establish responsibility for a release. Adding a security product around an undocumented workflow will not supply that missing evidence.

Level 1: software delivery improves, but model work remains separate

At level 1, parts of the data and software process become automated. Data collection may follow a pipeline, and code is placed in a repository. Software engineers have a clearer handover from the data team and can automate application builds, tests and packaging.

Model development still depends on manual experiments and partially managed environments. Data preparation and training may not be reproducible from a single recorded specification. The team creates evaluation or scoring scripts manually, although it now versions them. Releasing a new model still requires direct involvement from the data team.

Level 1: a data pipeline and catalog feed model development, with code capture and tests but manual model deployment
Level 1: version control and software delivery practices improve coordination, while the model lifecycle remains partly manual.

This reduces some release friction, but application tests do not establish model quality. The software may start correctly and accept requests while the model performs poorly on real inputs. Feedback from production is still limited, and experiment history may not explain why one model replaced another.

Security remains dependent on people handling data carefully. Uncontrolled data copies, broad access and informal GPU allocation can persist even when application builds are automated. The next step is to bring training and its evidence into a managed process.

Level 2: automated, traceable training

At level 2, data engineers and data scientists collaborate through an automated training pipeline. Data ingestion and processing run in a managed environment. Experiments have recorded parameters, results and artifacts, and the model registry provides a stable handover to the application team.

Code, data references, model versions and application versions can be associated with one another. Evaluation results are stored with the model rather than living in a separate document or a person’s notebook. A model can be reproduced much more reliably, and a change can be traced to its inputs.

Level 2: a data catalog feeds an automated training pipeline, with captured experiment metadata and a model registry
Level 2: training becomes repeatable and its artifacts acquire a traceable history.

Release can still be manual. Software engineers may receive a model through a reliable interface without being closely involved in the training process. That is an improvement over file exchanges, but the connection between model evaluation and the quality of the complete product can remain weak.

A well-managed training environment also does not settle all governance questions. Who can change the training code? Who can inspect the dataset, download a checkpoint or approve deployment? Resource scheduling and deployment across several machines may still require separate work. Traceability makes these questions easier to answer; it does not answer them automatically.

Level 3: automated model deployment

At level 3, data science, data engineering and software engineering operate as parts of a connected delivery process. Training uses managed data and compute, experiments and evaluations are versioned, and releases follow an automated deployment pipeline.

A model is packaged with its environment and the evidence needed for release. Tests cover code and model integration, while quality assurance has a defined role in deciding what can reach production. Evaluation may combine automated checks with human review for the criteria that cannot yet be assessed reliably by machine.

Level 3: a registered model is packaged, assessed by quality assurance and released through an automated delivery process
Level 3: deployment becomes a controlled part of the lifecycle instead of a separate manual handover.

The organization can trace a deployed model back to the process that created it. It can also make release criteria explicit and apply them consistently. This improves the ability to investigate a failure and return to an earlier version.

The remaining gap is often the feedback loop. A deployment can pass its tests without delivering the desired user experience, and the pipeline may not yet turn production evidence into a validated replacement model. Automated delivery does not guarantee uninterrupted service, effective GPU sharing or confidentiality. Broad administrator access and permissive output channels can remain unchanged beneath an efficient release process.

Level 4: a connected lifecycle with enforced boundaries

At level 4, data, training, evaluation, deployment and monitoring form a connected operational cycle. Relevant changes can trigger a new training run. The resulting model passes through validation and release policy before replacing a production version. Feedback is recorded and used to improve later decisions.

Continuous integration, delivery and training—CI/CD/CT—support this cycle. Unit tests, integration tests and tests of externally observable behavior provide different kinds of evidence. Human oversight remains responsible for the purpose of the system, the criteria it must meet and the exceptions it may accept.

Level 4: monitored data changes trigger training, validation and deployment, with feedback from the production application
Level 4 connects production feedback to training and release. Automation still operates within validation and authorization rules.

For ZTAI, the important additional step is to remove unnecessary human handling of sensitive data. Developers define and test a process; approved execution uses the real data within a controlled environment. The organization must enforce the separation rather than relying on a request that developers avoid looking at the records.

This is a design target, not a claim of maximum security. An automated job can leak information through logs, model files or API responses. An infrastructure administrator may still be able to inspect memory. The controls needed to address those paths are developed in When people cannot see the data, has their access really been removed?.

Separate development from sensitive production processing

A useful level 4 arrangement gives developers public, synthetic or appropriately de-identified data for ordinary development. The production pipeline uses sensitive data inside a separately governed environment. Synthetic and de-identified datasets still need a disclosure assessment; those labels alone do not make a copy safe to release.

Development experiments and source control separated from production training, model registration, deployment and monitoring
Development defines the processing. Production executes the accepted version against controlled data and retains the required evidence.

The separation can be organized around four boundaries:

  1. Data preparation. Developers define the transformations. Approved execution collects and prepares the real data under access and provenance controls.
  2. Training. A specified version of the code and its parameters runs in an authorized environment. Artifacts enter a controlled registry, and monitoring records the conditions that may justify retraining.
  3. Evaluation. Automated and, where required, human review assess the results through approved interfaces. Evaluation does not become an unrestricted route to raw samples.
  4. Deployment. An accepted model is exposed through the intended service. The application team consumes that service without automatically receiving its training data or weight files.

These are related boundaries. A policy that governs the training input but ignores the evaluation report or model download leaves an indirect access path open. Release permissions must apply to every result that someone can receive.

A constrained variant for isolated environments

Some sensitive or mission-critical workloads cannot support an always-connected platform or a complete CI/CD/CT stack. Network restrictions, operational constraints and cost may require a smaller arrangement: develop with non-sensitive data, move an accepted processing specification across a controlled boundary and perform final training inside an isolated environment.

Public-data development separated from protected-data training, with controlled transfer, management and deployment targets
A conceptual variant for isolated or tightly controlled environments. Its suitability depends on the required data flows and threat model.

The architectural objective can survive a reduction in automation. Routine development should not require the sensitive dataset to leave its protected environment. But isolation, a one-way transfer mechanism or an automated training job does not by itself protect plaintext from every privileged administrator. The design must state which administrators are trusted, what their privileges permit and what evidence supports any stronger claim.

The same discipline applies to maintenance. If every difficult failure requires exporting a memory dump or granting unrestricted access, the separation will disappear in practice. Development, diagnostics and recovery tools are part of the security architecture.

Comparing the levels

LevelMain capabilityTypical activitiesRemaining concerns
0Manual, isolated model workIndividual experiments and file-based handoversPoor reproducibility, unclear ownership, uncontrolled copies and resource use
1Software delivery practicesVersioned code, application tests and more organized releasesModel development and feedback remain dependent on people
2Automated trainingManaged environments, experiment tracking and model registrationManual release, incomplete product feedback and unresolved access policy
3Automated deploymentConnected teams, release tests and traceable promotionFeedback, continuity and confidentiality still require explicit design
4Connected lifecycle with a ZTAI separation goalControlled CI/CD/CT, monitored feedback and separated development and sensitive processingIndirect access, administrator privileges, output release and exceptions must be enforced and audited

Human responsibility does not disappear as the level increases. People still decide what the system is for, which risks are acceptable and when it must stop. The aim is to reduce unnecessary contact with raw data while making those decisions clearer and more accountable.

Use the model to identify the next missing capability in an actual workflow. A higher label is less useful than evidence that a previously uncontrolled path is now governed. The next article maps that work to ZTAI principles and practical controls; the collection page keeps the full sequence together.

Open article in a new tab
4

Zero Trust AI: principles and practical controls

Enforce identity, data, model and output policies.

Read chapter: Zero Trust AI: principles and practical controls

The ZTAI maturity model describes how an organization can move toward a controlled AI lifecycle. That progression needs specific controls: what is verified, which action is authorized, where a decision is enforced and what happens when the required conditions no longer hold.

Zero Trust AI, as developed in this collection, applies zero trust to the data, models and workflows of AI. The principles below should be interpreted against an explicit threat model. A control that limits a developer does not necessarily limit the infrastructure administrator, and a control that verifies an artifact’s origin does not establish that its behavior is safe.

Verify explicitly and reassess access

Every request to use a dataset, model or compute resource should be tied to a verified identity and an authorized purpose. Being inside the network, belonging to the development team or running as an internal service is not enough.

Verification continues through the lifecycle. Data changes, models are replaced, dependencies evolve and user or workload behavior can change. A decision made when an account was created cannot serve as permanent approval for every later use.

The identity of the requester is only part of the decision. The relevant context may include the dataset, operation, software version, execution environment, destination of the result and duration of access. A training job authorized for one dataset and output location should not inherit permission to process another dataset or write results anywhere it chooses.

Grant only the access required for the task

Least privilege applies to people, services, training jobs and inference workloads. Roles provide a starting point, but the decision may also depend on attributes such as data sensitivity, current risk and the particular operation requested.

A data scientist does not necessarily need unrestricted access to every raw record. Schemas, distributions, approved statistics and carefully selected development samples can support substantial work. Developers can define preprocessing and training parameters while an authorized pipeline performs the sensitive processing elsewhere.

Development and production separated so that code and experiments can be prepared without routine access to production data
Separating development from sensitive processing reduces the need for direct access. The code, output and administrator paths still need their own controls.

This arrangement requires a usable development process. If developers cannot inspect a schema, reproduce a failure or evaluate a model through approved interfaces, they will repeatedly need exceptions. The operational capabilities described in MLOps as the foundation for ZTAI help make least privilege practical.

Permission to define processing also needs limits. Someone who can run arbitrary code against sensitive data and receive arbitrary output may be able to reconstruct the access that the architecture intended to remove. Indirect data access examines that combination in detail.

Assume a component can be compromised

Design for a compromised account, a tampered dataset, an unsafe dependency or a malicious model artifact. The system needs to contain the effect, preserve useful evidence and support recovery.

Sensitive data should remain behind controlled interfaces and explicit processing boundaries. Engineers and administrators should not acquire unrestricted access simply because their role is operational. Where an administrator remains trusted, state that assumption; where the design intends to exclude them, the underlying technology must support that stronger boundary.

Segmentation limits how far a compromised component can reach. Separate authorization for changing policy, running a job and releasing a result also makes it harder for one compromised role to remove its own restrictions. These protections must survive routine changes and failures, rather than applying only to the ideal path through a diagram.

Review outputs according to their consequences

Model outputs need different levels of verification depending on their use. An internal summary with limited consequences may be suitable for lightweight automated checks. A recommendation that influences a staff decision may need human review. Customer-facing, financial, clinical or operational decisions can require stronger validation, traceability and explicit approval.

The relevant questions include who will act on the output, what could go wrong and whether the decision can be reversed. The source data, model version and review history should be available at the level required by that risk.

A fluent answer is not evidence of accuracy, and an accurate answer may still disclose information that its recipient is not authorized to receive. Quality review and confidentiality controls address related but different properties.

Improve the review process through feedback

Review decisions can reveal recurring failure patterns. Recording those patterns helps teams improve tests, refine policies and reduce repetitive manual work. Some low-risk checks may eventually be automated, while difficult or consequential cases remain subject to human judgment.

Feedback should enter a controlled improvement process. A reviewer action is not automatically a trustworthy training label, and a production correction should not silently change a model or release policy. Changes need evaluation before they affect later decisions. Automation should make review more consistent without turning an unexamined feedback loop into a new source of error.

Protect data throughout its lifecycle

Protecting storage and network traffic is necessary, but sensitive information may also be exposed while it is being processed. Encryption at rest and in transit addresses different threats from protection during execution. Confidential computing and cryptographic approaches such as homomorphic encryption have different capabilities, performance costs and trust assumptions; the choice must fit the workload.

Development and testing should use data whose disclosure risk has been assessed. Masking, de-identification and synthetic generation can reduce exposure, but their effectiveness depends on the transformation and the information retained. Treating every transformed dataset as public would undermine the separation between development and production.

Provenance and integrity records help establish where data came from and whether it changed. Signed artifacts, hashes and protected audit records support that evidence. They cannot establish that the original source was truthful or that a correctly signed dataset contains no poisoning.

Classification must also follow derived artifacts. An output should retain the relevant restrictions of its inputs unless a controlled release decision justifies a different treatment. That includes statistics, embeddings, synthetic datasets, checkpoints and models—not only files containing recognizable records.

Protect the model and its loading path

Model artifacts need access control, integrity verification and, where appropriate, encryption. Their APIs need authenticated, authorized access and monitoring for misuse or extraction attempts. Evaluation should include the adversarial conditions relevant to the intended application.

Adversarial training and limiting a model’s operating domain can improve robustness against particular threats. Neither establishes resistance to every attack. The system must still constrain what the model can access and what actions its outputs can cause.

Model ingestion deserves particular attention. Downloading a weight file can also introduce an unsafe deserialization path, custom code or unreviewed dependencies. A controlled admission process should inspect the package, restrict execution and accept only the formats and capabilities that the deployment needs.

Converting an artifact to a format such as SafeTensors can remove a later dependency on executable serialization, but conversion is not safe if it first loads an untrusted executable format in a privileged environment. Treat inspection and conversion as isolated, resource-limited processing of untrusted input. A safer weight format also does not prove the model’s behavior benign or make accompanying code trustworthy.

Moving-target techniques, including changes intended to make a model harder to probe, require evidence of their benefit and their effect on quality. They should not replace established access controls or be presented as a general solution to model theft and adversarial attack.

Secure the infrastructure and operational process

Separate development, training and serving environments according to their access needs and consequences of compromise. Monitor network activity, API use and model behavior for signs of extraction, misuse or unapproved AI services. Those records themselves may contain sensitive information and need controlled access and retention.

Use strong authentication, including multifactor authentication where appropriate for people. Automated services need verifiable workload identities and narrowly scoped credentials, preferably with limited lifetimes. Human authentication mechanisms should not be treated as a substitute for properly designed machine identity.

A policy engine such as Open Policy Agent can make authorization rules explicit and consistently enforceable. Its independence depends on who can modify the policy, deploy the engine or obtain its signing and administrative credentials. Running the engine in a separate service does not help if the processing workload can rewrite its rules.

Security work belongs inside CI/CD: dependency review, patching, static and dynamic analysis, artifact checks and tested incident procedures. Threat modeling should account for the boundaries between the organization, its platform operators and external providers. Each party needs a clear responsibility for the controls it actually operates.

AI-assisted monitoring and incident analysis may help identify patterns or prioritize work. Their recommendations still need suitable validation and access limits. Giving a security assistant broad action privileges creates another workload whose authority must be governed.

The hardware and platform choices matter too. The English GPU and server articles explain why a server’s operating environment, interfaces and supported configuration need to be assessed together with its accelerators.

Introducing the controls into an existing organization

The difficulties are not limited to selecting tools. Teams may lack the required expertise, legacy systems may not expose useful policy boundaries, and established working habits may rely on unrestricted access to real data. Rebuilding pipelines and providing safe development environments takes time and money.

A staged introduction should start with a concrete workflow and its most consequential access paths. Establish the evidence needed to understand it, then reduce broad privileges, control release paths and test how the restrictions behave during maintenance. Existing systems may need architectural changes before stronger claims become realistic.

ZTAI is an ongoing combination of policy, engineering, operations and review. A useful test is whether the organization can explain a specific access decision, show where it was enforced and demonstrate what happens when the required conditions fail. The final article in the collection applies that test to the difficult claim that people no longer have access to sensitive data.

Open article in a new tab
5

When people cannot see the data, has their access really been removed?

Examine code changes, output release and administrator powers.

Read chapter: When people cannot see the data, has their access really been removed?

Imagine an organization that keeps sensitive training data in a separate environment. The development team has no database account, real files are not copied to staff computers, and training runs through an automated pipeline. At first glance, the problem seems solved: people cannot see the data, and machines do the necessary work.

But what if a developer can change the training program and place a few records in an error report? What if the team may download the model, and that model reveals information from its training data? And if an infrastructure administrator can read the execution environment’s memory or obtain its decryption key, what has removing a database account actually guaranteed?

An earlier note on data confidentiality and authorized processing (in Persian) distinguished the right to process information from the right to receive the raw data. Making that distinction real is an architectural problem: a particular computation must be possible without giving its operator information beyond the permission granted. This article examines the conditions needed to do that.

Removing people from the data path is an architectural goal

In this collection’s formulation of Zero Trust AI, an important goal is to design and automate data workflows so that routine execution does not require people to inspect, move or manipulate raw sensitive data. People define the problem and its permitted uses, build the process and examine evidence of its performance. Sensitive data stays in a controlled path.

Level 4 of the ZTAI maturity model introduces this goal through the separation of development and production. The next question is which powers remain after separation, and whether combining them recreates access to the data.

This is the architectural approach discussed in this collection, not the definition of an independently approved ZTAI standard. NIST’s zero trust architecture already covers resources, services and workflows. The emphasis here is on redesigning a process to remove unnecessary direct human involvement with sensitive data.

The claim also needs a stated threat model. Does it restrict developers, infrastructure operators, cloud providers or some combination of them? Does it consider collusion between roles? Which hardware and software components remain trusted? Without those assumptions, “no human access” is too broad to evaluate.

The ability to change code can recreate the ability to read

A program authorized to process raw data necessarily receives some access to it. If a developer can change that program and receive unrestricted results, the developer can potentially instruct it to place the data in an output file. The database account is absent, but an indirect reading mechanism exists.

The same problem can arise without malicious intent. Debugging may record real inputs, error handlers may include a failing record, and telemetry may send samples to a system that the development team can inspect.

Direct viewing is blocked, but a developer can change an authorized program and receive a report containing sensitive data
Removing read permission is effective only if the combination of code changes and output access cannot recreate the same access through another route.

A processing request therefore needs more than a program name or the requester’s identity. It should specify the code version, authorized data, operation, output destination, validity period and resource limits. Permission to train on one dataset must not become permission to execute any program with any output against it.

Signatures and provenance records are necessary evidence, but they do not answer every question. A signature establishes the identity and integrity of an admitted package; it does not prove good behavior. Security scanning and review contribute to admission evidence. For arbitrary general-purpose code, a few preliminary checks cannot establish that every disclosure path is closed. Runtime restrictions and output controls must remain effective after admission.

Separate three kinds of authority

Separation of duties means more than writing three names in a table. Three powers need to be independently enforced: defining the processing, admitting it for execution and authorizing release of its results.

The development team can construct a proposed computation. An admission authority checks whether its specific version fits the task, environment and restrictions. A release authority decides which result may reach which recipient, and at what level of detail. Routine decisions can be automated; independence does not require three manual approvals for every job.

The processing workload must not be able to change its own governing policy, add another output destination or stop the recording of evidence. Likewise, the authority to change processing code must not also allow the developer to bypass output controls. If one administrator can still change all those restrictions, trust in that administrator remains part of the architecture and must be stated explicitly.

A policy engine is part of this problem. Deploying it as a separate service does not establish independence. Its policy-change authority, signing key, deployment path and administrator accounts all matter. A control is independent only to the extent that the component it controls cannot redefine it.

Versioning and reproducibility in MLOps provide the foundation. Without knowing which code version ran under which policy against which data, the separation of powers cannot be meaningfully examined.

The output boundary crosses every pipeline

The maturity model describes boundaries around data preparation, training, evaluation and deployment. Output control must run across all four. A data-cleaning report, evaluation metric and model file can each leave the confidential environment; each needs an appropriate release rule.

Blocking internet connectivity is not enough. If reports go to an internal experiment dashboard visible to the development team, information can cross the intended boundary there. Confidentiality depends on what the recipient is authorized to receive, not simply on whether the destination is internal.

Output pathWhat it might revealAppropriate controls
Error reports and logsReal inputs, identifiers or record contentsRestricted schemas, removal of sensitive content before recording and access limits
Temporary files and debug artifactsData fragments or process memoryRetention within the protected boundary and prevention of automatic download
Metrics and statistical reportsAn individual’s characteristics, a small group’s data or sensitive organizational conditionsLimits on detail, combined-output assessment and repeated-request controls
Weights, adapters and checkpointsInformation extractable from a model, or data deliberately inserted into a fileStructural checks, disclosure assessment and separate model-release authorization
Embeddings and retrieval indexesInformation derived from sensitive documentsInherited classification and permissions, user separation and download controls
Synthetic dataReproduction or inference of information from the source datasetAssessment of the generation method and disclosure risk before release
Backups and snapshotsCopies of data, secrets or captured memoryEncryption, key separation and policy enforcement during restoration

These paths do not all have the same risk, and their existence does not mean disclosure has occurred. The table helps complete the threat model. Removing names or changing a file format is not, by itself, permission to cross a boundary.

In a conservative design, a workload cannot send an arbitrary file to a recipient. It submits a defined result to an independent component with release authority. That component may check output type and size, value ranges, permitted detail and request history. As processing freedom and output diversity increase, demonstrating the adequacy of that control becomes harder.

The release component handles information that has not yet been approved for disclosure. Its own execution environment and connection to the workload must therefore receive protection appropriate to that sensitivity. Sending decrypted information from a confidential environment to an external filter visible to the host administrator defeats the original protection goal.

Word filters, regular expressions and sensitive-data classifiers provide only part of the defense. Code can encode information as numbers, split it across small outputs or disguise it as innocuous content. Highly sensitive workloads may therefore need limits on what can be computed and returned. In some cases, approved operations are a better fit than arbitrary submitted code.

Releasing a model is a separate decision from training it

Permission to train on confidential data does not decide whether someone may download the weights. An organization may permit the trained model to run as a service inside the environment while prohibiting export of its artifact. The service’s responses still need control; keeping weights inside does not automatically prevent disclosure through inference.

Two risks need to be distinguished. An untrusted program may deliberately insert information into an output file. Separately, legitimate training can produce a model that retains information about its training examples and reveals it under some conditions. File-format checks only partly constrain the first risk. Behavioral disclosure assessment addresses the second.

SACRO-ML provides tools for assessing model disclosure risk before and after training in trusted research environments. This approach adds evidence to a release decision. Failure of the tested attacks is not a mathematical guarantee of non-disclosure, and a tool’s coverage should not be generalized to every model architecture.

Synthetic data needs the same care. Information generated from a sensitive dataset may reveal information about that dataset. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees, published in March 2025, discusses the limits of such guarantees. In suitable applications, differential privacy can limit the influence of one person or other defined protected unit on released results. Its guarantee depends on that unit, the parameters and the implementation; it does not cover every operational secret an organization holds.

Small results can reveal a great deal together

Output control cannot assess every request as though it were the first. Suppose two precise, permitted reports return sums for groups that differ by one member. Subtracting the results can reveal that member’s value, even though neither report contains a name or a raw record.

Authorization therefore needs to consider request history, group overlap, repetition and combinations of released information. Minimum group sizes and request limits can contribute to protection, but they do not provide a universal guarantee. Differentially private systems also need cumulative privacy accounting across releases, rather than restarting the budget with each request; see NIST SP 800-226.

This has an architectural consequence. The release authority needs the state required to assess related requests. If every short-lived job applies policy without knowledge of earlier jobs, a restriction that works within one execution may fail across the sequence.

When the infrastructure administrator must also be excluded

Removing development-team access leaves the host administrator question unresolved. In many conventional systems, sufficient control over the operating system or virtualization layer permits interference with processes and memory. Disk encryption alone does not solve that problem: ordinary processing requires decryption somewhere.

As discussed in Containers are not independent security boundaries (in Persian), isolating an application from its development environment is different from protecting it against the host administrator. If that administrator is included in the threat model, the execution environment must support the assumption.

Confidential computing can provide part of this protection through hardware-backed trusted execution environments, or TEEs. The general pattern keeps data and models encrypted until execution-environment evidence is accepted, then releases a key to an environment that satisfies admission policy. NVIDIA’s confidential computing guidance describes the combination of confidential CPU and GPU environments, attestation and conditional key release.

This changes the trust boundary; it does not remove every trusted component. Protection against the host administrator does not necessarily constrain an administrator inside the guest or an authorized malicious program within the protected environment. Hardware and software vulnerabilities, side channels and denial of service must be considered according to the technology selected. Adding “TEE” to a diagram does not settle them.

For an AI workload, protection may need to extend from CPU memory to GPU memory and the transfer path between them. Chip support alone is insufficient: the server, firmware, drivers and deployment mode must work together. That connection is explored in AI infrastructure security from the kernel to the GPU. The English GPU server platform overview provides the broader hardware context.

Attestation must drive an enforcement decision

Attestation helps restrict access when its result changes an effective decision—for example, whether a specific execution receives a decryption key. A report that is only archived does not prevent an unacceptable environment from processing data.

Controlled processing architecture with code admission, confidential execution, evidence-based key release and an independently governed protected output gate
A conceptual design: code, environment, keys and outputs each have acceptance conditions. Output controls cover logs, files, models and service responses.

In the proposed design, evidence must relate to the intended environment and execution, and its freshness must be checked. Policy defines acceptable measurements and the identity and protected channel to which the key is delivered. Changes to the reference policy must also be controlled. Otherwise, the person who changes the environment can simply declare its new state acceptable.

Protection must continue after key delivery. If the process can load new code, alter admitted files or transmit its key, acceptance of its initial state is insufficient. Conversely, refusing future key requests does not necessarily erase a key already delivered or data already decrypted. Termination policy, credential lifetime and cleanup must reflect that fact.

Attestation has a limited scope: measuring a component does not establish the semantic correctness of every computation. The related note on configuration drift, attestation and rollback (in Persian) develops this distinction. Valid evidence needs a reference state, an enforcement decision and a practical response when conditions change.

Maintenance must not create a permanent access route

Maintenance exceptions can be more powerful than ordinary operations. Temporary access is opened for debugging, a snapshot is moved elsewhere for investigation, or a memory dump is sent to a contractor. Some of that access may survive after the incident is resolved.

An architecture intended to reduce human access must design diagnostics from the start: structured and restricted logs, failure reproduction with test data, diagnostic execution inside the boundary and recovery from an accepted version. A repair request must not automatically become permission to receive raw data.

Where human inspection is unavoidable, treat it as a bounded exception. Record the problem, the authorized person, the data scope and the expiry. Afterward, restore the restrictions and review the effects of the access. This is more accurate than claiming complete removal of access, and it provides evidence for reducing future exceptions.

Automation must also stop when required conditions are invalid. A system that disables its output controls or falls back to a less protected path to keep running abandons its restrictions at a critical moment. Workloads requiring continuity need a fallback designed and tested in advance with explicit protection limits.

What evidence shows that access has actually been restricted?

An architectural claim should lead to testable questions. The absence of a human database account does not account for every route to information. An audit must examine powers that can be combined and outputs that can be received.

Audit questionEvidence to look forWhat is insufficient on its own
Can a developer run arbitrary code against the data?Admission of a specific version, runtime restrictions and attempted-bypass testsA Git repository or package signature
Can results leave by another path?An inventory of output paths and tests covering logs, files and communicationNo internet connection
Can the workload change its own restrictions?Separation of policy, key and release authority from processingA policy engine in a separate service
Can the host administrator observe data?An explicit threat model and evidence that the protection configuration supports itDisk encryption or container deployment
Can repeated outputs defeat the limit?Related-request assessment and appropriate cumulative accountingIndependent approval of each output
Has debugging created a lasting access path?Exception history, expiry and verified restoration of controlsA support ticket

Tests have limits. Passing a set of scenarios provides evidence about those scenarios, not proof that every possible attack is closed. Timing, output size and error patterns may themselves be communication channels in stricter threat models. The strength of the claim must match what was designed and tested.

Track two measures separately: how much of the workflow is automated, and how much human access has been reduced. A fully automated pipeline may still allow many people to obtain its data, memory or unrestricted outputs. Count direct and indirect paths, privileged powers that bypass controls and the use of exceptions alongside automation metrics.

Keep authorized use of data possible

The purpose is to make authorized processing feasible. Shutting down every computation may prevent disclosure, but it does not meet the development need. Granting new access whenever development becomes difficult also makes the boundary unsustainable. Good development and diagnostic tools are part of the architecture’s feasibility.

OpenSAFELY provides a practical example in this direction: research code goes to the data, development can use dummy data, and researchers receive aggregate results. The example has a defined scope. The project excludes data-centre owners from its no-access claim. It demonstrates development without unrestricted researcher access, rather than every assumption of the stronger architecture discussed here.

In ZTAI, removing human access becomes meaningful when direct viewing, processing changes, information release and infrastructure administration are considered together. People remain responsible for the purpose, permitted authority and accepted risk. Routine execution should continue within those limits without repeatedly returning to raw data.

The final test is not only who can see the data today. It is who can obtain information after a code change, during a failure and when receiving a result—and which control preserves the intended limit in each case.

Return to the ZTAI technical collection or review the principles and controls behind these boundaries.

Open article in a new tab
6

ZTAI for autonomous agents: bounded authority throughout execution

Task permissions, delegation, retrieval, memory and tools, with independent evaluation and effective revocation.

Read chapter: ZTAI for autonomous agents: bounded authority throughout execution

An agent is assigned to review a factory’s maintenance reports and create inspection requests for high-risk equipment. In one report it encounters a sentence instructing it to send the records to an external address to complete the analysis. The model might treat the sentence as a valid instruction, or it might disregard it. The security architecture cannot leave the entire burden of protection to that judgment. A more important question is: even if the agent tries, does it have the tool, credential and route needed to do it?

“When people cannot see the data, has their access really been removed?” examined how access can be reconstructed through code changes, outputs and infrastructure administration. “Data and model engineering without viewing confidential data” (in Persian) described how engineering can continue through data contracts, protected execution and authorized evidence. Now there is another component: a system that does more than follow a fixed pipeline. During execution, it decides what to read, which tool to invoke and what to delegate to another agent.

In the process-oriented formulation used by this ZTAI collection, Zero Trust AI means redesigning and automating data workflows to eliminate the need for direct human access to, and intervention in, confidential data. People remain policy setters, designers and reviewers of evidence. This is the collection’s architectural goal, not a claim that a separate standard exists under this exact name and definition. Delegating execution to an agent serves that goal only if it does not open another route to unrestricted observation or modification of the data.

Authority removed from people should not be transferred wholesale to an agent. Work that previously required broad access must become a set of limited operations that can be controlled and evaluated.

The agent proposes; the environment authorizes execution

The distinction between a coding assistant and a programming agent (in Persian) illustrates the boundary between producing an answer and making a change. A model can generate text resembling a database command; the runtime turns it into a real operation. Policy belongs at that boundary: model output proposes an action but does not authorize it.

This is consistent with NIST SP 800-207, published in August 2020: being on an internal network or belonging to an organization does not by itself establish trust. The architecture proposed here applies that principle to agent identity, tools and the state of each task. What follows is a design interpretation for this application, not a ready-made blueprint quoted from the standard.

The model and planner may choose their path flexibly, but they must not set the limits of their own authority. The policy engine, credential issuer, tool executor and evidence system must be separated so that an agent cannot remove a restriction by editing a configuration file. Running two separate services is not enough if the agent’s credentials can modify both.

The model itself must also run in an authorized processing environment. If confidential data is sent to an unauthorized external API, controls on later tool calls do not solve the problem. Here we assume that inference and supporting components, including logging and monitoring, comply with the data policy. If the host administrator is part of the threat model, protection requires kernel, GPU and runtime security controls. Restricting the agent does not replace them.

Separate identity, task-specific authority

A shared account named “enterprise assistant” makes auditing difficult. We need to identify which agent, program and model version acted, in which execution, and under whose authority. Service identity, task-run identity and requester identity are different concepts. A permanent account for every run is unnecessary, but credentials and evidence must preserve the distinction.

Task authority is more than a list of tool names. “Access to the maintenance system” is too broad for the maintenance agent. Its assignment might permit reading reports within a defined scope, analyzing them in a protected environment and creating a limited number of draft inspection requests. It need not permit equipment reconfiguration, external messaging or purchase orders.

The execution contract must translate the permitted purpose into enforceable constraints: resources, operations, output destinations, time window, spending and action limits, stopping conditions and delegation rights. A prompt saying “for maintenance only” cannot make linguistic intent detection the sole security control.

A subtle distinction matters in ZTAI. A user may be authorized to request a statistical analysis without being allowed to see its input records. The processing service then accesses the data through its own independent, limited authorization, while the user receives only the permitted output. In a personal document assistant, by contrast, the agent must not exceed the user’s access. These patterns cannot be collapsed into “always use the user’s permissions.”

In both cases, effective authority is constrained by organizational policy, task authorization, service authority, resource policy and the current execution state. Permission to process, permission to see the result and permission to act on it must be defined separately.

Delegation must not manufacture permissions

A lead agent may delegate text analysis, document retrieval and inventory checks. Giving every child the parent’s full credentials only multiplies the holders of broad authority.

A child should receive only the necessary subset of authority that the parent is allowed to delegate. Delegation depth, credential lifetime, audience and resources must be specified. Children’s combined budgets must not exceed the task budget: creating ten agents must not turn a ten-request ceiling into one hundred requests. Shared accounting outside the agents’ control must enforce the limits.

RFC 8693, OAuth 2.0 Token Exchange, distinguishes delegation that retains the actor’s identity from impersonation and can represent a delegation chain. It does not, by itself, guarantee reduced authority or automatic revocation of every derived token. Issuance policy and revocation propagation remain implementation responsibilities.

Cancelling a task must therefore affect every branch: no new tokens, renewed checks on queued requests, and no new execution by a child using an earlier credential. Short-lived credentials reduce the exposure window but do not replace revocation controls. Sensitive operations need authorization revalidated close to the point where they take effect.

Every operation is controlled outside the model

The tool-execution gateway must check identity, task, operation, resource, parameters and current state. An allowed tool with forbidden parameters remains dangerous. Permission to read one file is not permission to read any path; creating a draft does not authorize sending it to any recipient.

The tools themselves should be narrow. “Create an inspection-request draft” can expose a clear input schema and fixed destination. A general shell or arbitrary SQL gives the same task a much larger scope. When code execution is needed, file, network, process and resource restrictions must be enforced by the runtime. Leaving a tool out of the model’s list does not prevent generated code from reaching the same capability.

Checks must precede effects. Inspecting output after an email or money transfer does not undo the action. For sensitive operations, first prepare an action plan and bind resource identifiers and state versions. At final commit, recheck authorization and preconditions. If the recipient, amount or document version changes, the earlier approval is no longer valid. This reduces the time-of-check/time-of-use gap; distributed systems also need a clear commit point and an explicit statement of what that point guarantees.

The following boundaries turn the design into testable acceptance criteria, rather than model settings alone.

Execution boundaryControl outside the modelRequired failure test
Task startSeparate identity, valid contract, restricted credentialReject the same request under an expired task
DelegationNarrower scope, depth limit, shared budgetA child cannot create authority or extra budget
RetrievalResource and chunk authorization before context entryA similar but unauthorized document never reaches an unauthorized model or reranker
Memory and cacheProvenance, permission version, use-time checksCached answers become unusable after permission revocation
Tool callOperation, parameter, destination and precondition validationAn allowed tool cannot act on a forbidden identifier or destination
Information releaseContent, recipient and related-release controlsSplitting output across requests does not bypass the limit
Consumption and effectsAtomic budget reservation across the task treeParallel calls, retries and new agents cannot exceed the ceiling
StopBlock new operations, cancel queues, contain running workDetached children and delayed messages cannot revive the task
Evaluation and feedbackIndependent criteria, trusted provenance, separate acceptance pathThe agent cannot label its own result as correct or authorize its release

In RAG, permissions must stay with the document

An enterprise document assistant (in Persian) typically searches, retrieves chunks, reranks, assembles context and generates a response. If an unauthorized chunk has already reached a model or reranker outside the permitted boundary, removing it from the final answer does not undo that disclosure. Controls must precede each unauthorized disclosure in the chain. If a search engine needs to inspect a broader candidate set inside the secure boundary, that processing requires its own authorization.

The Azure AI Search document-level access-control documentation distinguishes maintaining permission metadata from enforcing it at query time. Some native features use the 2026-08-01-preview API. Query-time enforcement compares the user’s permissions with metadata stored in the index. Until a source permission change reaches that index, the check uses old state. Permission synchronization is part of effective revocation latency.

In this design, identity and access filters come from trusted execution context, not a username or group string supplied by the model. Missing permission metadata must not mean “public.” Titles, result counts, download links and even the existence of a file may be sensitive.

A synthesized answer creates no new permission. Summarizing confidential documents does not automatically remove their confidentiality. If only a limited statistic may be released, the transformation must follow an approved release path; the model’s claim that it has “anonymized” the data is insufficient.

Memory must not turn yesterday’s access into today’s entitlement

An agent might legitimately read a document yesterday and lose permission to use it today. The issue extends beyond answer caches: conversation summaries, persistent memory, temporary files, checkpoints, active context and some inference caches may retain its information.

Derived memory should preserve the provenance and dependencies needed to enforce policy. Cache keys must account for the tenant boundary, access context and relevant policy/data versions. Sharing cached output between users is acceptable only when their authorization for that output is equivalent. Identical questions do not establish that equivalence.

Permission changes require both invalidation of known derivatives and checks when they are used. Background cleanup alone may miss copies. If a revoked document influenced active context, continuing the session may be invalid. Sensitive applications should stop that run and rebuild clean context from authorized sources, rather than merely remove the document from a citation list.

Not every derivative can be traced to one document. A summary without reliable provenance may need to be discarded entirely. Keeping less memory for less time simplifies revocation. Persistent memory should not be every agent’s default.

Revocation cannot erase the past. Changing an ACL does not retrieve information already delivered to an authorized person or external system. Removing a document from an index also does not prove its influence has been removed from trained model weights. Protection claims and retention policy must acknowledge those limits.

MCP standardizes connectivity, not trust

MCP gives agents a common interface to tools and resources. Successfully connecting to a server does not authorize every operation it exposes, and a tool’s name or description is not a security credential.

The Authorization section of the MCP specification dated 28 July 2026 covers HTTP-based transport and requires attention to token validity for the intended server. It should not be indiscriminately applied to local stdio execution. The official Security Best Practices also discusses token passthrough to downstream services, confused-deputy risks and local execution with excessive authority.

Server identity and tool versions need a controlled acceptance path. Changes to the tool list or parameter semantics must not automatically expand an active task’s permissions. Labels such as “read-only” cannot replace enforcement. The Tools specification says tool annotations should not be assumed trustworthy unless they come from a trusted server.

Prompt injection also arrives through documents, tool descriptions, errors, web results and saved memory. Received content must be distinguished from trusted policy, but textual delimiters and the model’s judgment are not a complete defense. Even Anthropic’s report on prompt-injection defenses, which describes improved browser-agent resistance, does not present the problem as solved.

The practical assumption is that the model will sometimes be deceived. Even then, the maintenance agent must not be able to create a new destination, build an exfiltration tool or expand its authority. A tool’s “more permission required” response does not authorize the agent to acquire that permission automatically. An increase must pass through policy and a competent authority; otherwise the task stops within its existing limits.

Budgets must limit effects, not just tokens

An agent with legitimate data access can still consume resources or make permitted but harmful changes. Token and time limits do not necessarily limit purchase requests, message recipients, modified rows or total spending.

Each task needs a compute budget and an effect budget. The first covers execution time, calls, memory and processing cost. The second covers mutable objects, contacts, released data and financial commitments. The scope of the effect matters too: ten internal drafts are not equivalent to ten official letters sent.

Accounting must be shared across the task and all its branches. Reserve capacity before parallel execution so that simultaneous requests cannot all spend the same remaining balance. Retries must depend on the previous outcome and use an idempotency key where the operation supports it. A timeout does not prove that an external operation failed.

Small releases can collectively reveal substantial information. Disclosure limits and checks on related requests must not reset with every new session. Depending on the threat, this accounting may extend beyond one task to related agents or requesters.

Stop, revoke and undo are different capabilities

Turning off the planner does not necessarily stop the work. A tool may have started a long-running job, a message may be queued, a child may continue independently, or a later retry may reactivate an operation.

The stop path must lie outside the agent’s authority and cover the whole execution tree. First, block admission of new operations. Then revoke or suspend permissions and queues, and contain running work according to each tool’s contract. Finally, reconcile actual effects with the destination service: what completed, what was rejected and what remains uncertain. A task-generation identifier or equivalent mechanism can prevent old requests from executing after a restart.

Undo is not always possible. Drafts can be deleted and some data changes can be reversed with concurrency controls. A sent email, disclosed information or physical effect on machinery cannot be undone by restoring the agent system’s snapshot. These cases require compensating actions and an incident process, with stronger controls before irreversible effects where possible.

In an industrial setting, a safe stop does not necessarily mean instantly cutting off every component. Equipment may need an orderly shutdown sequence. A language-model agent must not bypass independent safety interlocks or industrial controls. The process owner must define the safe state beforehand, not leave the model to invent it during a crisis.

If an essential policy or evidence service becomes unavailable, sensitive operations cannot continue uncontrolled. Continuity arrangements need a predefined fallback with limited authority and duration. Keeping work moving must not become a permanent exemption from the protection boundary.

Review evidence without exposing all the data

Removing routine data inspection does not remove human responsibility. A reviewer must be able to examine the task, policy version, authority, rejected events, resource use and operation outcomes. That usually does not require putting complete prompts, retrieved documents and raw tool responses into a general-purpose dashboard.

Audit logs are themselves data outputs. This design records controlled identifiers, reason codes, component versions and minimal evidence. Sensitive content stays within its authorized domain under an appropriate retention policy. Even a plain hash of a low-entropy value may be identifiable by guessing; hashing does not replace a confidentiality design for logs.

Human approval must be based on the precise proposed effect and authorized evidence, not just the agent’s persuasive explanation. If a responsible judgment genuinely needs limited data inspection, treat it as an exception: identify the subject, authorized person, scope, duration and end of access. A process that still depends on such inspection cannot accurately claim to have eliminated human intervention entirely.

Independent evaluation: the agent does not judge its own success

“Task completed successfully” is not sufficient evidence. Quality criteria must connect to externally verifiable state: was the right request created without duplication? Did an unauthorized source enter the context? Did an effect occur after a stop? Was the result delivered only to an authorized recipient?

Evaluation is independent when the acting agent cannot change the criteria, test data, recorded result or release authorization. A second model can help, but two models are not necessarily independent of the same poisoned input or shared error. Deterministic tests, actual tool-state checks and policy controls must accompany semantic judgment.

AgentDojo, version 3 published on 24 November 2024, provides an environment for evaluating tool-using agents exposed to untrusted data. Its useful lesson here is to measure both task completion and attack resistance. A system that rejects everything may avoid unauthorized disclosure while failing its purpose.

Acceptance tests should cover prompt injection, mid-task ACL changes, stale caches, changed tools, parent revocation while children are active, retries after timeouts and policy-service failure. Results need claims of appropriate scope: passing the available attacks does not prove universal immunity.

Feedback is untrusted input to the next cycle

When agent executions feed future training or memory, another loop appears. An agent that calls its own work correct must not turn that claim into a ground-truth label. User feedback also needs provenance and context; one user or coordinated accounts can distort it.

In the proposed path, execution events are recorded with provenance and versions, then validated and quarantined before acceptance. Only accepted data reaches persistent memory, retrieval indexes or training sets. Writing memory is itself an authorized operation. A tool must not turn a sentence about acquiring more authority into an agent’s permanent rule.

Generated outputs, human feedback and independent evaluation results need distinct labels. A held-out test set must not return to training after every cycle. Changes to models, prompts or indexes should first be evaluated within a limited scope, with promotion to production governed by independent criteria. Automation does not conflict with independence; the producer simply must not own its acceptance criteria.

The NIST AI 100-2 E2025 adversarial machine learning taxonomy, published in March 2025, can help distinguish attack origins, affected lifecycle stages and attacker capabilities. It is not a reason to label every incorrect piece of feedback “poisoning” without examining it.

From the maintenance scenario to acceptance criteria

Return to the opening assignment. In a hypothetical implementation, the agent may process the previous thirty days of reports for one production line inside an authorized environment and create at most five draft inspection requests. It cannot send data outside, buy parts or change equipment settings.

If a retrieved report instructs it to export data, a detection layer can flag the instruction. Even if the model fails, the tool gateway rejects the external destination. Delegated children draw from the same five-action budget. If access to a source is revoked, dependent derivatives are excluded and analysis resumes only from clean, authorized context. Final submission also checks the current asset state and task authorization.

A reviewer can inspect draft counts, reason categories, rejected actions and stop status without reading every confidential report. An independent evaluator checks that the drafts actually exist and no out-of-scope action occurred. If detection quality is inadequate, improvement begins with controlled evidence, not unrestricted delivery of all data to the developer.

This is the connection to ZTAI: retain the ability to perform the work while reducing direct human observation and intervention, and keep processing authority limited as well. Without adequate diagnosis, debugging and evaluation, the architecture will revert to broad human access at its first serious failure.

What do recent developments support?

The OWASP Top 10 for Agentic Applications 2026, published on 9 December 2025, treats the security of planning and acting agents as a distinct subject. On 1 September 2026, the OWASP Agent Control Standard, or ACS, page also appeared, focusing on control hooks and runtime policy enforcement. These developments show technical attention to agent control. They do not certify this process-oriented definition of ZTAI or guarantee a product’s security.

Together with MCP specifications and retrieval access-control features, they point toward tool connectivity accompanied by enforceable, testable authority boundaries. A feature in a document, extension or preview API is still not equivalent to a complete implementation in our own system. Actual versions, configuration and behavior must be tested.

Automation must not mean unbounded authority

Autonomous agents help ZTAI when they reduce a process’s dependence on direct human inspection and manipulation without reconstructing that access through tools, memory or outputs. Completed-task counts and staffing reductions are not sufficient measures of success.

Measure outcome quality, reduced need to view data, human exceptions, unauthorized effects, effective revocation and stop latency, and the cost of maintaining the controls. Authority and responsibility belong together, but that does not require giving the executor unlimited power.

The final test is what happens when the agent makes a mistake, the data changes or permission is withdrawn. Can the system still enforce its limits? The answer must come from execution evidence, not the model’s promise to follow instructions.

Open article in a new tab