Cybersecurity, Data, and Design

Heath Nieddu Phd(c), CISSP, MBA, GCIH

AI System Drift: You’ve Changed, ‘Cause I Still Feel the Same

by | May 29, 2026 | AI, Cyber Security

AI system drift presents a communication challenge between CISOs and their stakeholders. Enterprise colleagues are bound to ask, “If you’re so sure that this AI-powered SaaS solution doesn’t use or store any of our data, how does drift occur between vendor releases?” In AI-enabled security tooling, drift is a problem because performance can degrade, operational processes can be disrupted, and it raises questions about whether the vendor is using customer data to enhance the service. Governing these changes aligns with the Measure and Manage functions in the NIST AI Risk Management Framework (AI RMF), which emphasize continuous monitoring, evaluation, and adjustment of AI systems over time.

Identifying drift in AI services becomes an observability goal. The model and services are changing in operational and external environments outside the CISO’s control, but the service’s outputs directly affect the CISO’s areas of responsibility. AI operational observability is part of the solution to this drift challenge. Before entering into that broader IT operations conversation, stakeholders need agreement on definitions.

Non-model Drift: It’s Me, Not You

As data-pipeline inputs change, data drift can occur. This type of drift, called covariate shift, refers to changes in the input data distribution (the environment outside the model) while the data relationships in the model remain stable. In other words, this type of drift is not a result of fundamental retraining of the model. For example, suppose AI-powered models in an IDS solution assign a probability of harm whenever PowerShell is used. Then, everyone in the firm starts using PowerShell legitimately for a new project. When the AI-powered model continues to report a high likelihood of threat or false positives with every use, even though the data distribution has changed, this is an example of covariate drift. This type of drift is distinct from behavioral, or system, drift, discussed later, in which the environment also changes, specifically in how humans interpret and act on model outputs rather than in the inputs themselves.

Another type of drift is concept drift. Sticking with the earlier PowerShell scenario, suppose attackers shift tactics and begin using entirely different tools (e.g., living-off-the-land binaries other than PowerShell). In this case, the original relationship between PowerShell usage and threat likelihood breaks down. The model continues to associate PowerShell with risk, but that relationship is no longer valid. This is concept drift—the underlying pattern the model learned has changed, requiring retraining or redesign rather than simple recalibration.

Both covariate shift and concept drift are referred to as external drift for our purposes because they are caused by changes in the external environment rather than the model itself. These are not examples of models updating themselves at runtime based on data they saved about a customer, but rather of the world changing in ways that affect the performance of comprehensive AI systems.

Now that we’ve explored externally-caused drift, let’s introduce the concept of performance drift. Performance drift is the measurable downstream effects of covariate and concept drift. Returning to the PowerShell scenario, suppose your model initially flags malicious PowerShell activity with high precision. Over time, due to a mix of increased legitimate PowerShell usage (covariate shift) and evolving attacker behavior (concept drift), you observe that precision declines, resulting in more false positives during normal operations and occasional missed detections. The observable decline in detection quality is performance drift. This is more of a symptom or result of some upstream change. This is the drift you can observe with greater accuracy using trends, detection rates, and distribution shifts.

“How is there any drift without retention of our data?”

We have explained so far that much of the observed change in AI user experience is not because the model is necessarily saving enterprise data, but rather due to other external changes in behavior. Most SaaS AI systems don’t learn from your data in real time, don’t persist state across sessions, and certainly don’t update model weights during inference. Vendors are increasingly introducing various types of memory into the AI system stack that wraps the model and gives it stateful characteristics, but this is a topic for a different post. CISOs need to be able to crisply answer questions about the most common types of drift and then shift the conversation to the need for observability capabilities.

A Practical 2×2 Conversation Starter for CISOs

CISO communication often requires translating the latest technical jargon into the simplest language possible to reach a broad audience. To make the vague idea of AI drift more understandable, it helps to separate drift along two dimensions:

  • What is drifting? (Model vs. System)
  • Who controls it? (Vendor vs. You)

Once these basic themes are understood, CISOs can move on to the latest developments in how responsibilities are shared.

What to Do About It

CISOs must pivot the conversation quickly from discussing AI system drift to a risk-based discussion of drift detection and operational performance gains. To make this pivot, CISOs need to demonstrate an expanded definition of drift and a nuanced understanding of the AI system architecture. CISOs can gain this perspective by starting with vendor artifacts such as system cards and release notes, but treating them as inputs to architectural validation. They will not be a cure-all. Developing internal architectures, or, better yet, data flow diagrams, will aid the pivot to data governance.

In the long term, CISOs should invest in observability features. These will initially feel operational, but over time, will allow for a discussion of drift and a greater understanding of potential risk. The fact that these features are arguably operational means there is a greater potential to share administration and costs with IT operations.

Accountability and governance cannot be automated. While many activities conducted by security teams will change over the coming months and years, there will always be a need for accountability when things go wrong. Therefore, governance will also be needed to reduce the likelihood of such events in the future. The theme has stayed the same while the context has changed dramatically.

Resources