Resources
As enterprises increasingly adopt artificial intelligence (AI) and machine learning across business operations, AI systems are becoming part of critical workflows, products, and decision-making processes. This growing adoption also introduces a broader range of AI security risks that traditional cybersecurity controls may not fully address. AI systems depend on training data, models, APIs, pipelines, and inference environments, creating additional points that attackers can target. NIST notes that AI systems can face both conventional cybersecurity risks and AI-specific attacks across different stages of their lifecycle.
Attackers may attempt to manipulate training data, gain access to proprietary models, or interact with model interfaces to reproduce model behaviour. These AI security threats can affect data integrity, intellectual property, model confidentiality, and system reliability. Key examples include AI data poisoning, AI model theft, and AI model extraction attacks. OWASP identifies data poisoning and model theft among the security risks that can affect AI/ML systems, while model extraction can allow attackers to reproduce aspects of a model through inference queries.
For this reason, AI model security should not be treated as a final-stage security check. It needs to be considered throughout the AI lifecycle from data collection and model development to deployment, monitoring, and ongoing updates. Taking this approach can help organisations identify AI model vulnerabilities earlier and build security controls into the systems as they evolve.
• Training-data attacks: Attackers may manipulate or contaminate training, fine-tuning, or embedding data to introduce vulnerabilities, biases, or unwanted model behaviour.
• Model attacks: AI models themselves can be targeted through techniques designed to alter, compromise, or extract information about their behaviour and parameters.
• Inference and API attacks: Attackers can exploit model interfaces and repeated queries to manipulate outputs, probe system behaviour, or extract information about the underlying model. NIST identifies model extraction as a type of attack that can reveal information about model architecture or parameters.
• Unauthorized model access: Weak access controls can expose models, model endpoints, development environments, or other AI assets to users who should not have access.
• Intellectual property risks: AI models, training datasets, prompts, configurations, and other development assets can contain valuable proprietary information. Unauthorized access or extraction can therefore create intellectual property and confidentiality risks.
• Adversarial manipulation: Carefully crafted inputs can cause an AI system to produce incorrect or unintended results. These attacks can occur during inference and are an established category within adversarial machine learning.
These threats are not confined to a single stage. They can affect data collection, training, model development, deployment, inference, and ongoing operation, making AI security a lifecycle-wide responsibility rather than a single security checkpoint.
Understanding How Data Poisoning Attacks Operate
Imagine you have a machine learning model that relies on clean, accurate data to function well. A data poisoning attack disrupts this process in four critical steps: collecting data, manipulating the dataset, training the model, and altering the model's behavior. First, an attacker finds a way to influence the data used for training or fine-tuning. This could be during the initial collection phase or later when the data is being prepared for use. They introduce harmful or distorted samples into the dataset, which is like sneaking a few bad apples into a basket of good ones. When this tainted data is fed into the model during training, the model begins to learn incorrect patterns or relationships. The effects of this tampering might only become evident in specific situations, or they could lead to a general decline in the model's performance. Experts at NIST have labeled this type of interference as a training-stage attack. Essentially, attackers can add or alter samples within the training data, compromising the model's integrity and reliability.
• Unauthorised access to model repositories: Weak permissions or compromised credentials can expose model weights, parameters, or related files.
• Compromised cloud environments: Attackers who gain access to cloud infrastructure may locate and copy stored model assets.
• Exposed model files: Unsecured storage, backups, or deployment files can make models accessible to unauthorised users.
• Insider access: Employees or contractors with excessive access may copy or disclose proprietary model assets.
• Insecure deployment environments: Poorly protected production environments can make deployed models easier to access or copy.
How Model Extraction Attacks Work
Model extraction attacks typically involve an attacker sending queries to an accessible AI system and studying its responses. By collecting and analysing these outputs, the attacker may build an approximation of the target model. The attacker does not necessarily obtain the original model files but may attempt to reproduce aspects of the model’s behaviour through its accessible interface.
AI Model Reverse Engineering
AI model reverse engineering involves analysing a model's observable behaviour to infer characteristics of how it works. When proprietary models are exposed through APIs, repeated querying can create risks to their intellectual property and capabilities. Appropriate access controls, monitoring, rate limiting, and usage restrictions can help reduce the risk of model extraction attacks.
Protect AI Models Protecting AI models requires controls that prevent unauthorized access, copying, or modification. Encryption helps protect model files from unauthorized access. Strong access controls ensure that only authorized users can access models, while secure model repositories protect model files during storage. Identity and access management helps control who can access specific AI resources, and monitoring model access can help detect unusual or unauthorized activity.
Secure AI APIs and Inference Endpoints Securing AI APIs and inference endpoints helps prevent unauthorized access and misuse of AI systems. Authentication verifies user or application identity, while authorization ensures they can access only permitted resources. Rate limiting helps control excessive requests, and query monitoring helps identify unusual usage patterns. Abuse detection can further help detect and respond to suspicious or malicious activity.
Monitor for Suspicious AI Activity Monitoring AI systems can help identify unusual or potentially unauthorized activity. Unusual query patterns and repeated API requests may indicate misuse, while unauthorized model access can signal a security breach. Monitoring unexpected model behaviour can help identify potential issues, and security logging and alerting provide visibility into suspicious activity so organizations can investigate and respond promptly.
Apply AI Security Across the Lifecycle AI security should be maintained throughout the AI lifecycle, not just at deployment. Regular monitoring and security reviews help identify AI security threats, model vulnerabilities, and emerging AI security risks and strengthen AI model security over time.
What Are the Major AI Security Threats
AI security threats can emerge at different points in the development, deployment, and operation of an AI system. Adversarial machine learning (AML) is an umbrella term for attacks that exploit the data-driven nature of machine learning systems, including attacks aimed at manipulating inputs, compromising data, extracting information, or influencing model behaviour. NIST’s AML taxonomy categorises attacks according to factors such as the AI lifecycle stage, attacker objectives, and the capabilities available to an attacker. The major areas of concern include:• Training-data attacks: Attackers may manipulate or contaminate training, fine-tuning, or embedding data to introduce vulnerabilities, biases, or unwanted model behaviour.
• Model attacks: AI models themselves can be targeted through techniques designed to alter, compromise, or extract information about their behaviour and parameters.
• Inference and API attacks: Attackers can exploit model interfaces and repeated queries to manipulate outputs, probe system behaviour, or extract information about the underlying model. NIST identifies model extraction as a type of attack that can reveal information about model architecture or parameters.
• Unauthorized model access: Weak access controls can expose models, model endpoints, development environments, or other AI assets to users who should not have access.
• Intellectual property risks: AI models, training datasets, prompts, configurations, and other development assets can contain valuable proprietary information. Unauthorized access or extraction can therefore create intellectual property and confidentiality risks.
• Adversarial manipulation: Carefully crafted inputs can cause an AI system to produce incorrect or unintended results. These attacks can occur during inference and are an established category within adversarial machine learning.
These threats are not confined to a single stage. They can affect data collection, training, model development, deployment, inference, and ongoing operation, making AI security a lifecycle-wide responsibility rather than a single security checkpoint.
AI Data Poisoning: How Attackers Manipulate Training Data
AI data poisoning is a type of attack in which an attacker deliberately manipulates part of the data used to train, fine-tune, or otherwise update an AI model. The goal is to influence what the model learns, potentially introducing vulnerabilities, biases, incorrect associations, or hidden behaviours. National Institute of Standards and Technology (NIST) defines data poisoning as an attack in which an adversary controls part of the training data, while OWASP notes that poisoning can affect pre-training, fine-tuning, and embedding data.Understanding How Data Poisoning Attacks Operate
Imagine you have a machine learning model that relies on clean, accurate data to function well. A data poisoning attack disrupts this process in four critical steps: collecting data, manipulating the dataset, training the model, and altering the model's behavior. First, an attacker finds a way to influence the data used for training or fine-tuning. This could be during the initial collection phase or later when the data is being prepared for use. They introduce harmful or distorted samples into the dataset, which is like sneaking a few bad apples into a basket of good ones. When this tainted data is fed into the model during training, the model begins to learn incorrect patterns or relationships. The effects of this tampering might only become evident in specific situations, or they could lead to a general decline in the model's performance. Experts at NIST have labeled this type of interference as a training-stage attack. Essentially, attackers can add or alter samples within the training data, compromising the model's integrity and reliability.
AI Model Theft: Protecting Proprietary AI Models
AI models can become valuable intellectual property because organisations invest significantly in model development, training, proprietary data, and engineering. Protecting these models is therefore an important part of AI model security. AI model theft refers to the unauthorised access, copying, or extraction of an AI/ML model or its capabilities. This can involve directly obtaining model files or using repeated queries to reproduce aspects of a model’s behaviour. How Can AI Model Theft Occur? Several security weaknesses can expose proprietary models:• Unauthorised access to model repositories: Weak permissions or compromised credentials can expose model weights, parameters, or related files.
• Compromised cloud environments: Attackers who gain access to cloud infrastructure may locate and copy stored model assets.
• Exposed model files: Unsecured storage, backups, or deployment files can make models accessible to unauthorised users.
• Insider access: Employees or contractors with excessive access may copy or disclose proprietary model assets.
• Insecure deployment environments: Poorly protected production environments can make deployed models easier to access or copy.
AI Model Extraction Attacks: Reconstructing Models Through Queries
A model extraction attack occurs when an attacker repeatedly queries an accessible AI system and studies its outputs to learn how the model behaves. The collected responses can then be used to build an approximation or functional copy of the target model.How Model Extraction Attacks Work
Model extraction attacks typically involve an attacker sending queries to an accessible AI system and studying its responses. By collecting and analysing these outputs, the attacker may build an approximation of the target model. The attacker does not necessarily obtain the original model files but may attempt to reproduce aspects of the model’s behaviour through its accessible interface.
AI Model Reverse Engineering
AI model reverse engineering involves analysing a model's observable behaviour to infer characteristics of how it works. When proprietary models are exposed through APIs, repeated querying can create risks to their intellectual property and capabilities. Appropriate access controls, monitoring, rate limiting, and usage restrictions can help reduce the risk of model extraction attacks.
How Organizations Can Reduce AI Security Risks
Secure AI Training Data Securing AI training data helps protect models from corrupted, unauthorized, or manipulated data. Dataset validation helps identify errors or suspicious data before training. Data provenance tracks where data comes from and how it has been collected or modified. Access controls ensure that only authorized users can access or change training datasets. Data integrity monitoring helps detect unexpected changes, while dataset versioning maintains a record of previous datasets and changes over time.Protect AI Models Protecting AI models requires controls that prevent unauthorized access, copying, or modification. Encryption helps protect model files from unauthorized access. Strong access controls ensure that only authorized users can access models, while secure model repositories protect model files during storage. Identity and access management helps control who can access specific AI resources, and monitoring model access can help detect unusual or unauthorized activity.
Secure AI APIs and Inference Endpoints Securing AI APIs and inference endpoints helps prevent unauthorized access and misuse of AI systems. Authentication verifies user or application identity, while authorization ensures they can access only permitted resources. Rate limiting helps control excessive requests, and query monitoring helps identify unusual usage patterns. Abuse detection can further help detect and respond to suspicious or malicious activity.
Monitor for Suspicious AI Activity Monitoring AI systems can help identify unusual or potentially unauthorized activity. Unusual query patterns and repeated API requests may indicate misuse, while unauthorized model access can signal a security breach. Monitoring unexpected model behaviour can help identify potential issues, and security logging and alerting provide visibility into suspicious activity so organizations can investigate and respond promptly.
Apply AI Security Across the Lifecycle AI security should be maintained throughout the AI lifecycle, not just at deployment. Regular monitoring and security reviews help identify AI security threats, model vulnerabilities, and emerging AI security risks and strengthen AI model security over time.
