CWE-1426: Improper Validation of Generative AI Output
Weakness ID: 1426
Vulnerability Mapping:
DISCOURAGEDThis CWE ID should not be used to map to real-world vulnerabilities Abstraction: BaseBase - a weakness that is still mostly independent of a resource or technology, but with sufficient details to provide specific methods for detection and prevention. Base level weaknesses typically describe issues in terms of 2 or 3 of the following dimensions: behavior, property, technology, language, and resource.
View customized information:
For users who are interested in more notional aspects of a weakness. Example: educators, technical writers, and project/program managers.For users who are concerned with the practical application and details about the nature of a weakness and how to prevent it from happening. Example: tool developers, security researchers, pen-testers, incident response analysts.For users who are mapping an issue to CWE/CAPEC IDs, i.e., finding the most appropriate CWE for a specific issue (e.g., a CVE record). Example: tool developers, security researchers.For users who wish to see all available information for the CWE/CAPEC entry.For users who want to customize what details are displayed.
×
Edit Custom Filter
Description
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
Common Consequences
This table specifies different individual consequences associated with the weakness. The Scope identifies the application security area that is violated, while the Impact describes the negative technical impact that arises if an adversary succeeds in exploiting this weakness. The Likelihood provides information about how likely the specific consequence is expected to be seen relative to the other consequences in the list. For example, there may be high likelihood that a weakness will be exploited to achieve a certain impact, but a low likelihood that it will be exploited to achieve a different impact.
Scope
Impact
Likelihood
Integrity
Technical Impact: Execute Unauthorized Code or Commands; Varies by Context
In an agent-oriented setting, output could be used to cause unpredictable agent invocation, i.e., to control or influence agents that might be invoked from the output. The impact varies depending on the access that is granted to the tools, such as creating a database or writing files.
Potential Mitigations
Phase: Architecture and Design
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
Phase: Operation
Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
Phase: Operation
Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
Phase: Build and Compilation
During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
Relationships
This table shows the weaknesses and high level categories that are related to this weakness. These relationships are defined as ChildOf, ParentOf, MemberOf and give insight to similar items that may exist at higher and lower levels of abstraction. In addition, relationships such as PeerOf and CanAlsoBe are defined to show similar weaknesses that the user may want to explore.
Relevant to the view "Research Concepts" (CWE-1000)
Nature
Type
ID
Name
ChildOf
Pillar - a weakness that is the most abstract type of weakness and represents a theme for all class/base/variant weaknesses related to it. A Pillar is different from a Category as a Pillar is still technically a type of weakness that describes a mistake, while a Category represents a common characteristic used to group related things.
The different Modes of Introduction provide information about how and when this weakness may be introduced. The Phase identifies a point in the life cycle at which introduction may occur, while the Note provides a typical scenario related to introduction during the given phase.
Phase
Note
Architecture and Design
Developers may rely heavily on protection mechanisms such as input filtering and model alignment, assuming they are more effective than they actually are.
Implementation
Developers may rely heavily on protection mechanisms such as input filtering and model alignment, assuming they are more effective than they actually are.
Applicable Platforms
This listing shows possible areas for which the given weakness could appear. These may be for specific named Languages, Operating Systems, Architectures, Paradigms, Technologies, or a class of such platforms. The platform is listed along with how frequently the given weakness appears for that instance.
Languages
Class: Not Language-Specific (Undetermined Prevalence)
Architectures
Class: Not Architecture-Specific (Undetermined Prevalence)
Technologies
AI/ML (Undetermined Prevalence)
Class: Not Technology-Specific (Undetermined Prevalence)
chain: GUI for ChatGPT API performs input validation but does not properly "sanitize" or validate model output data (CWE-1426), leading to XSS (CWE-79).
Detection Methods
Dynamic Analysis with Manual Results Interpretation
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Dynamic Analysis with Automated Results Interpretation
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Architecture or Design Review
Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
Memberships
This MemberOf Relationships table shows additional CWE Categories and Views that reference this weakness as a member. This information is often useful in understanding where a weakness fits within the context of external information sources.
Nature
Type
ID
Name
MemberOf
Category - a CWE entry that contains a set of other entries that share a common characteristic.
(this CWE ID should not be used to map to real-world vulnerabilities)
Reasons: Potential Major Changes, Frequent Misinterpretation
Rationale:
There is potential for this CWE entry to be modified in the future for further clarification as the research community continues to better understand weaknesses in this domain.
Comments:
This CWE entry is only related to "validation" of output and might be used mistakenly for other kinds of output-related weaknesses. Careful attention should be paid to whether this CWE should be used for vulnerabilities related to "prompt injection," which is an attack that works against many different weaknesses. See Maintenance Notes and Research Gaps. Analysts should closely investigate the root cause to ensure it is not ultimately due to other well-known weaknesses. The following suggestions are not comprehensive.
Command Injection. Use this CWE for most cases of 'prompt injection' attacks in which additional prompts are added to input to, or output from, the model. If OS command injection, consider CWE-78.
Improper Encoding or Escaping of Output. Use this CWE when the product is expected to encode or escape genAI outputs.
Notes
Research Gap
This entry is related to AI/ML, which is not well
understood from a weakness perspective. Typically, for
new/emerging technologies including AI/ML, early
vulnerability discovery and research does not focus on
root cause analysis (i.e., weakness identification). For
AI/ML, the recent focus has been on attacks and
exploitation methods, technical impacts, and mitigations.
As a result, closer research or focused efforts by SMEs
is necessary to understand the underlying weaknesses.
Diverse and dynamic terminology and rapidly-evolving
technology further complicate understanding. Finally,
there might not be enough real-world examples with
sufficient details from which weakness patterns may be
discovered. For example, many real-world vulnerabilities
related to "prompt injection" appear to be related to
typical injection-style attacks in which the only
difference is that the "input" to the vulnerable
component comes from model output instead of direct
adversary input, similar to "second-order SQL injection"
attacks.
Maintenance
This entry was created by members
of the CWE AI Working Group during June and July 2024. The
CWE Project Lead, CWE Technical Lead, AI WG co-chairs, and
many WG members decided that for purposes of timeliness, it
would be more helpful to the CWE community to publish the
new entry in CWE 4.15 quickly and add to it in subsequent
versions.
[REF-1443] Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien
and Jonathan Cohen. "NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails". 2023-12.
<https://aclanthology.org/2023.emnlp-demo.40/>. URL validated: 2024-07-11.
[REF-1445] Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan
and Xiaowei Huang. "Building Guardrails for Large Language Models". 2024-05-29.
<https://arxiv.org/pdf/2402.01822>. URL validated: 2024-07-11.
Content History
Submissions
Submission Date
Submitter
Organization
2024-07-02 (CWE 4.15, 2024-07-16)
Members of the CWE AI WG
CWE Artificial Intelligence (AI) Working Group (WG)
More information is available — Please edit the custom filter or select a different filter.
Description
This table specifies different individual consequences associated with the weakness. The Scope identifies the application security area that is violated, while the Impact describes the negative technical impact that arises if an adversary succeeds in exploiting this weakness. The Likelihood provides information about how likely the specific consequence is expected to be seen relative to the other consequences in the list. For example, there may be high likelihood that a weakness will be exploited to achieve a certain impact, but a low likelihood that it will be exploited to achieve a different impact.
