CWE

Common Weakness Enumeration

A Community-Developed List of Software Weakness Types

CWE/SANS Top 25 Most Dangerous Software Errors
Home > CWE List > CWE- Individual Dictionary Definition (3.1)  
ID

CWE-1039: Automated Recognition Mechanism with Inadequate Detection or Handling of Adversarial Input Perturbations

Weakness ID: 1039
Abstraction: Class
Structure: Simple
Status: Incomplete
Presentation Filter:
+ Description
The product uses an automated mechanism such as machine learning to recognize complex data inputs (e.g. image or audio) as a particular concept or category, but it does not properly detect or handle inputs that have been modified or constructed in a way that causes the mechanism to detect a different, incorrect concept.
+ Extended Description

When techniques such as machine learning are used to automatically classify input streams, and those classifications are used for security-critical decisions, then any mistake in classification can introduce a vulnerability that allows attackers to cause the product to make the wrong security decision. If the automated mechanism is not developed or "trained" with enough input data, then attackers may be able to craft malicious input that intentionally triggers the incorrect classification.

Targeted technologies include, but are not necessarily limited to:

  • automated speech recognition
  • automated image recognition

For example, an attacker might modify road signs or road surface markings to trick autonomous vehicles into misreading the sign/marking and performing a dangerous action.

+ Relationships

The table(s) below shows the weaknesses and high level categories that are related to this weakness. These relationships are defined as ChildOf, ParentOf, MemberOf and give insight to similar items that may exist at higher and lower levels of abstraction. In addition, relationships such as PeerOf and CanAlsoBe are defined to show similar weaknesses that the user may want to explore.

+ Relevant to the view "Research Concepts" (CWE-1000)
NatureTypeIDName
ChildOfClassClass - a weakness that is described in a very abstract fashion, typically independent of any specific language or technology. More general than a Base weakness.697Incorrect Comparison
ChildOfClassClass - a weakness that is described in a very abstract fashion, typically independent of any specific language or technology. More general than a Base weakness.693Protection Mechanism Failure
+ Relevant to the view "Development Concepts" (CWE-699)
NatureTypeIDName
MemberOfCategoryCategory - a CWE entry that contains a set of other entries that share a common characteristic.19Data Processing Errors
+ Modes Of Introduction

The different Modes of Introduction provide information about how and when this weakness may be introduced. The Phase identifies a point in the software life cycle at which introduction may occur, while the Note provides a typical scenario related to introduction during the given phase.

PhaseNote
Architecture and DesignThis issue can be introduced into the automated algorithm itself.
+ Applicable Platforms
The listings below show possible areas for which the given weakness could appear. These may be for specific named Languages, Operating Systems, Architectures, Paradigms, Technologies, or a class of such platforms. The platform is listed along with how frequently the given weakness appears for that instance.

Languages

Class: Language-Independent (Undetermined Prevalence)

+ Common Consequences

The table below specifies different individual consequences associated with the weakness. The Scope identifies the application security area that is violated, while the Impact describes the negative technical impact that arises if an adversary succeeds in exploiting this weakness. The Likelihood provides information about how likely the specific consequence is expected to be seen relative to the other consequences in the list. For example, there may be high likelihood that a weakness will be exploited to achieve a certain impact, but a low likelihood that it will be exploited to achieve a different impact.

ScopeImpactLikelihood
Integrity

Technical Impact: Bypass Protection Mechanism

When the automated recognition is used in a protection mechanism, an attacker may be able to craft inputs that are misinterpreted in a way that grants excess privileges.
+ Weakness Ordinalities
OrdinalityDescription
Primary
+ Notes

Relationship

Further investigation is needed to determine if better relationships exist or if additional organizational entries need to be created. For example, this issue might be better related to "recognition of input as an incorrect type," which might place it as a sibling of CWE-704 (incorrect type conversion).
+ References
[REF-16] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow and Rob Fergus. "https://arxiv.org/abs/1312.6199". 2014-02-19. <https://arxiv.org/abs/1312.6199>.
[REF-17] OpenAI. "Attacking Machine Learning with Adversarial Examples". 2017-02-24. <https://blog.openai.com/adversarial-example-research/>.
[REF-15] James Vincent. "Magic AI: These are the Optical Illusions that Trick, Fool, and Flummox Computers". The Verge. 2017-04-12. <https://www.theverge.com/2017/4/12/15271874/ai-adversarial-images-fooling-attacks-artificial-intelligence>.
[REF-13] Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang and Carl A. Gunter. "CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition". 2018-01-24. <https://arxiv.org/pdf/1801.08535.pdf>.
[REF-14] Nicholas Carlini and David Wagner. "Audio Adversarial Examples: Targeted Attacks on Speech-to-Text". 2018-01-05. <https://arxiv.org/abs/1801.01944>.
+ Content History
Submissions
Submission DateSubmitterOrganization
2018-03-12CWE Content TeamMITRE

More information is available — Please select a different filter.
Page Last Updated: March 29, 2018