A Comparison of the CWE Development and Research Views
|
| Audience | 7PK | 699 | 1000 |
|---|---|---|---|
| Developers | Yes | Yes | No |
| Tool Vendors | Yes | Yes | No |
| Academic Researchers | No | No | Yes |
The intended audiences for 7PK and CWE-699 are basically the same.
CWE-1000 is not necessarily intuitive to developers. It might not be intuitive to tool vendors either, since the vendors must present results that can be automatically detected and explained in ways that are understandable to developers.
| Metric | 7PK | 699 | 1000 |
|---|---|---|---|
| Total CWE Entries | 97 | 672 | 622 |
| Roots | 8 | 2 | 12 |
| Max Tree Depth | 2 | 9 | 7 |
| Ave # Children | 11.1 | 4.7 | 4.6 |
| Max Children | 28 | 21 | 26 |
| Ave # Parents | 1.0 | 1.1 | 1.2 |
| Max Parents | 1 | 3 | 3 |
Note that the "Total CWE Entries" statistic only counts entries that have ParentOf/ChildOf relationships within the view.
It can be seen that the hierarchies in CWE-699 and CWE-1000 are deeper than the one in Seven Pernicious Kingdoms. This might be reasonable to expect, since CWE tracks many more weaknesses than 7PK does. The 7PK phyla are generally at CWE's Base or Variant level of abstraction. It should be noted that Fortify's [Vulncat] covers many more issues than the original 7PK paper.
A low average number of children was not an explicit goal during development of CWE-1000. However, the CWE team sometimes paid special attention to large categories to see if there were additional patterns that might provide a greater degree of granularity, which had the effect of reducing the average number of children. In addition, creation of a more abstract entry might cause entries in other parts of the hierarchy to be shifted to that abstract entry. However, some areas of CWE-1000 clearly need additional granularity, such as CWE-693 (Protection Mechanism Failure) and CWE-668 (Exposure of Resource to Wrong Sphere), which both have 26 children.
It is interesting to note that while one of 7PK's original goals was to use "seven-plus-one" kingdoms for psychological acceptability to humans, this only applies to the top-level kingdoms; the input validation kingdom contains 28 items. This may be due to the limited depth of the hierarchy. Vulncat, which expands 7PK, adds a top level based on individual programming languages; each level is then broken down by the 7PK categories. The average number of children still appears to be higher than those in CWE.
The hierarchy in 699 seems deeper than it really is, because of the small number of roots at the top. It does not really begin to branch out until a couple levels down into the hierarchy. So, it is fairly equivalent to 1000 in terms of depth.
Both CWE-699 and CWE-1000 average more than 1 parent per node. In the case of CWE-699, this is expected due to the large number of categories in use. For the Research view, while mutual exclusivity is a goal, further work is needed.
Quoting from [Howard], and originating from [Amoroso], taxonomies should have the following properties:
| 1) | mutually exclusive - classifying in one category excludes all others because categories do not overlap, |
| 2) | exhaustive - taken together, the categories include all possibilities, |
| 3) | unambiguous - clear and precise so that classification is not uncertain, regardless of who is classifying, |
| 4) | repeatable - repeated applications result in the same classification, regardless of who is classifying, |
| 5) | accepted - logical and intuitive so that they could become generally approved, |
| 6) | useful - can be used to gain insight into the field of inquiry. |
Each view has different stated and implied goals with respect to the quality and quantity of the elements that it identifies, so it would be informative to examine each view in light of these criteria and related criteria.
| Goal | 7PK | 699 | 1000 |
|---|---|---|---|
| Mutually Exclusive | No | No | Yes |
| Models Inter-Relationships | No | Partial | Yes |
| Exhaustive | No | Yes | Yes |
| Unambiguous | No | No | Yes |
| Unified Perspective | Yes | No | Yes |
| Repeatable | No | No | Yes |
| Accepted | Yes | Yes | No |
| Categories Allowed | Yes | Yes | No |
Additional explanation of these comparisons is below.
The 7PK authors acknowledge that the goal was not to be exhaustive, stating that the kingdoms are only focused on the most important implementation-level coding errors that can also be automatically detected by tools. The CWE-1000 and CWE-699 views include weaknesses found in design, implementation, and other SDLC phases. Even if a weakness cannot be detected automatically, it is included in CWE, so the scope is wider than current tool capabilities.
With respect to weaknesses, CWE-699 and CWE-1000 have the same goals for exhaustiveness. However, some weakness pillars in CWE-1000 are too abstract to have a good fit in CWE-699. With CWE-699's use of categories and CWE-1000's explicit intention to avoid them, CWE-699 is generally going to be the most comprehensive graphical view of CWE, although the CWE-2000 view always lists every entry.
A CWE mapping is regarded as repeatable if multiple people with equivalent technical expertise assign the same CWE identifier to a security issue.
The 7PK paper does not state whether repeatability of mappings is a goal. It probably isn't, since the focus is more on education of developers and automated detection.
For CWE, regardless of the view, the requirements for repeatability are different. Since CWE is intended to facilitate communication of weakness information between diverse communities and tools, repeatability is important - people have to be able to map to the same identifier.
Accordingly, the Research view seeks to achieve repeatability, although strong technical knowledge and an understanding of chains is required. Repeatability has been informally verified in many situations in which two separate CWE team members classified issues the same way. However, there are limits - ambiguous classification can still happen because of perspective issues or incomplete parts of the hierarchy.
In the case of the Development view, it is hoped that the large number of navigation categories will make it easier for people to reach the desired weakness ID. However, with so many categories, there is a risk of incomplete or incorrect mappings if one user follows one path of the tree, and another mapper follows a different path. Guidance will need to be written, or the tree structured in a way that minimizes this risk. Ultimately, the mapping consistency problem in the Development view could be addressed by improving search capabilities, ensuring that all categories are complete, creating additional categories to fill perceived gaps, and conducting usability studies.
By defining a narrow audience and subset of weaknesses, 7PK effectively unifies its perspective. 699 relies heavily on categories, with many different perspectives, to support different types of users. 1000 attempts to achieve a unified perspective, such as the intended omission of all categories and a focus on the core weakness in terms of its behaviors and resources. 1000 tries to omit attacks or consequences, although some consequences are treated as the last link in a chain. It also treats protection mechanism failures (CWE-693) as chaining relationships. For example, insufficient input validation (CWE-20) can lead to many issues such as SQL injection (CWE-89) and XSS (CWE-79), so in CWE-1000, these are chaining relationships instead of parent/child relationships. This approach also has the side effect of allowing modeling of other protection mechanism failures, such as insufficient output sanitization (CWE-116), which is a significant factor in SQL injection and XSS.
It should be noted that while 7PK presents a unified perspective, it is not viewed as unified with respect to 1000's model of the world. For example, 7PK's "input validation" kingdom specifies many issues as children such as SQL injection (CWE-89) and XSS (CWE-79). This practice is also followed in the Development view. However, input validation is treated as a protection mechanism within 1000, which is separated (through chaining relationships) from issues that could be fixed by input validation. Vulncat's perspective is not unified in the 1000 model, either; it has categories such as "Denial of Service" (which are consequences in 1000's model), "Log Forging" (an attack), and "Buffer Overflow: Off-by-One" (a chain).
Some specific consequences (e.g. NULL pointer dereference) are assigned CWE identifiers, but generally as the final link in a chain. Within CWE, a NULL dereference is always treated as the result of some other error, such as the failure to check an error condition. This approach ignores the pathological case in which a programmer directly provides the NULL constant as an argument (perhaps as a way to make a quick exit from an application).
The 7PK paper does not state whether mutually exclusive classification is a goal. The layout does not imply that the same item could have multiple parents. However, there are cases in which items could be classified in multiple ways. Vulncat's language-specific breakdown at the top level reuses the 7PK kingdoms at the next level below the individual languages.
For example, CWE-491 (Public cloneable() Method Without Final (aka 'Object Hijack')) is classified under the "Insufficient Encapsulation" kingdom (CWE-485), but it is also a fairly clear example of another kingdom, "API Abuse" (CWE-227). Path Manipulation (CWE-249) is described as a low-level variant of a buffer overflow (CWE-120), and it is in the same Input Validation kingdom; however, CWE-251 (Often Misused: String Management) also identifies the risk of string manipulation functions that enable buffer overflows, but it is classified under CWE-227 (API Abuse) instead. The rationale for these classification choices is not immediately clear.
CWE-699 does not aim to be mutually exclusive. Instead, it uses diverse categories with many relationships. However, as explained previously, this has implications for repeatability.
While CWE-1000 has mutual exclusivity as a goal, it has been difficult to accomplish. It tries to reduce overlap as much as possible. The explicit modeling of chains and composites has simplified this problem somewhat. However, there are still multiple perspectives embedded within the hierarchy, and not all chains or composites have been detected.
The explicit modeling of relationships (besides parent/child) seems to be unique to CWE; it has not been handled in [7PK], [CLASP], or [Krsul].
CWE-1000 captures most of the chains and composites in CWE. Since CWE-699 is more developer-friendly, it generally does not specify those relationships, although they are readily visible on the CWE web site.
Even if mutual exclusiveness is ultimately achieved, modeling inter-relationships would still be useful. For example, in biological classification, humans and dogs aren't in the same Order, but their association with each other is notable enough to be recorded.
The "Seven Plus One" principle used in 7PK, along with its emphasis on a single audience of developers, makes it psychologically acceptable. The 699 view, which borrows from previous taxonomic efforts and uses other categories to fill in gaps, is likely to be acceptable since it will be familiar to its intended audience. On the other hand, the 1000 view is not intended for basic developers, and its approach to classification is different than previous efforts. As a result, it requires a different mind set than people are used to, which could make it more difficult to adopt. However, we believe it is an important step forward in the understanding of weaknesses.
7PK uses a limited set of categories. CWE-699 uses a rich, overlapping set of categories, although this poses maintenance challenges. For the most part, CWE-1000 has been able to rid itself of categories, although some might argue that the Pillars are so abstract, that they might as well be categories.
The three views discussed in this paper have different strengths and weaknesses, depending on the audience. Given CWE's support for diverse audiences, these different views each serve a different purpose.
[7PK] Katrina Tsipenyuk, Brian Chess, Gary McGraw. "Seven Pernicious Kingdoms: A Taxonomy of Software Security Errors". November 2005. NIST Workshop on Software Security Assurance Tools, Techniques, and Metrics. http://cwe.mitre.org/documents/sources/SevenPerniciousKingdoms.pdf
[Amoroso] Edward G. Amoroso. Fundamentals of Computer Security Technology. 1994. Prentice-Hall PTR, Upper Saddle River, NJ.
[Howard] John D. Howard. "An Analysis of Security Incidents on the Internet 1989-1995", section 6.1. 1997. Carnegie Mellon PhD Thesis. http://www.cert.org/research/JHThesis/Chapter6.html
[Vulncat] Fortify Software Security Research Group, Dr. Gary McGraw. "Fortify Taxonomy: Software Security Errors". 2008. http://www.fortify.com/vulncat/en/vulncat/index.html
