Feature Scoring

To allocate testing resources effectively, applications must be broken up into features. Looking at the number of features in many modern software applications, it's clear that all features cannot be tested in a finite amount of time. Therefore, to hone in on features most likely to contain security vulnerabilities, we use a scoring mechanism that assigns weights to features to help allocate testing resources. Application scoring is not new. Michael Howard and David LeBlanc present an excellent feature-scoring scheme based on source-code access in their book Writing Secure Code, Second Edition (Microsoft Press, 2002). Our scoring has the requirement that features must be assigned values quickly—without access to, or analysis of, the application's source. In our approach, testers simply mark-off against a list of properties that may have security implications.

• Is it a security feature? [+4]
• Is it a new feature? [+3]
• Does it use the network? [+3]
• Does it use permissions? [+2]
• Does it run as a privileged user? [+1]
• Are there known vulnerabilities in previous versions? [+1]
• Does it use a Database Management System (DBMS) such as SQL? [+2]
• Does the feature use structured data? [+2]
• Are there severe consequences of failure of this component to the rest of the application? [+3]
• Does this feature use concepts of persistent state? [+3]
• Are there any dynamic conversions? [+2]
• Are certain categories of inputs/actions blocked (as malicious)? [+2]

The weightings have been derived through iterative refinement on multiple testing projects. The score an application receives as a result is highly correlated to the number and severity of security vulnerabilities discovered, not just during our testing, but also by the user community after software release.