Large-scale comparisons using historical data, while suggestive, have
certain limitations. I touched on many of these in my open letter on
the interpretation of vulnerability statistics [1] when talking about
trend analysis in vulnerability databases, but many of the points
apply here.
For example, there appears to be distinct difference in editorial
policy between Oracle and Microsoft in terms of publishing
vulnerabilities that the vendors discovered themselves, instead of
third parties. This might produce larger numbers for Oracle, which
appears to include internally discovered vulnerabilities in their
advisories, whereas this is not necessarily the case for Microsoft
[2], [3]. In both cases, the lack of details can mean that multiple
issues wind up with one public identifier; for example, Oracle Vuln#
DB01 from CPU Jul 2006 (CVE-2006-3698) might involve 10 different
issues, and this is not an isolated case. This can further muddy the
waters.
Another difficulty that I originally mentioned involved possible
differences in research community bias. I don't think we really know
how many researchers are looking at which product, what their skill
levels are, and how much effort they're putting into looking. David
touches on this topic briefly:
Do the SQL Server 2005 results have no flaws because no-one is
looking at it?
No. I know of a number of good researchers are looking at it. SQL
Server code is just more secure than Oracle code.
It would be nice if the research community could begin to quantify the
level of effort they are putting into their analyses. If researchers
are putting 10 times more effort into one product than another, then
you might expect to find 10 times more issues (assuming the products
started at the same baseline of quality).
In short, we need better metrics before we can really compare the
relative *inherent* security of products. (The work from Wing,
Manadhata, and Howard for measuring relative attack surface shows
promise). However, public stats are the best we have for now.
- Steve
[1] Open Letter on the Interpretation of "Vulnerability Statistics"
http://lists.grok.org.uk/pipermail/full-disclosure/2006-January/041028.html
[2] Skeletons in Microsoft.s Closet - Silently Fixed Vulnerabilities
Steve Manzuik, eEye
http://www.blackhat.com/presentations/bh-europe-06/bh-eu-06-Manzuik.pdf
[3] Microsoft Patches: When Silence Isn't Golden
Ryan Naraine, eWeek
http://www.eweek.com/article2/0,1895,1951186,00.asp
