Hi Justin,
One thing you have to keep in mind is that a lot of things are incredibly
variable when dealing with this subject. For instance, suppose you want to
ensure that the URI in a web server is not overflowable. So you test with
something like
GET /[AAAAAAAAA x 4096] HTTP/1.1
Host: foobar.com
Connection: close
This is all fine and well, unless at 8192 is where the overflow takes
place,
or if I can stick as many characters as I want in, so long as I am using
HTTP 1.1 and not HTTP 0.9, or if I am using HTTP/1.1 and Host doesn't
contain a 36 backslashes, et cetera.
This is generally why fuzzing is mostly inconclusive because it often
misses
a lot of conditions and you have essentially assured nothing. Without
in-depth analysis of each software package you are basically pushing snake
oil. There are just far too many variables to really standardize such a
thing.
There are a few things to remember:
1) There are still too many products that fall to simple fuzzing. Having a
standard that employs fuzzing as part of it means that (hopefully!) vendors
will develop at least to that level - this raises the bar so to speak.
2) Not all fuzzers are born equal. Having written a fair few in my time I do
realize that condition based fuzzing is important. A very simple but quite
common example, to add to the ones you given, is with SMTP fuzzing. Some
overflows only trigger after an EHLO greeting but not after a HELO. A good
fuzzer and a good fuzzing process should take into consideration as many
conditions as possible.
3) Fuzzing would only be part of the standard to be proposed. There are
code/assembly scanning tools which can be incorporated amongst other things.
Cheers,
David
Best Regards,
Justin Ferguson
Reverse Engineer
NNSA IARC
702.942.2539
"It is a capital mistake to theorize before one has data. Insensibly one
begins to twist facts to suit theories, instead of theories to suit
facts."
-- Sir Arthur Conan Doyle
-----Original Message-----
From: Adam Shostack [mailto:adam@homeport.org]
Sent: Friday, May 12, 2006 11:35 AM
To: David Litchfield
Cc: bugtraq@securityfocus.com;
full-disclosure@lists.grok.org.uk;
ntbugtraq@listserv.ntbugtraq.com; dbsec@freelists.org
Subject: Re: How secure is software X?
Hi David,
Very briefly because I'm swamped today: Please consider
bringing some of this to Metricon
(https://securitymetrics.org/content/Wiki.jsp?page=Welcome)
Also there's a project of US DHS/NIST and probably others
called SAMATE Software Assurance Metrics and Tool Evaluation
http://samate.nist.gov/index.php/Main_Page
which might be of interest.
Adam
On Fri, May 12, 2006 at 02:59:17AM +0100, David Litchfield wrote:
| How secure is software X?
|
| At least as secure as Vulnerability Assessment Assurance
Level P; or Q
| or
| R. Well, that's what I think we should be able to say. What
we need is an
| open standard, that has been agreed upon by recognized
experts, against
| which the absence of software security vulnerability can be
measured -
| something which improves upon the failings of the Common
Criteria. Let's
| choose web server software as an example. When looking for
flaws in a new
| piece of web server software there are a bunch of well
known checks that
| one would throw at it first. Try directory traversal
attacks and the
| several variations. Try overflowing the request method, the
URI, the query
| string, the host header field and so on. Try cross site
scripting attacks
| in server error pages and file not found messages. As I
said, there's a
| bunch of checks and I've mentioned but a few. If these were
all written
| down and labelled with as a "standard" then one could say
that web server
| software X is at least as secure as the standard -
providing of course the
| server stands up.
|
| For products that are based upon RFCs it would be trivial
to write a
| simple
| criteria that tests every aspect of the software as per the
RFCs. This
| would be called Vulnerability Assessment Assurance Level:
Protocol. If a
| bit of software was accredited at VAAL:Protocol then it
would given a
| level of assurance that it at least stood up to those attacks.
|
| Not all products are RFC compliant however. Sticking with
web servers,
| one
| bit of software might have a bespoke request method of
"FOOBAR". This opens
| up a whole new attack surface that's not covered by the
VAAL:Protocol
| standard. There are two aspects to this. Anyone with a
firewall capable of
| blocking non-RFC compliant requests could configure it to
do so - thus
| closing off the attack surface - from the outside at least.
As far as the
| standards go however - you'd have to introduce criteria to
cover that
| specific functionality. And what about different
application environments
| running on top of the web server? And what about more
complex products such
| as database servers? I suppose at a minimum for DB software
you could at
| least have a standard that simply checks if the server
falls to a long
| username or password buffer overflow attempt and then fuzz
SQL-92 language
| elements. It certainly makes standardization much more
difficult but I
| think by no means impossible.
|
| Clearly, what is _easy_ is writing and agreeing upon a VAAL:Protocol
| standard for many different types of servers. You could
then be assured
| that any server that passes is at least as secure as
VAAL:Protocol and for
| those looking for more "comfort" then they can at least
block non-RFC
| compliant traffic.
|
| Having had a chat with Steve Christey about this earlier
today I know
| there
| are other people thinking along the same lines and I bet
there are more
| projects out there being worked on that are attempting to
achieve the same
| thing. If anyone is currently working on this stuff or
would like to get
| involved in thrashing out some ideas then please mail me -
I'd love to hear
| from you.
|
| Cheers,
| David Litchfield
| http://www.databasesecurity.com/
| http://www.ngssoftware.com/
_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/
