In-Reply-To: <41DD51DF.9080407@immunitysec.com>
Dave,
I've been following this thread for the last week or so. As you may know, we
like your CRI tool, and indeed we use it to test RPC handling and inspection in
Top Layer's network IPS products. I also appreciate many of your comments
regarding the importance of the IDS/IPS properly handling IP fragments, TCP
segments, and RPC fragments in order to defeat evasion attempts.
I'm not quite sure, however, why you're bashing the NSS IPS tests. Your
comments seem to be applying a very narrowly defined criterion as the basis for
dismissing the entire NSS IPS test suite. Specifically, I must take exception
to your claim that "They largely test for things you don't care about, such as
pushing packets down a wire."
As a vendor of IPS products, I can tell you that organizations planning to
deploy network IPS technology are VERY interested in how well the IPS can push
packets down the wire! They all run businesses, and the packets being "pushed
down their wire" are their lifeblood: payment transfers, sports bets, media
delivery, internal application requests, etc. In my experience, the ability of
the IPS to handle legitimate traffic as a "good networking device" is often
used as the *first* set of criteria in selecting an inline IPS product. I've
had many a customer who literally said that they didn't even want to *talk*
about the protection mechanisms until we'd proven that our device could operate
as a "good network citizen."
We've run our products through the NSS IPS tests, and I just can't agree with
the rest of your comments:
"They're not open tests."
The NSS test methodologies are published in full.
"They're outdated."
The first IPS test was a year ago and the NSS methodology was brand new.
You're right that it's mostly the same this year, save for some new exploits,
but I would not consider it outdated. I don't know of a more recent or more
comprehensive set of tests for a network IPS.
"They largely test for things you don't care about, such as pushing packets
down a wire..."
My experience shows that organizations DO care about the things that NSS tests
for: signature coverage, baseline performance, performance under load, latency,
application response times, anti-evasion capabilities, stateful operation,
management and configuration. I already mentioned my view about "pushing
packets down the wire."
Bob Walder from NSS can chime in here, but my understanding is that the NSS
signature coverage tests include many RPC-related exploits and their variants,
run both "in the clear" and with various evasion techniques, including modified
exploit code and RPC fragmentation.
"No scientific test should be non-repeatable"
We've been able to repeat the majority of the NSS tests consistently in our
lab. You might be talking about the fact that the capture files for the attack
recognition tests are not publicized. This topic was addressed in the thread
regarding the Tipping Point Tomahawk tool already. Clearly the set of
"attacks" used is the result of work that NSS has performed, and I understand
their desire to keep that proprietary.
"and no scientific test should require such large amounts of money to change
hands."
Do you mean the test fees? The report fees? If so, why not? It's called
"business." Only by charging money can a test house spend the amount of time
necessary to REALLY test advanced products like network IPS. In fact, you
might be able to use CRI to create your own mini-test, and charge IPS vendors
to participate in it! Or why not work with Walder directly to have him use CRI
to enhance his evasion section of his test?
Mike Paquette
VP Technology,
Top Layer Networks, Inc.
--------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------
