In-Reply-To: <41DD51DF.9080407@immunitysec.com>
Received: (qmail 13780 invoked from network); 7 Jan 2005 00:27:04 -0000
Received: from outgoing.securityfocus.com (HELO outgoing2.securityfocus.com)
(205.206.231.26)
by mail.securityfocus.com with SMTP; 7 Jan 2005 00:27:04 -0000
Received: from lists.securityfocus.com (lists.securityfocus.com
[205.206.231.19])
by outgoing2.securityfocus.com (Postfix) with QMQP
id 110EB14375C; Thu, 6 Jan 2005 17:32:05 -0700 (MST)
Mailing-List: contact focus-ids-help@securityfocus.com; run by ezmlm
Dave Aitel wrote:
I guess the interesting thing is that you actually bought something for
your millions of dollars. Or perhaps it's a look into the Speed vs.
Accuracy trade off. Lots of other people have spent millions of dollars
on professional engines, but still fail the simple tests like this
because all nss.co.uk is testing for is extremely old attacks and
whether an IDS can take the load of millions of packets at once. This is
going to favor Snort-like systems largely at the expense of parsing
engines. I think it's telling that nss doesn't test MSRPC at all. It's
funny how the IDS industry has tuned itself. But set your MTU low
enough, and you can bypass some systems even if you're the only packets
on the wire. Doing SMB fragmentation basically guarantees it.
If you're looking for a misleading test, the NSS.CO.UK tests are what
you want. They're not open tests. They're outdated. They largely test
for things you don't care about, such as pushing packets down a wire. No
scientific test should be non-repeatable, and no scientific test should
require such large amounts of money to change hands.
I really suggest reading the reports that NSS issues including their market
overview and test methodology in order to learn about how to analyze and test
security devices or any other communication devices. To say that NSS's tests
are out of date is simply not true.
Evaluation of IPS products raises a great challenge for the evaluator. In my
experience, the NSS group does a very thorough and, perhaps most importantly,
un-biased work with their round of tests of IPS devices.
By examining NSS's test methodologies (published in their site and in every
report they issue), it is easy to recognize the level of understanding that the
NSS group has in regarding to the IPS market and product positioning (this
understanding is the first step in establishing the correct test scenarios and
success criteria).
Regarding to Evasion techniques, NSS's tests comprise more than enough methods
that try to evade detection. These include: Packet fragmentation which include
19 different methods of IP packet fragmentation and Stream segmentation, URL
Obfuscation which include 9 URL obfuscation techniques (e.g., URL encoding,
premature URL ending, session splicing etc), other miscellaneous evasion
techniques... Of course there will always be new evasion techniques but it
seems that NSS has chosen to use the most updated and common ones. Let's
remember that no test can include all the possible evasion techniques but the
important thing is to aim as high as possible.
NSS includes also special evasion techniques in order to test rate-based NIPS
which are usually based on time-dependant thresholds. In order to test these
detection engines NSS generates DoS attacks, network scans and self-propagating
Worm activities with different delays between packets(e.g., very slow scans,
random time between events, slow TCP connection floods, slow SYN attacks etc.).
In this way NSS analyzes how sophisticated these rate-based detection engines
are.
According to NSS reports, they have all the equipment and experience that is
needed in order to simulate background traffic that emulates "real" world
legitimate user behaviors (throughout several popular applications). This is a
very important capability that helps to reveal false positive and misdetection
percentages of the detection and prevention engines maybe the most important
test for IPS devices (as high percentages of false positive renders the IPS
devices useless).
NSS indeed pushes the products to their limits. I think that this is certainly
necessary in order to reveal how much "brain" work was invested in the
hardware and software architecture. NSS's performance test includes playing
with parameters such as number of simultaneous TCP connections, TCP connection
rates, Packet sizes, packet rates, etc. This capability allows an analysis of
the immunity of the detection engines against false positive and misdetections.
It is also interesting and educating to see how NSS approaches differently
rate-based NIPS and Content-based NIPS with their performances and false
positive rate tests.
Avi Chesla.
--------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------
