Julius Detritus wrote:
They are outdated. The most recent exploit tested must be two years old...
They are copy and paste from IDS tests which are far older.
[...]
Do you really care about the phf exploit? Or maybe the old sshutupteo from
gobbles? Are you talking about organizations or museums?
[,,,]
So your exploit database must be very old
This raises the question: when testing IDSs - are the exploits used
important ? Do you have to use the latest ? Are the results invalidated
by old exploits ?
Back when I tested IDSs myself (that's now 4 years ago...), I didn't
think it was very important - I made sure, though, that I had various
"classes" of attacks (directory-traversal, shellcode/buffer-overflow).
Reproducability is a big problem, too. NSS probably wants test from
Q1/04 to be comparable with test from Q1/05, at least to some degree. So
I'd say it is - under this objective - mandatory to change as little
factors as possible - and that surely includes the exploits and
evasion-techniques (though they seem to vary nonetheless).
People flaming NSS should also keep in mind that testing IDSs today is
really a battle of materials. Back in late 2000 (when I did the work for
my thesis), I could use our training-lab with a dozen (well equipped)
PCs and some hubs to get a good picture of the capabilities of half a
dozen products.
Today, you need a big lab with switches, taps, packet-generators, big
servers - and you still can't simulate what it will look like in the
real-world with dozens of sensors and remote-locations - costs are
probably skyrocketing anyway.
I don't think NSS's test are the Holy Grail - but it cannot be disputed
that they have at least a methodically correct approach and re-test old
and new products continuously.
If anybody can do better - please stand up now ;-)
cheers,
Rainer
--
===================================================
~ Rainer Duffner - rainer@ultra-secure.de ~
~ Freising - Munich - Germany ~
~ Unix - Linux - BSD - OpenSource - Security ~
~ http://www.ultra-secure.de/~rainer/pubkey.pgp ~
===================================================
--------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------
