Hi Ivan,
There is a lot of debate about what the "background" traffic should look
like, so we test based upon what is closest to "real world". And I
think this approach has kept us from being surprised in customer
environments.
Our Security Research team use Spirent's Avalanche & Reflector to dish
up "real" content based upon popular web sites like Google & Yahoo as
well as "real" smtp, ftp, pop-3, imap etc. Average transaction size is
13K. We have found that focusing on transactions per second is as
important as the actual throughput since it flexes the state engine
within the IPS.
To test signature effectiveness, our Security Research team uses as many
real exploits as we can get our hands on in addition to Metasploit and
other free and commercial tools. At the end of the day, if it is not
real, it is not real. Period.
Along those lines, using VMWare whenever possible, we have set up an
extensive lab in order to replicate OS/Applications & their
vulnerabilities. VMWare is an awesome tool. You can set up a host,
create an image for future use, compromise the host, reinstall using the
image you created earlier and then test again to verify repeatability.
IMHO, it should be part of every serious tester's toolkit.
Here is our procedure:
1) Verify that the exploit can compromise a host (gain root shell, etc.)
2) Obfuscate the attack using fragrouter and verify again (got shell?)
3) Insert ipANGEL and verify detection & successful dropping of attack
4) If a signature successfully drops an attack, verify it is not due to
coincidence. (We've seen signatures out there that drop an attack
because they drop nearly everything!)
5) Once the signature has been tested and verified it gets certified and
published.
I think this is the right way of doing things, and I'd love it if
testers would make their tests more "real". We've seen cases where
other products fire off with testing tools, but not with real tests. My
conclusion is that they wrote their signatures based upon the testing
tool, and not the actual vulnerability. I think Greg Shipley mention
this phenomenon in another post.
Bottom line: If you don't have the money for Avalanche & Reflector,
tcpreplay can be used, but it has some serious limitations. At most,
use it for generating background traffic, but don't use it to verify
attacks get stopped the way they should. Use the real attacks,
Metasploit (which uses attacks) or other tool that can compromise a
host. That way you know whether the attack was successful or not and
are not basing your judgement on an unknown replay.
Best Regards,
-Vik
Vikram Phatak, CTO
Lucid Security
http://www.lucidsecurity.com
-----Original Message-----
From: Ivan Arce [mailto:ivan.arce@coresecurity.com]
Sent: Monday, February 20, 2006 8:52 PM
To: focus-ids@securityfocus.com
Subject: Re: Testing IDS with tcpreplay
Aaron Turner wrote:
On 2/14/06, ehanselman@netscape.net <ehanselman@netscape.net> wrote:
Rather than replaying, you'll get a much better view of how well the
IDS works if you use real attacks with real obfuscation techniques.
Metasploit is a great tool for this (www.metasploit.org).
Setting up Metasploit doesn't have to be hard. There are a bunch of
tutorials on using a Whax (bootable Linux CD) ISO image to run from.
Simply pop in the CD and boot.
The one hitch is that you'll need to have real victims to attack.
Setting up a few target systems as VMWare images makes testing
simple.
You can use the snapshot capability to return the victim to a
pre-attack state.
Generally speaking, tcpreplay is better when one or more of the
following is true:
1) Trying to do comparative analysis and you want to make sure each
device sees exactly the same thing
Hmm, why is that harder to accomplish with Metasploit than with
tcpreplay?
2) Need to automate or do a lot of regression testing and want a
stable and relatively simple lab environment
same as above....
3) Already have a library of pcap's (either from customers, the wild
or capturing traffic of real tools like Metasploit)
Yeah, but that is an entirely different kind of testing. Replaying the
packets captured from the execution of an exploit is not the same as
executing the same exploit again.
4) Don't want to worry about re-installing or fixing target systems
after they've been 0wn3d. VMware of course helps, but there is still
a lot more administrative overhead.
Hmm, maybe or maybe not... Actually you can pretty much automate the
entire process (or a big part of it):
1. set up of the proper VMware images (specially if you're using GSX or
a similar virtualization server that lets you manage images
programatically and from remote)
2. run a set of exploits in the appropriate order
3. generate reports or other output with the results
4. correlate output with IDS/IPS alerts/logs/etc.
5) You don't want to have to install and then maintain 10's or 100's
of applications and their operating systems to break into.
Thats a valid point...however you could pre-install these on your VMware
images...
In general, tcpreplay isn't all that useful IMHO when you're first
starting off and "want to do some IDS/IPS testing" or only intend to
run a few tests or tests only once or twice unless you already happen
to have a nice pcap library.
Ahh that's interesting, I see it in exactly the opposite way: tcpreplay
is ok when you want to scratch the surface of your IDS capabilities or
perhaps more appropriate for stress or throughput testing or very basic
regression testing. However, if you truly want to check if the IDS
recognized real attacks you need to test with real exploit runs not with
a replay of their captured traffic.
Obviously the biggest limitation of tcpreplay is it doesn't come with
a library of pcaps. Maybe one of these days I can figure out the
In my view, the biggest limitation is that replaying captured packets an
overly simplified manner of modeling real world attacks. Today's exploit
code is a lot "smarter" than simple PoC that send the same fixed data on
each run. modern exploits make runtime decisions based on the state of
the target system/application and several other things. To successfully
simulate the execution of real exploits you need to maintain state about
both endpoints (target and attacker's systems) and properly simulate the
meaningful state changes in them that would change exploit-code's
execution flow and elicit different traffic patterns that those from
previous runs.
BTW... there is a commercial product mentioned at the footer of all
emails in the IDS list, notably no-one commented on it :)
-ivan
--
Aaron Turner
http://synfin.net/
The problem with pcaps is that you're working with exploits that have
already been seen and are static. If your goal is to determine IDS
effectiveness, using real attacks is better.
- Eric
------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to
http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------
--
---
"Buy the ticket, take the ride" -HST
Ivan Arce
CTO
CORE SECURITY TECHNOLOGIES
http://www.coresecurity.com
PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836 B25D 207B E78E 2AD1 F65A
------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------
------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------
