I've run WebInspect (WI, SPIDynamics) and AppScan (AS, Sanctum, now
Watchfire) against 2 large ecommerce site code bases, written in IIS/ASP/VB
and Apache/IBM Websphere - Websphere Commerce Server/JSP. Both products seem
equally comprehensive. AS is faster and Watchfire says they do support
better because they are bigger and have a support organization. I found both
organizations to be responsive. The Watchfire folks didn't seem to know as
much about the world outside their product as the WI folks did. Another
dimension that I found very important to evaluate, is how useful the
reporting is for planning remediation projects and actually fixing the
problems. Both products might show 100 vulnerabilities. This might actually
only involve 5 web pages because they count a vulnerability as a single
failed test on a single field on 1 page. 5 failed XSS tests on the same
field in the same page will count as 5 vulnerabilities. If the XSS tests are
in different categories (HTML injection vrs unchecked parameter) they will
be in different parts of the reports. AS is very poor at telling you
1. what are the different unique web pages (no repeated web pages) with a
vulnerability
2. for each vulnerability type, what pages does it occur on
Just finding out exactly what different pages were involved took 2 hours
looking at the AS report screen. They report the same page over and over
again in different places.
WI is somewhat better and I haven't seen a version later than 5 months ago.
The pie charts, executive reports and spreadsheet exports don't help
planning the work to fix the problem.
----- Original Message -----
From: "Tom Stracener" <strace@gmail.com>
To: <webappsec@securityfocus.com>
Sent: Sunday, October 10, 2004 2:45 PM
Subject: Re: Of the three expensive vulnerability scanners
In-Reply-To: <20041007153115.28058.qmail@www.securityfocus.com>
Hi! I sought to answer this question for myself a while back, so hopefully
you'll find my own experiences here useful. First, consider
the types of applications and the application environment you will be
securing. Depending upon the complexity of the web application you're
dealing with, your likely to get quick diminishing returns from the tools
you have mentioned. Strong manual testing capabilities are a must, in my
opinion, and sadly a lot of commercial apps fall short there.
When possible, you should contact the vendors and acquire a demo license
in order to get a feel for how a tool actually performs. If that's not
available, then you should sit down with the vendors and get a hands on
session.
SPI Dynamics is very demo friendly. You'll find their people polite,
professional, and quick to respond once you download the product. So if
you want to take a look at it, just contact Natalie Hinkle
<nhinkle@spidynamics.com> if you have any questions or run into problems
downloading it. Also, if you go this route be sure to download the SPI
Toolkit, which includes some manual pen testing utilities.
With Sanctum, acquiring a demo was more difficult, I had to speak with
the salesperson's manager and then wait a few days, only to be declined.
Only after sending an email to their VP Internal Sales together with my
resume did I manged to get a demo. You may have better results. Jane
Foulkes <jfoulkes@sanctuminc.com> is a sales person you can contact over
there.
Last I checked Scando did not have a demo available at all.
I would also strongly encourage you to contact Cenzic and discuss having a
look at their up and coming version of Hailstorm 2.0. Its by far the most
extensible of the available commercial offerings. The tool provides a nice
balance of automated verses manual app spidering, allows you to record and
replay complicated HTTP sessions (which they call traversals) and then you
can apply different types of security policies as Hailstorm iteratively
steps through the web application. You can also create your own policies
and have full control over the fault injectors which interrogate the app,
as well as types of response conditions you're interested in detecting.
This tool shows an incredible amount of promise, so it would probably be
in your interest to evaluate it. You can contact Mandeep Khera over there
<mandeep@cenzic.com> if you're interested finding out more about it.
Also, browse the recent archives of this list because your question
has surfaced in various forms and you'll be able to find a variety of
useful perspectives.
--Tom
