Spidering

Hi All

One of the most important aspects of assessing a web application is
separating the dynamic code from the static HTML. I tend to do this though
spidering, and use a number of tools including Paros and a commercial tool.

Basically, I like to know:

The names and places of the scripts
The variables they take
The Comments etc in HTML
Hidden form values

All the usual.

Whilst I don't expect it to get everything (JavaScript etc is going to take
manual intervention, so is a number of other possible technologies), I have
never really found a tool that I consider to be the defacto spidering tool
from this perspective. One of the biggest problems is a lot of the spiders
seem to choke on really big sites, or go into infinite loops  etc etc.

Has anyone got a reliable spider that they've used time and again (even on
big sites) that they could recommend? Or are most people WGETing the site
and then regex'ing the local mirror?

Thanks in advance

------------------------------------------------------------------------
This list is sponsored by: Cenzic

Need to secure your web apps NOW?
Cenzic finds more, "real" vulnerabilities fast.
Click to try it, buy it or download a solution FREE today!

http://www.cenzic.com/downloads
------------------------------------------------------------------------