The particular web site itself is more the cause of the discrepancy than
anything else. What is clear though is you have to test
using both methods no matter what.
Agreed. Actually all four methods (scan, pentest, static analysis, code
review)
If you are in the web app sec world and you aren't
scanning binaries, you're wasting your customer's
time and money.
This is a bit strong and largely subjective without describing what is
supposed achieved using the results. In every engagement, whether your an
attacker or a tester, you not always going to have access to the source
code or binary. This is the framework your given to conduct your work.
Good point. I should clarify. If your customer won't let the security
analysts analyze the code, then *they* are wasting their time and money --
at least if the goal is to find the most serious security flaws in the most
cost-effective manner. If they're looking to "check the box" or have some
political goal for the review, then perhaps not finding flaws efficiently is
what you want.
It's not my experience that there is no flexibility in how these assessments
are done. I think we have an obligation to help companies understand the
weaknesses with doing only scanning or only pentesting. If you think a
combination of techniques IS more cost-effective, then you should be
explaining to them that using a combined approach yields more complete, more
accurate results for the dollar.
1) What exactly is scanning strategy they employe to find these issues?
I've had this half answered. I could be wrong, but these tools appear try
to find areas of input. Then decide if the data has been properly
sanitized. Looking at the results they appear to point you to the area
where a potential issue might be, but fall short of certainty. Is this
your impression?
Data-flow analysis is only one part of static analysis. These tools also
look at where security mechanisms are invoked to ensure that they're being
used properly. They look for coding patterns that generally indicate a
security vulnerability. And they look at the structure of the code, to see
if security mechanisms are properly isolated and centralized. They
definitely fall short of certainty -- but they're very good at steering an
analyst to the right parts of the code to look at. You should make sure it
covers the technology you're using (especially extra libraries) and that
it's easy to train it to know about the custom security libraries involved
in the application.
2) How well have have performed in practice on real production web
application code? Have you used many? What were the results like?
Our first J2EE static analysis tool was built in 1999, and we've been using
them internally ever since. Recently we've been using some of the
commercially available tools as well. Our results have been excellent --
especially in combination with the other approaches. They are very good for
security analysts, as they can quickly point to sections of a large software
baseline that need manual review. For example, reviewing an application for
SQL injection problems is just a matter of clicking through the database
interactions to be sure they don't include unvalidated user input. In many
cases this will be faster than scanning, and will almost certainly be more
accurate.
--Jeff
Jeff Williams
Aspect Security, Inc.
http://www.aspectsecurity.com
