On Wednesday, February 23, 2005, at 09:36 PM, Jeff Williams wrote:
There are essentially four ways of finding flaws in an application:
- Vulnerability scanning with signatures
- Manual penetration testing
- Static analysis of source or binary
- Manual code review
In your description, aside from automation, what is the difference
between bullet 1 and bullet 2? I've view the testing process as 3
fundamental methods. Automated "scanners" can be applied to each
method, but essentially they are performing the same process we'd do by
hand.
- black-box testing (functional testing)
- static binary analysis
- source code review
Any application review that relies on a single technique is going to
be less cost-effective than one that uses a combination. And when I
say "cost effective" I mean more serious flaws found for the $. The
80-20 rule is nice, but you've got to find the right 80%. Finding
4000 minor flaws and missing a major obvious hole doesn't help
anything.
"Find the right 80%". This is really good way to look at it.
Just so I'm totally clear here -- no matter how much you're planning
to spend on finding vulnerabilities in your application, you're better
off including some scanning, some code review, some static analysis,
and some penetration testing. Skilled analysts with a full toolbox of
techniques is what produces the best bang for the buck.
Well said and I think people are leaning that direction. The challenge
customers face is how much of what method to use and when. This answer
is not readily apparent and certainly not agreed upon in the industry.
We've seen people on the list comment on how they do (or have done)
secure software in the past. A process that made sense to them. It
always appears to be a combination... but the way the process is
implemented is always a bit different. This an area within the industry
I see lacking good information. Assisting customers to find the right
balance.
Also, I haven't come across much of anyone doing scans on binaries, but
I am in the web app sec world)
Anyone claiming that a single technique can find nearly all (if not
all)vulnerabilities or costs less than another technique just doesn't
get it. Savvy consumers will look for reviewers that use best of
breed scanning and static analysis tools. They'll also look for
serious pentest and code review expertise with the technologies
they're using.
One testing method might actually find all the vulnerabilities (bugs),
but its not something you want to rely upon. The premise is
theoretically code can free of bugs, we just have no way to prove it.
Standard Turing logic. Similarly to the way we diversify our investment
portfolio to provide safe and consistent results. Its no different with
security, we should diversify our solutions and go for defense in depth.
Regards,
Jeremiah-
