On Monday, February 21, 2005, at 12:06 PM, Mark Curphey wrote:
At Foundstone we have tested literally thousands of apps and the most
frequent and most insidious issues are rarely data validation issues
these
days. They were a few years back but not today. They are issues with
authorization systems, user management systems, business logic, data
protection in (storage and transport), broken protocols, badly
implemented
cryptography and such like. The bar has been raised.
A few people keep asking me where I think validation should be done. I
always say in the code. The argument sometimes comes back that that's
not
always possible. While I completely disagree (every commercial company
I
have worked for has a release cycle of every few weeks and consistently
changes the code base so either I have been living under a rock in
major
financial services companies or the vendors marketing thought they had
a
good sales angle)
The same is true of automated testing, especially pen testing. I
personally
firmly believe that the most cost effective and efficient way to test
software is to use the code. To say that code review is too expensive
is
misleading and outright wrong. It's a common misconception perpetuated
by
the uneducated. I wrote an article about it at www.softwaremag.com. Pen
testing web apps is a Turing problem, the number of inputs has to
equal the
number of outputs and that's computationally unfeasible in a modern
app. For
low hanging fruit its great but for the type of issues being found now
the
bar has been raised I don't think its appropriate. Again at work we
have
driven trucks past app that have been given a clean bill of health by
the
scanners. And that's pretty frequent not an edge case BTW. If you look
at
code analysis tools as well they are getting good at finding bugs like
overflows but flaws like people using poor PRNG or storing keys in
config
files they are next to hopeless. Guess what, flaws are the most
insidious
type of issue we find. Its not that these tools are of no value. We use
automated tools for code review and will likely buy or build something
for
web app pen tests this year. For the LHF it's a no brainer. The point
is any
automated solution is a subset of the problem space. At the moment I
would
say its less than 25% for the web app firewall space and about 20% for
the
pen test space (gut feeling).
<the above has been snipped to highlight the key points>
Per your "balanced" approach, I find conflict within several key points
of your assertions.
Web application software is not the same as other forms of software we
are used to dealing with. They're rate of change, the types of attacks,
and the scale of usage differs entirely. As such, so do they're
security requirements. You said above "...every commercial company I
have worked for has a release cycle of every few weeks and consistently
changes the code..." on this we agree. Then you went onto say "...code
review is too expensive is misleading and outright wrong." This is
where we disagree.
1) Source Code Reviews ARE expensive
While we could argue on how to define "expensive", we can all agree
line-by-line analysis is time intensive and requires intelligent
testers. As a result price quotes will go up into the several 10's of
thousands range, if not higher depending on the scope of work. So we
can use this as a basis for further discussion.
The first time a company has a code review performed, this is where
they'll get the most bug for the buck. Through an exhaustive process,
a good code review can knock out a ton of security issues and prevent a
lot of pain down the road. They also can find problems no scanner or
black-box pen-tester will ever find, like those pesky backdoors and
easter eggs. The draw back is the cost of third-party source code
reviews will prevent them from being repeatable. This is important to
understand if you subscribe to security is a process principal.
If companies have a release cycle every few weeks (per your quote),
having a third-party assessor constantly in-house is certainly going to
extend way beyond anyones budget. However, if your shipping shrink
wrapped software or a newly designed website where the code won't/can't
be changed for the next year, source code review is the way to go. Your
maximizing ROI. Pro and Con.
2) You compare Source Code Reviews and Security Assessments (Pen-Tests)
and its like comparing apples to oranges. Both solutions should be used
in different circumstances and maintain different ROI models.
If your web application is frequently updated, this is where you might
find security assessments (pen-tests) more cost effective. Security
assessments are typically faster and less expensive than your typical
source code review. Current assessment methodology will normally
identify nearly all (if not all) of the easy issues and most of the
hard ones. They'll also give you visibility into what a real world
attacker would see. I find it rare for attackers to have access to your
source code prior to targeting your website. Its going to be far more
common for attackers to black-box you and go from there. Vulnerability
Scanners and continuous assessment services help make make the process
more repeatable. But neither Source Code Reviews or Security
Assessments will ever claim to find everything.
The point I'm trying to make here is that customers have several
options available to them depending on they're needs. They have access
to Source Code Reviews, Penetration Tests, Vulnerability Scanners,
Application Firewalls, etc etc. Each with they're own set of pros and
cons.
To say your way is the one and only way is the part I find misleading
and irresponsible.
3) Scoping the Web Application Security problem.
We also assess the security of tons of web sites as well. As such, we
have comparable results. You said your results show the most frequent
and insidious issues today are "...authorization systems, user
management systems, business logic, data protection in (storage and
transport), broken protocols, badly implemented cryptography and such
like". On the severity of these issues when they are identified, I
believe our results would closely match. However, as far as the total
number of vulnerabilities found by class, nothing tops basic SQL
Injection and Cross-Site Scripting. The discrepancy might have
something to do with our mutually different approached and/or
methodologies.
4) Hypocritical discussion of "Thinly disguised marketing posts in
public lists"
You began and ended your email harping on vendors for what you said are
"Thinly disguised marketing posts in public lists". With a good
moderator present, this is something many of us find annoying yet
tolerable. You then proceeded to lace your message with the very same
negatives issues you were accusing the other vendors of. Mentioning
your company and offerings numerous times. Lets be honest, you have a
product to sell no different than anyone else. Yours just happens to be
called consulting. And if anyone were to read closely... you mirror
your own comment. "I think x and BTW I happen to make a product that
does exactly this."
I'm going to skip over the application firewall topic because thats a
whole other issue and I've written enough for now.
Regards,
Jeremiah Grossman
