Fred Cohen wrote, On 2/27/2006 10:43 AM:
On Feb 27, 2006, at 8:18 AM, Andrew Steingruebl wrote:
Fred - do you think it at all appropriate to actually capture metrics
about the security program? In any process we can simply measure the
results (risks) but we might also want to understand how cost effective
our security program is.
I think it is vital to collect metrics on the security program, but the
right metrics must be collected or you are wasting money and time and
not solving the critical problem. Measuring the "risks" seems to me to
be the hardest thing of all here and nothing gets even close to doing
this from a technical standpoint. Cost effectiveness would necessitate
understanding the impact of different ways of spending money on the
risks, which is even harder than measuring risks. My view is that these
are largely wastes of time in the security space today. It's not that I
wouldn't like to see it, but rather that we can't do it well enough to
make it worthwhile yet.
I guess I don't fundamentally disagree but we're a long way from having
actuarial tables that tell us true likelihoods for certain threats, etc.
People are working on them, but the lack on concrete threat frequency
data shouldn't deter me from putting in place at least some security
controls and metrics to comply with those sorts of things we believe
will generally reduce risk. Yes we're in the stone ages, we're not
doing civil engineering, reliability, etc.
At the same time simply not having certain controls in place, whether
effective against security risks or not, is itself a risk in terms of
liability. It doesn't keep my data more secure, but it keeps my
business more secure. This doesn't mean we should focus on 100%
meaningless metrics like "number of 'naughty' sites stopped by our
content inspecting proxy server".
We're not banks that can use credit scores to evaluate the risk of a
certain customer with high statistical likelihood. Its going to be
years before we are there. So, while I wouldn't spend a fortune on
security metrics, I'm not going to punt either and give up on trying to
measure the effectiveness of my program against relatively elusive goals.
I'm a big fan of metrics that track process performance that just
security results.
And, I guess I'll just have to disagree with your patch metrics
analysis. I don't think we're to the place yet where all worms/viruses
are zero-day and where having patch and AV processes in place doesn't
help. Certainly there have been those cases and the window is getting
smaller, I don't think that means tracking deployment of countermeasures
is a waste of time.
--
Andy Steingruebl | e-mail: asteingruebl@cccis.com
Information Security Architect| phone: (312) 229-2409
CCC Information Services | post: 444 Merchandise Mart
| Chicago, IL 60654-1005
