On Feb 27, 2006, at 8:18 AM, Andrew Steingruebl wrote:
Fred Cohen wrote, On 2/24/2006 12:50 AM:
On Feb 22, 2006, at 9:19 AM, Steve R. Smith wrote:
Some of the typical manually generated security
metrics that I've seen are things like:
Patch cycle times vs. unpatched machines (from SUS or
automated patch tools, manual patch application and
monitoring, etc)
But in what sense does this help actually answer any important or
useful
questions? Say I measure it and I find that the patch cycle time
for a
computer is 5 days and the number of unpatched machines is 20% of all
machines. What value does this have for me. Should it be more?
less? at
what cost?
This metric doesn't necessarily compare against a baseline, but in
some
ways it helps us calculate risk. Especially whne a new threat emerges
for which there is a patch. We have some idea fo how long we'll
specifically be vulnerable, and what the costs might be. This metric
isn't useful in isolation, but combined with a look at the
threats/risks, we get a sense of our exposure.
Under certain conditions, that is, where we can identify that the
patch process is the only defense in place and that the systems have
identified value in terms of the potential negative consequences of
failure to patch, and when we have a likelihood of causing that harm,
nearly 100% with no other protection in place for a widely spreading
worm, we can measure this. And what we will find in most cases is
that the time till infection is very short (on the order of less than
a day and often less than an hour), so unless the harm comes linearly
with time (in most cases there is a large early impulse of harm from
such a thing), the result is we had better patch within a few hours
of initial deployment of the worm or we will lose most of what we are
likely to lose. So the time to patch becomes relevant only in the
sense that if it is less than a few hours from initial work
deployment, we save, and otherwise we lose. So as a metric, it is
highly nonlinear and all we really want to know is whether it is fast
enough. Odds are it is not in this case because it takes too long to
get the patch. In other cases, for example a virus deployed but witha
later triggering date, the time till patch is driven by the deadline,
in which case the consequences are nearly zero (except for cost of
mitigation) until the deadline and then high. In which case, the time
till patch can really only tell us if we are screwed or not and
cannot alter the outcome significantly. In other words, if we have
planned for some set of scenarios we are willing and able to handle,
then we can tell from the metrics whether we have met the plan or
not, but that is all. And indeed, we may have a multifaceted plan
where the patch time links to other tactics - like deployment of an
IUPS solution to this particular attack mechanism until the patch can
be safely deployed - in which case the consequence date/time tells us
which approach to use. So the real metrics here are (1) is the time
till patched requirement met, and (2) the deadline for the attack
vector. So these are not continuous variables, just time requirements
for a plan to work. The CEO - if s/he needs to know anything about it
- has to know that the plan calls for mixed strategies and that the
time frames have been met in deployments to affect these strategies.
Helpdesk calls- virus cases, password resets, static
token requests
Suppose the helpdesk calls is 500 / week and virus cases is 17 /
password rests are 4,589 . static token requests are 987. What
does that
mean exactly. Am I doing well or poorly? How should I change what
I am
doing - toward what goal at what price?
In an of itself this is purely an operational metric, but it does
potentially track the impact that given security decisions are
having on
the rest of the organization. Increase or decrease the conditions on
passwords such as length, frequency of change, etc. and you're
certainly
going to see movement in these numbers, which can be considered
direct/indirect costs of the security policy. Not measures of security
itself, but certainly of the costs.
So we know that we can increase costs by 'improving' these things we
can measure. Not very helpful in my view.
Security policy compliance (ISO17799)
What am I measuring here?
Well, potentially regulatory compliance. Being out of compliance, and
how far can certainly be an issue.
There are a lot of ways to comply to regulation and policies, and
they don't necessarily match up.
I have seen lots of dashboards - and so far I didn't find even one
that
did much to help me manage risks better. What am I missing?
Fred - do you think it at all appropriate to actually capture metrics
about the security program? In any process we can simply measure the
results (risks) but we might also want to understand how cost
effective
our security program is.
I think it is vital to collect metrics on the security program, but
the right metrics must be collected or you are wasting money and time
and not solving the critical problem. Measuring the "risks" seems to
me to be the hardest thing of all here and nothing gets even close to
doing this from a technical standpoint. Cost effectiveness would
necessitate understanding the impact of different ways of spending
money on the risks, which is even harder than measuring risks. My
view is that these are largely wastes of time in the security space
today. It's not that I wouldn't like to see it, but rather that we
can't do it well enough to make it worthwhile yet.
For a few of the metrics measured above they aren't useful in
isolation
and/or without some guidance as to acceptable, but that doesn't make
them worthless. In some cases specific security measures aren't going
to be directly to risk reduction in a purely security sense. That
said,
an organization that doesn't patch its machines and has an incident is
going to much worse liability wise than one that does. So, if we
believe our goal should be to have n% of machines patched, then
tracking
how we're doing is probably a good idea.
I think that is not likely to be a very useful goal to the program.
It is a gross sort of measurement that misses the really critical
issues of effective protection programs - matching surety to risk,
meeting duties to protect, and limiting consequences are examples or
better goals - in my view.
The metrics themselves don't result in more security, but they
certainly
help tell us how we're doing against our own measures, which
admittedly
are still just ballpark ideas about how to reduce risk.
This is what I am missing. The link between these metrics and risks.
They are essentially unlinked today and without the linkage - such as
that described above - they are meaningless and should not be
collected. For each thing of import to the strategy, we have to do
the analysis to understand what is important and to what extent, set
realistic goals, and measure our ability to achieve them.
Do you have any other approaches you think work better?
My belief is that you start with business understanding, develop
strategies, and then measure how well you carry them out and how
effective they are. You adapt the strategies when they fail and the
quality of carrying them out when it is inadequate. This is detailed
at a layer more depth in the picture (enterprise security
architecture) at my Web site (all.net).
FC
--
Andy Steingruebl | e-mail: asteingruebl@cccis.com
Information Security Architect| phone: (312) 229-2409
CCC Information Services | post: 444 Merchandise Mart
| Chicago, IL 60654-1005
-- This communication is confidential to the parties it is intended
to serve --
Security Posture securityposture.com tel/fax
University of New Haven unhca.com 925-454-0171
Fred Cohen & Associates all.net 572 Leona Drive
ASP Press asp-press.com Livermore, CA 94550
