Re: Security Dashboard

On Feb 27, 2006, at 7:52 AM, mieke.kooij@ba.com wrote:

> Fred,
> You are asking some very good security questions, but seem to be  
> confusing security metrics with security goals.

The goal of the metrics had better be to support the security goals.  
My point is that the goals should drive the metrics and what we  
measure, not the other way around. The things identified for  
measurement here are things that may be easily measured, but they are  
not necessarily things that are meaningful to the enterprise.

> The point of the metrics is that it allows you to constructively  
> ask precisely the questions you outline below and to then set  
> appropriate targets depending on what you are willing/able to spend.

I think that the point of the metrics is to determine how effective  
the program is at protecting the enterprise and to determine where  
and whether it needs to be better or can be worse. The metrics  
provided don't fulfill either your criteria or mine.

> You as a security expert are in a position to advise the CEO in a  
> way they can understand.  This is where a 'traffic light' gets  
> used.  E,g. if you think a 5 day patch cycle is OK then it could be  
> green light, if you are concerned about this and want it sped up,  
> then red light.

I don't agree with this. I think you have the cart pushing the horse  
rather than the horse pulling the cart. Metrics that are going to be  
effective for the enterprise must start with the enterprise, not with  
the technology. The 5-day patch cycle is an example of something you  
can readily measure but is meaningless without context. In order to  
understand it, you need to start with the strategy for mitigating the  
risks of known vulnerabilities - which is what the patching cycle  
acts to meet - or not. If the strategy is that barriers are adequate  
to eliminate the need far patching critical systems, then the patch  
cycle is not even worth looking at for those systems. If patching is  
to protect small amounts of low valued data in mobile systems, then  
the patch cycle can be measured, but is so low down the priority list  
of things for the enterprise that the CEO should never see it or be  
aware of it - unless they happen to want to.

Just because we can measure it does not make it meaningful as a  
measurement. And the ability to measure some finite value out of an  
infinite range is also relativistic only and likely to be nearly  
meaningless. We caught 10 vulnerable systems today and patched them.  
Yesterday we caught 20 and patched them. Are we doing better or worse  
today than yesterday? The answer from this data is - I have no idea!

> Hope this helps!

Hope this helps...

FC

> Regards,
> Mieke.
>
>
>
> To:        "Steve R. Smith"
> cc:        horizons nouveaux
> loganalysis
> security-management
> bcc:
> Subject:        Re: Security Dashboard
>
>
>
> On Feb 22, 2006, at 9:19 AM, Steve R. Smith wrote:
> >
> > Some of the typical manually generated security
> > metrics that I've seen are things like:
> >
> > Patch cycle times vs. unpatched machines (from SUS or
> > automated patch tools, manual patch application and
> > monitoring, etc)
>
> But in what sense does this help actually answer any important or
> useful questions? Say I measure it and I find that the patch cycle
> time for a computer is 5 days and the number of unpatched machines is
> 20% of all machines. What value does this have for me. Should it be
> more? less? at what cost?
>
> > Phishing, SPAM content blocked at SMTP gateways
>
> What portion of what spam content - and by whose definition? You mean
> some spam blocked at some gateways? I don't even know what to count
> here.
>
> > Helpdesk calls- virus cases, password resets, static
> > token requests
>
> Suppose the helpdesk calls is 500 / week and virus cases is 17 /
> password rests are 4,589 . static token requests are 987. What does
> that mean exactly. Am I doing well or poorly? How should I change
> what I am doing - toward what goal at what price?
>
> > Vulnerabilities over time (Qualysguard, Foundscan,
> > IP360, Nessus, consulting reports, etc)
>
> I have reports of 47 vulnerabilities every time I scan with Nessus -
> what does it mean?
>
> > Security policy compliance (ISO17799)
>
> What am I measuring here?
>
> > Typically, this information gets sent up the chain to
> > various IT Executives such as CIOs, CTOs, CISOs, Audit
> > Directors, and/or corporate governing bodies.
>
> I send the report with the numbers above up to a CEO. What do they
> do? How do they interpret it?
>
> I have seen lots od dashboards - and so far I didn't find even one
> that did much to help me manage risks better. What am I missing?
>
> FC
>
> > Regards,
> > Steve
> >
> > --
>
> -- This communication is confidential to the parties it is intended
> to serve --
> Security Posture            securityposture.com          tel/fax
> University of New Haven               unhca.com        925-454-0171
> Fred Cohen & Associates                 all.net      572 Leona Drive
> ASP Press                         asp-press.com    Livermore, CA 94550

-- This communication is confidential to the parties it is intended  
to serve --
Security Posture            securityposture.com          tel/fax
University of New Haven               unhca.com        925-454-0171
Fred Cohen & Associates                 all.net      572 Leona Drive
ASP Press                         asp-press.com    Livermore, CA 94550