return to ICG Spaces home    ICG Risk Blog    discussions    newsletters    login    

ICG Risk Blog - [ Structured IT risk remediation: Integrating security metrics and Design Basis Threat to overcome scenario spinning and fear mongering ]

Structured IT risk remediation: Integrating security metrics and Design Basis Threat to overcome scenario spinning and fear mongering


Industry absorption of effective metrics for realistic threat and risk analysis in IT is moving far too slowly. A 2003 article, Information security: why the future belongs to the quants, contained a useful metric, Business-adjusted risk (BAR), "for classifying security defects by their vulnerability type, degree of risk, and potential business impact." The BAR used Risk of exploit ("how easily an attacker can exploit a given defect") and Business impact ("the damage that would be sustained if the defect were exploited"). The BAR's use of "relative ratings for both likelihood of occurrence and business impact [allowed it to behave] similarly to insurers’ annual loss expectancy calculations."

Four years on, the quants are still waiting while scenario spinning and FUD continue to flow from the unskilled or the commercially craven; Too many members of management, IT included, are among the former while too many security vendors populate the latter. A co-author of that 2003 piece, Andrew Jaquith, has recapitulated and expanded his work in security metrics in Security Metrics: Replacing Fear, Uncertainty, and Doubt, providing a one-stop shop for defining and implementing IT metrics for risk. It has merit to me as the metrics can form inputs to a Design Basis Threat (DBT) calculation for IT in place of the fear mongering from certain security firms. (Expansion for special nuclear material here.) There are threats, numerous and growing, but often not the threats solvable by the security products on offer. Worse, too many firms, Symantec among them, sell products that are consumptive of system resources while providing attack windows in their own code. Enterprise clients are generally deprived of a realistic means of identifying and interdicting realistic, often trivial, penetrations of their infrastructure.

I refer readers to The danger of confusing terrorist interdiction with the consequences of terrorist action for the perils inherent in pursuing scenario-based responses, and, as a start, to FEMA 452 - Risk Assessment: A How-To Guide to Mitigate Potential Terrorist Attacks for its introduction to assessment of threat, asset value, vulnerability and risk.

I fear that Jaquith's efforts have been ignored in the main as Escaping the Hamster Wheel of Pain which forms the first chapter of Security Metrics has been around since May 2005 as has his criticism of Symantec (easily 2005) and a useful but overlooked The Vulnerability Supply Chain (also 2005).

Useful metrics have been out there but have not been picked up en masse, but then neither has DBT, especially in its pure form used on the weapons side of DoE as opposed to the scenario laden approach on the nuclear power side. The combination of effective metrics shorn of histrionics with the Design Basis Threat process offers a realistic means to enterprises needing to formulate a cost effective and sustainable defense posture. We are among the few that have successfully applied DBT to Intellectual Property (IP) threats and remediation.

It cannot be overemphasized that the solution to this problem is NOT an Information Technology (IT) solution but IS primarily a Counterterrorism (CT) and Counterintelligence (CI) solution applied to corporate infrastructure, augmented by IT as the CT/CI process demands. Were it solely an IT solution, then one might suppose that this class of problem could be solved at least as often as major IT applications succeed (which depending upon whose statistics one chooses to accept, some 40 to 60% of large IT solutions either fail, are withdrawn, or at best suboptimal in their performance). The solution path can only be hinted at in this brief survey and the requisite CT/CI practionership and its understanding of an asymmetric attacker takes years to develop (which is one of the reasons that it occurs in so few instances and why the market tolerates so many pretenders as the clients cannot properly estimate the skill set needed to address the problem).

It is also a substantial systems analysis problem. In asking Scott Borg for a current copy of the Cybersecurity Checklist, I noted that I refer clients to his PPT, The New US-CCU Cyber-Security Check List, and its flagged need to address both physical and IT/cybersecurity, but add the following to it:

  • (First I have to describe Ackoff's three laws of systems - people can grasp the first two but the third floors them)
  • Systems fail at their boundaries, and that includes boundaries between components and clusters of components that act as subsystems.
  • Physical and cyber are two of those subsystems; there are many more, all interacting to Ackoff's third law.
  • A check list is a still frame from a motion picture, but people rip the frame, losing the underlying assumptions and context in the process.
  • A check list without a date/time stamp is useless, even dangerous.
  • Process-based threat and vulnerability assessment are key in defining appropriate levels of protection; remediation steps are then pulsed to insure that they deliver against the threats.
  • Scenario-based defense, while useful in estimating consequences of a particular scenario, is dangerous as it spins out of control, usually missing the fatal payload.
  • Good security is process-based rather than hardware-based (process is 10:1 over hardware, and process comes first as it will define the needed hardware).
  • Defenders never see themselves as attackers do, especially asymmetrical attackers, and so rarely protect the right mix against legitimate threats.
  • Defenders too often look for "peer attackers" instead of a simple asymmetric.

Scott's reply mirrors our own experience:

You are right in pointing out how hard it is for most people to think in terms of dynamic systems and processes.  I like the way you have formulated the problem in your e-mail.  We have been struggling with many of the same issues when it comes to getting people to understand the problems they will increasingly face.

The following is derived from an unclass analysis, Asymmetric Threat Detection in the Material Security Environment, we performed for a DLA unit in 2005. Seasoned practitioners will easily envision frontloading Jaquith's metrics into the threat side of DBT.

Evolving Nature of Threats

Technological surges in many sectors, so many as to effectively shield the collective effect from many investigators, coupled with globalization, the availability of WME (weapons of mass effect) has changed the risk landscape, most notably in the means to effectively address low-probability, high-consequence threats.

Too many fail to properly differentiate threat from risk, i.e., a threat is a source of harm (loss) whereas a risk is the estimation of the likelihood of that harm occurring coupled with the potential impact from its occurrence. Threat assessment is only one aspect of a larger and more complex risk analysis process, yet too many remain fixated on threat analyses as the sole basis of applying protective measures without sufficient attention paid to precision or control in their application.

Too many designs for low-probability, high-impact threat sources tend to skew the design of the security plan to costly countermeasures when precision could have provided cohesion and freed up resources. Too often, an organization adopts what it assumes is an extremely ‘secure’ system that either cannot be implemented, cannot be sustained, is impractical for its users or overlooks active threat paths because finite resources are fully engaged elsewhere.

Threat Levels

A threat can be defined as the intended potential to cause an undesirable consequence. The result of a threat assessment documents the result of an analysis of the credible motivations, intentions, and capabilities of potential adversaries that could cause undesirable consequences... The threat level provides a current estimate of ongoing risk to personnel, facilities, or interests from terrorist attack. Analyses deriving threat levels at Department of Defense (DoD) are commonly performed by the intelligence staff at each command level, and resulting threat levels can differ by echelon. Threat Levels range from Negligible to Critical, are based on a systematic analysis of the factors of existence of terrorism, terrorist capability, history of terrorism, intentions of terrorist groups, and targeting by terrorist groups. The system is not perfect but can be effective in a relatively contained risk environment, as it inherently allows for a concentration of resources for periods of elevated risk, conserving those resources in the process.

Threat Analysis

To supplement a risk responsive approach, such as in the use of threat levels, ICG prefers to create a risk matrix for each identified threat group so as to perform a more precise capabilities analysis. ICG prefers this more extensive version -- as it allows greater ability to profile the group under examination and to create a baseline for ongoing comparative analysis, a means to capture outlier data that may indicate an emerging threat:

Variant 2: Threat Analysis Factors

Factor must be Present: X; Factor may or may not be Present: O


































In response to threat levels, companies or commands adopt or change Force Protection Conditions (FPCONs), which are measures to protect people and facilities from the postulated current threat. Each FPCON potentially entails increasingly stringent security measures. A nominal DoD matrix contains intelligence assessments, warning reports, spot reports and law enforcement reports. The Department of State (DoS) adds broader factors, such as political violence which encompasses terrorism, counterintelligence, anti-U.S. technical intelligence, and activities against the U.S. community in determining its threat levels.

Risk is a function of threat, likelihood, consequence, vulnerability, and asset value. Impact is a function of:

  • Resources (the adversary's resources to execute and the defender's resources to defend, respond and recover post-attack)
  • Unexpected Methods by the adversary
  • Adversary's understanding of our infrastructure and the means to achieve exploitation
  • Defender's vulnerabilities
  • Effect Multipliers -- Where typical effect multipliers are:
    • Disruption of cyber infrastructure
    • Prevention or reduction of response and retaliation
    • Decrease or suppression of initiative to respond politically
    • Employment of psychological operations (Psyops)
    • Generation of fear and indecision
    • Introduction of WME (Weapons of Mass Effect)

Asymmetrical Rules Base (Attacker Rules)

Crafted from the 'success' of an earlier World War I static defensive war, the French Maginot Line failed under the newer concept of a mobile mechanized infantry. Accordingly, current defenses will fail under attack by the small-scale, high-impact operations of an asymmetrical attacker employing unexpected, non-traditional and broadly applicable methods unless we learn the current methods of the adversary and adopt simple effective measures.

Threat assessment must include the ability to impute an asymmetrical rules base as part of the threat definition so as to permit the defender to think more like a terrorist (as opposed to a defender) in defining a realistic threat posture, i.e., act without the self-imposed rules and limitations of the defender so as to view the risk calculation through the eyes of any number of threat groups, be they Muslim fundamentalists, Patriot right, Millennialists, single-interest terrorists such as the Earth Liberation Front (ELF), or various groups aggrieved at US actions. Each threat group has 'rules' such as preferences in targets and timing, varying motives for action, specific means or technical capability for action, and the later the threat detection the greater the threat group’s opportunity for action.

Asymmetric adversaries employ very different variables in their calculations for risk than the defender where the adversary is essentially interested in forestalling detection and accomplishing mission fulfillment. As previously noted in threat definition, a study of each category of attacker and, in specific cases, individual adversary groups, will identify a typology of action such that we can view risk and reward through the eyes of the asymmetric attacker. Without that view, much of successful defense is happenstance.

Introduction to Design Basis Threat (DBT)

The successful approach to defer (delay hostile efforts), deflect (move hostile intent to another target) or defend (interdict an incipient hostile attack) against an asymmetric attacker is almost all proactive process with a modest amount of strategically placed hardware that adds specific and reliable value to the process.

The core of that process is the Design Basis Threat (DBT) that will capture and formulate risk management objectives that balance commercial and security objectives, providng a means to evaluate threats over time. The DBT becomes an integral, inseparable part of corporate governance. The DBT becomes the mechanism that informs management of the types of threats it may face over time and allows them to define the threats that are in or out of scope, the response level that will be committed to each threat, and the cost for that response level.

The DBT absorbs the 5-Step Risk Management Process of FM 100-14, Risk Management, which is the commander’s principal risk reduction process to identify and control hazards and make informed decisions:

  • Identify hazards
  • Assess hazards
  • Develop controls and make risk decisions
  • Implement controls
  • Supervise and evaluate

The DBT, just as all sound risk management, does not:

  • Inhibit the commander’s and leader's flexibility and initiative
  • Remove risk altogether, or support a zero defects mindset
  • Require a GO/NO-GO decision

The DBT will include threat assessment, a safety-oriented hazard assessment, asset value assessment and an asset risk assessment that draw upon technical insights and the results of internal and external pattern detection. Where the best DBT implementations differ from almost all conventional DBTs is that the DBT must NOT be a scenario-based risk process but rather a rigorous procedural analysis. As noted above, a solution to IT risk identification and remediation is not solely an IT solution but rather the application of a CT/CI approach to a firm's infrastructure, augmented by IT as required. The DBT process is used to assess risk more effectively, enshrining speed to flag rising risk for inspection and action.

The DBT process can be used also to identify security guidelines that should be migrated across supplier relationships on both the buy (outsourcing) and make (manufacturing) side. Upstream outsourcing is a too often overlooked failure point. See Multisourcing: belated recovery of forgotten first principles, parts 1 and 2.

If history is any guide, integration, implementation and wider adoption of IT metrics DBT will be slow while phishers and penetrators will lunge ahead (here and here), but at least the path is there.

Part 2, Generic elements and process of a DBT protection system

Security Metrics
Posted by samzenpus on Wednesday May 16, @03:35PM
May 16, 2007

8 Questions For Uncovering Information Security Vulnerabilities
Tips for testing information security vulnerability hypotheses with questions designed to head off potential problems.
By Andrew Jaquith
16 May, 2007

Google: 10 percent of sites are dangerous
By Tom Espiner,
Published on ZDNet News
May 15, 2007, 7:56 AM PT

Do you know what’s leaking out of your browser?
Posted by Ryan Naraine @ 11:22 am
Zero Day
May 14, 2007

Using Metrics to Diagnose Problems: A Case Study
When initially deploying transactional financial systems it's wise to make sure perimeter and application defenses are sufficient.
By Andrew Jaquith
11 May, 2007

Models for Assessing the Cost and Value of Software Assurance
John Bailey, Antonio Drommi, Jeffrey Ingalsbe, Nancy Mead, Dan Shoemaker
Software Engineering Institute,
Carnegie Mellon University
Last modified 2007-05-10 10/07 4:38:24 PM

Security Metrics: Replacing Fear, Uncertainty, and Doubt
Andrew Jaquith
Addison-Wesley Professional; March 26, 2007
ISBN-10: 0321349989

ebook: ISBN: 0321509471
File Size: 4393 kb
Released online for download: 03-03-2007

Making the Business Case for Software Assurance
Nancy R. Mead
Software Engineering Institute,
Carnegie Mellon University
2007-02-06 12:30:16 PM

Victor-Valeriu PATRICIU, Iustin PRIESCU, Sebastian NICOLAESCU
Department of Computer Engineering
Military Technical Academy, Bucharest, Romania
Journal of Applied Quantitative Methods
JAQM, Vol 1, No. 2, Winter 2006

Rational Choice of Security Measures via Multi-Parameter Attack Trees
Ahto Buldas, Peeter Laud, Jaan Priisalu, M¨art Saarepera, and Jan Willemson
In J. Lopez, ed.
Proc. of 1st Int. Wksh. on Critical Information Infrastructures Security, CRITIS '06 (Samos Island, Aug./Sept. 2006), pp. 232-243. Univ. of the Aegean, 2006

NOTE: The following PDF of a PPT presentation by Buldas et al is useful for stepping a reader through the attack tree process under discussion:

Rational Choice of Security Measures via Multi-Parameter Attack Trees
Ahto Buldas, Peeter Laud, Jaan Priisalu, M¨art Saarepera, Jan Willemson
August 30 – September 2, 2006, Samos Island, Greece

Checklist outlines new cyberthreats
BY Michael Arnone
Published on April 26, 2006, updated at 5 p.m. May 5, 2006

The New US-CCU Cyber-Security Check List
Scott Borg
GSC-11 Chicago

The Vulnerability Supply Chain
by Andrew Jaquith
6 December, 2005
last changed on 00:06 07-Dec-2005

Asymmetric Threat Detection in the Material Security Environment
With Initial Recommendations Regarding Disposition of WMD-Related End-Items For Defense Reutilization and Marketing Service
Prepared by Intellectual Capital Group LLC
21 September, 2005

The Symantec Threat Report: Read Between the Lines
by Andrew Jaquith
September 20, 2005
last changed on 09:51 22-Sep-2005

A Few Good Metrics
Information security metrics don't have to rely on heavy-duty math to be effective, but they also don't have to be dumbed down to red, yellow, green. Here are five smart measurements—and effective ways to present them.
By Scott Berinato
July 2005

Escaping the Hamster Wheel of Pain
By Andrew Jaquith
4 May, 2005
Last changed on 11:56 04-May-2005

The Metrics Quest
Under pressure from the CFO to quantify security benefits, a CSO finds measures that matter
November 2004

Nuclear Security: DOE Must Address Significant Issues to Meet the Requirements of the New Design Basis Threat.
Testimony Before the Subcommittee on Oversight and Investigations, Committee on Energy and Commerce, House of Representatives
GAO-04-773T, General Accounting Office (GAO)
May 11, 2004

Collecting Effective Security Metrics
By Chad Robinson
Robert Frances Group
April 09, 2004

Information security: why the future belongs to the quants
Daniel Geer Jr, Kevin Soo Hoo, Andrew Jaquith
Security & Privacy Magazine, IEEE
Volume 1, Issue 4, July-Aug. 2003 Page(s): 24 - 32
Posted online: 2003-08-11 14:23:28.0
ISSN: 1540-7993


Risk Management
FM 100-14
Field Manual Headquarters
No. 100- 14 Department of the Army
Washington, DC, 23 April 1998

Gordon Housworth

Cybersecurity Public  InfoT Public  Infrastructure Defense Public  Intellectual Property Theft Public  Risk Containment and Pricing Public  Strategic Risk Public  Terrorism Public  


  discuss this article

<<  |  March 2017  |  >>
view our rss feed