return to ICG Spaces home    ICG Risk Blog    discussions    newsletters    login    

ICG Risk Blog - [ Rising awareness and increasing value of 'Commercial-off-the-shelf' tools and companies for intelligence gathering and analytics ]

Rising awareness and increasing value of 'Commercial-off-the-shelf' tools and companies for intelligence gathering and analytics


More a point along a continuum of behavioral shift than a notable milestone, I see the FBI's formal retirement of its packet sniffing tool, DCS 1000 - which debuted with the politically tone-deaf name of Carnivore, in favor of commercially available tools often run by the ISPs themselves as another recognition of changes in the commercial and political marketplace:

  1. In terms of product, what I like to call the 'glide slope to the desktop' of ever increasingly competent COTS tools is allowing federal and state agencies to buy in tools for data gathering, analysis, and synthesis, at lower cost, shorter lead/lag times, and with better bug maintenance and enhancement releases.
  2. In terms of companies, the growth of vertically integrating commercial "intelligence [agencies], gathering data, applying analytics" is allowing federal and state agencies to distance themselves from political redress while availing themselves of low cost information simultaneously being sold to other commercial firms needing credit or background information.
  3. In terms of what I call 'boundary exploitation,' states are increasingly able to make richer data sets available to federal authorities in a manner that does not violate Posse Comitatus, Privacy Act of 1974, and DoD internal policies.

I hasten to add that there is nothing new here, no tectonic shift. See, for example: TSA helped JetBlue share live passenger data with contractor, Privacy and Consumer Profiling, and 'Ask' without 'task': legal circumvention of Posse Comitatus and Privacy Act of 1974.

What does stun me is the price paid in either money or political damage for some custom tools. Carnivore was estimated to have cost "between US$6 million and $15 million." As an analyst and SME (subject matter expert) that works closely with tool developers for pattern detection and data fusion, I am astonished.  It was also prone to embarrassing failures that compromised on-going investigations. The FBI's Virtual Case File (VCF) appears to have failed wholesale in terms of architecture and coding such that functionality, operability, scalability, and extensibility were crippled; the only landmine not publicly stepped on so far is interoperability. The developer, SAIC, is a year-late in delivering but a fraction of the required functionality of what is a $170 million effort, again, an astonishing figure.

Some terms you should remember in terms of tools providers and corporate data citadel aggregators:

  • Data mining (now politically discredited, often replaced by Predictive analytics or Knowledge discovery in database (KDD)): The application of database technology and techniques, such as statistical analysis and modeling, to uncover hidden patterns and subtle relationships in data and to infer rules that allow for the prediction of future results.
  • Dataveillance: Surveillance of large groups of people, sorting large amounts of personally identifying data, to find individuals who might fit a terrorist profile.

DHS CIO, Steve Cooper has long held that government must look outside:

[Pointing] to the financial-services industry's well-developed business processes and technology to exploit data it collects on individual customers as expertise the government could learn from [Cooper says] "There's far more information outside the federal government about us, as individuals, than anybody inside has a clue about. Why not build upon their assembled expertise?" The feds also want state cooperation.

The value of firms such as ChoicePoint become clear when Cooper states that DHS is:

 "looking to a "capture once and reuse many" approach to data gathering, meaning data will be shared among agencies [and that] Data mining would also be a central IT function for homeland security. "What we’re talking about is pattern recognition, or the use of software intelligent agents to peruse data, [which are] driven by algorithms and rules that define themselves over time," Cooper said. Such tools "can marry statistically derived outcomes from known events to predictive models."

The battle over the intentions of ChoicePoint continues. See An insider says EPIC hasn't done homework on credit agencies and Two law professors defend EPIC's letter slamming creditagencies.

Firm Quietly Finds Wealth In Information
By Robert O'Harrow Jr.
Washington Post
January 20, 2005

VCF assessor: Stop funding
BY David Perera
Federal Computer Week
Jan. 18, 2005

Gordon Housworth

InfoT Public  Infrastructure Defense Public  Strategic Risk Public  


  discuss this article

<<  |  May 2020  |  >>
view our rss feed