Investigative Data Warehouse Explained

Investigative Data Warehouse (IDW) is a searchable database operated by the FBI. It was created in 2004. Much of the nature and scope of the database is classified. The database is a centralization of multiple federal and state databases, including criminal records from various law enforcement agencies, the U.S. Department of the Treasury's Financial Crimes Enforcement Network (FinCEN), and public records databases. According to Michael Morehart's testimony before the House Committee on Financial Services in 2006, the "IDW is a centralized, web-enabled, closed system repository for intelligence and investigative data. This system, maintained by the FBI, allows appropriately trained and authorized personnel throughout the country to query for information of relevance to investigative and intelligence matters."Morehart 2005, op. cit.

Overview

The size of the database appears to be growing rapidly. In 2004, according to a government solicitation for bids to manage the project, it was approximately 10TB in size. In 2005, according to one FBI official, the IDW contained approximately 100 million documents. In 2006 it contained more than 560 million documents and was accessible by more than 12,000 individuals. According to the FBI's website, as of August 22, 2007, the database contained 700 million records from 53 databases and was accessible by 13,000 individuals around the world.

, the FBI is the subject of a lawsuit brought by the EFF (Electronic Frontier Foundation) because of a lack of public notice describing the database and the criteria for including personal information, as required by the Privacy Act of 1974. The lawsuits are a result of two Freedom of Information Act requests filed by the EFF in 2006.

It was built in part by Chiliad corporation,[1] [2] the FBI Office of the Chief Technology Officer,[3] and others. Companies listed on the FOIA files include Northrop Grumman [4] and others.

Purpose

Investigative Data Warehouse–Secret (IDW-S) "provides data and data processing/analysis services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and law enforcement missions". The core subsystem supports the Counter-Terrorism Division (CTD), the Special Event Unit, and via DOCLAB-S, the Joint Intelligence Committee Investigation (JICI) and IntelPlus.[5]

According to a 2005 email, "IDW will also be used for criminal and other authorized non-CT investigations as it evolves." (CT being counter terrorism) [6]

Subsystems

Within the system, there were subsystems named IDW-S Core, SPT, and DOCLAB-S[7]

The special projects team (SPT):

allows for the rapid import of new specialized data sources. These data sources are not made available to the general IDW users but instead are provided to a small group of users who have a demonstrated "need-to-know". The SPT System is similar in function to the IDW-S system, with the main difference is a different set of data sources. The SPT System allows its users to access not only the standard IDW Data Store but the specialized SPT Data Store.[8]

Privacy

According to internal emails, the FBI performed several Privacy Impact Assessments (PIAs) of the IDW system. They worked with lawyers from their National Security Law Branch (NSLB) to attempt to make sure their system was complying with various laws regarding sharing of information and secrecy [9] (for example, rule 6e of the Federal Rules of Criminal Procedure, regarding the secrecy of Grand Jury material [10]).

The Information Sharing Policy Group (ISPG) formed a Discretionary Access Control Team (DACT), to work on "approval of data sets" and "access control requirements" for IDW and DataMart, and responding to other Intelligence Community agencies requesting access.[11]

The EFF FOIA IDW website states "Despite the vast amount of personal information contained in the IDW, the FBI has never published a Privacy Act notice describing the system or explaining the ways in which the records might be used."[12] There was also a 2005 email from someone on the Office of General Council (OGC) about "preliminary staff musings that maybe we should limit FBI PIA requirements to non-NS systems" (NS being National Security).[13] There was also an email from 2006 saying that 'national security systems are exempt from E-Gov',[14] apparently referring to the E-Government Act of 2002, which has a section that deals with privacy.

Data sources

The IDW used many data sources. The FOIA documents from EFF are heavily redacted, but some of the sources are as follows:

There was also talk of linking the FTTTF "Data Mart" with IDW.[25]

The data in IDW is classified at the 'Secret' level or lower. Higher classifications are not allowed, and can be removed[26]

See also

References

Sources consulted
Endnotes

Notes and References

  1. Web site: Chiliad Case Study . https://web.archive.org/web/20120524130022/http://www.chiliad.com/docs/ChiliadCaseStudy_FBI.pdf . dead . 2012-05-24 . 2009-03-18 .
  2. Information Week . FBI Shows off Counterterrorism Database . David Gardner . 2009-03-18 . 2006-08-30 . 2011-06-13 . https://web.archive.org/web/20110613021335/http://www.informationweek.com/news/192500531?cid=ref-true . dead .
  3. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 Apr 8, idw01
  4. EFF FOIA files, 2008 Apr 8 idw01, page 27 of linked pdf
  5. FBI, IDW-S System Security Plan, 2005 Jan 24
  6. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA files, 2008 Apr 8 idw02
  7. FBI, IDW-S System Security Plan, 2005 Jan 24. It is unclear from the FOIA documents the difference between IDW-S and IDW, and thus whether Core SPT and DOCLAB-S are under IDW, or IDW-S.
  8. https://www.eff.org/files/filenode/foia_idw/20090609_idw04.pdf FBI, S-CONOPS IDW, 2004 Nov 29
  9. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 April 8 idw02
  10. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 April 8 idw02
  11. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 April 8 idw02
  12. Web site: EFF website, FOIA: DOJ's Investigative Data Warehouse . 2009-03-18 . 2009-03-28 . https://web.archive.org/web/20090328190702/http://www.eff.org/issues/foia/061773RBW . live .
  13. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 April 8 idw02
  14. https://www.eff.org/files/filenode/foia_idw/20090609_idw04.pdf EFF FOIA Files, 2008 Jun 9 idw04
  15. FBI, IDW Privileged Users Guide, 2004 Dec 1
  16. FBI, IDW-S System Security Plan, 2003 Dec 3
  17. FBI IDW Status Update, 2005 Sep 21
  18. FBI IDW Status Update, 2005 Sep 21. 'Open Source News' is, in other documents, referred to alongside MiTAP and/or DARPA TIDES.
  19. Note: Some FBI documents list DARPA TIDES, some list MiTAP, some simply say "Open Source News". They are related projects, if not perhaps the same thing.
  20. [Financial Crimes Enforcement Network]
  21. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA files 2008 Apr 8 idw02
  22. https://www.eff.org/files/filenode/foia_idw/20090609_idw04.pdf FBI S-CONOPS IDW 2004 Nov 29
  23. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 Apr 8, idw02
  24. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 Apr 8, idw01
  25. https://www.eff.org/files/filenode/foia_idw/20080408_idw02.pdf EFF FOIA Files, 2008 Apr 8, idw02
  26. https://www.eff.org/files/filenode/foia_idw/20080408_idw01.pdf EFF FOIA files, 2008 Apr 2, idw01

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "Investigative Data Warehouse".

Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.