Open-source intelligence

Open source intelligence or "OSINT" refers to an intelligence gathering discipline based on analyzing information collected from open sources, i.e. information available to the general public. These sources include newspapers, the internet, books, phone books, scientific journals, radio broadcasts, television, and many many others generally ignored by established "secret" intelligence organizations that are culturally and economically "invested" in the cult of secrecy. The most important open sources are human sources, and especially indigenous individuals who have local knowledge, and world-class experts who are more often than not foreigners--by definition, these latter two categories are "forbidden" for direct access out of mis-placed security concerns. The best OSINT is done by private sector parties that are truly internationalist and able to engage and integrate knowledge from the "eight tribes" of global intelligence: government, military, law enforcement, business, academia, non-governmental organizations, the media, and civil socities inclusive of citizen advocacy organizations, labor unions, and religions.

The term OSINT is related to the term open source as it applies to the five "opens" (open source software, open source intelligence, open spectrum, open access copyright, and Open Society) but in this context, is distinct from the much better organized and much more reliable free/open source software (F/OSS) social network thst shares source code which is publicly available (and modifiable). OSINT should also not be generally confused with OSIF (Open Source Information) on which OSINT is based. OSIF is any information that is publicly available; OSINT is an analytically-tailored intelligence product composed of OSIF which is designed to answer a specific tasking or to support decision-making.

Outside of the secret world, OSINT is considered an all-source discipline that integrates overt human, imagery, signals monitoring, and historical-cultural open source information. Inside the secret world, OSINT is considered subordinate to the secret aspect of each discipline, and is generally either disregarded or "classified" and withheld from the public by virtue of its association with the secret discipline. The US Intelligence Community in particular has an information technology architecture, as well as a management mind-set, a legal mind-set, and security mind-set, that demands that ALL information be stored (and generally not shared) at the Top Secret Codeword level. This is one reason why 9-11 occured, and according to several books now published, this lack of sharing has still not improved.

Although intelligence is thought to deal only with secrets or secretive information, this is a misunderstanding of the term. Intelligence can be best understood as "decision support," and can be accomplished by anyone trained in the proven and not secret process of intelligence: requirements definition, collection management, source discovery and validation, multi-source fusion, and compelling actionable timely presentation. As a general rule, the best OSINT is done by energy and pharmaceutical companies where great fortunes are at stake--all others are in roughly the 4th or 5th grade, with secret intelligence organizations being in the 2nd grade, at best.

The foremost proponents in the world for OSINT advocate the view that the public is poorly served by governments that obsess on secrecy and insist on spending 80% or more of their funds on secret sources and methods. In the U.S. Governmnent, for example, the secret budget is well over $50 billion a year, some say as much as $70 billion a year, and less than $600 million a year is spent on open sources and methods of information. It is the view of the OSINT pioneers, some of them with decades of classified world experience, that we need to flip the government intelligence world from its present position of being "inside out and upside down" so that 80% of the focus is on open sources of information that can be shared across national, agency, discipline, and domain boundaries. This is not only cost effective, it creates the "wealth of knowledge" for publics that is now denied to them by the wasteful and generally ineffective secret sources and methods that are not held accountable for failure.

Collection in OSINT is generally a different problem from collection in other intelligence disciplines where obtaining the raw information to be analyzed may be a major difficultly, particularly if it is to be obtained from non-cooperative targets. In OSINT, the chief difficulty is in identifying relevant, reliable sources from the vast amount of publicly available information. However, this is not as great a challenge for those who know how to access local knowledge and how to leverage human experts who can create new tailored knowledge on the fly. The needle in the haystack problem generally only afflicts those dumb enough to demand that everything be digitized and made sense of, rather than those who know how to get directly to the offline indigenous source with the answer.

Overt Human Intelligence (HUMINT) is the use of non-clandestine human information sources that are approved by security officers who have very limited educational and real-world backgrounds, for example, the interrogation of refugees, debriefing of legal travellers, and public reports from overt agents such as attachés and ambassadors. Truly professional OSINT practitioners understand that EVERY human being, including bus drivers, secretaries, priests, teachers, and so on, have something to contribute to historical, cultural, or contextual understanding. Similarly, the most effective intelligence managers understand that the best intelligence professionals will often have complex international backgrounds and will have spend decades dealing with foreigners--these managers are often limited to entry level employees who have no foreign experience or foreign contacts, simply because they are easier for the security officers to approve for employment. In a robust international OSINT and information sharing environment, such pathological approaches to employment and interaction are moot.

OSINT is now a subordinate part of modern Information Operations (IO), which consists of Strategic Communication (the message), OSINT (the reality), and Joint Intelligence or Inter-Agency Collaboration or Coordination Centers or Commands (JIOC). The U.S. military is heavily invested in OSINT, with the U.S. Strategic Command having the lead for IO, the U.S. Special Operations Command having the lead for the best operationally-oriented OSINT capability, and the U.S. Central Command being the most important consumer of tailored operationally-oriented OSINT.

The current definitive guides to OSINT are the NATO Open Source Intelligence Handbook, the NATO Open Source Intelligence Reader, and the NATO Intelligence Exploitation of the Internet guide. A history of OSINT in recent times is contained in the 30 volumes of Proceedings from the annual OSINT conference sponsored by OSS.Net

Why Open Source Intelligence is Important to the Public and to National Security:

According to The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction report submitted in March 2005, OSINT must be included in the all-source SECRET intelligence process for the following reasons (as stated in the report):

1. The ever-shifting nature of our intelligence needs compels the Intelligence Community to quickly and easily understand a wide range of foreign countries and cultures. - … today’s threats are rapidly changing and geographically diffuse; it is a fact of life that an intelligence analyst may be forced to shift rapidly from one topic to the next. Increasingly, Intelligence Community professionals need to quickly assimilate social, economic, and cultural information about a country—information often detailed in open sources.

1. Open source information provides a base for understanding classified materials. Despite large quantities of classified material produced by the Intelligence Community, the amount of classified information produced on any one topic can be quite limited, and may be taken out of context if viewed only from a classified-source perspective. Perhaps the most important example today relates to terrorism, where open source information can fill gaps and create links that allow analysts to better understand fragmented intelligence, rumored terrorist plans, possible means of attack, and potential targets.

1. Open source materials can protect sources and methods. Sometimes an intelligence judgment that is actually informed with sensitive, classified information can be defended on the basis of open source reporting. This can prove useful when policymakers need to explain policy decisions or communicate with foreign officials without compromising classified sources.

1. Only open source can “store history.” A robust open source program can, in effect, gather data to monitor the world’s cultures and how they change with time. This is difficult, if not impossible, using the “snapshots” provided by classified collection methods. (The Commission on the Intelligence Capabilities, 378-379).

The United States National Commission on Terrorist Attacks Upon the United States issued a report in July 2004 recommending the creation of an open-source intelligence agency, but without further detail or comment. Subsequently, the WMD Commission (also known as the Robb-Silberman Commission) report in March 2005 recommended the creation of an Open Source Directorate at the CIA.

Both of these reports, while recent, are relatively superficial, and both ignore the decades of advocacy for a proper national focus on OSINT from 1988 to date. They also err, severely, in assuming that the same Central Intelligence Agency that has obstinantly refused to take open source information seriously for decades, should be charged with developing new capabilities that are totally outside its existing culture and cult of secrecy.

An in-depth account of modern "The OSINT Story" is available. Below are a few highlights, first of public literature on emergent collective public intelligence, and then on the 25-year war between OSINT proponents and the secret US Intelligence Community that began in 1988.

In 1938 H. G. Well published "The World Brain" as a vision for the future that is emerging today, where the Internet makes it possible for a Wikipedia to emerge that not only has the potential to publish free useful knowledge on all topics, but also to bring together social networks of interested parties with varying levels of expertise. Since then a number of books have emerged including Pierre Teilhard de Chardin's "The Phenomenon of Man" with its concept of "noosphere," and more recently, books such as Howard Rheingold's "Smart Mobs," James Surowiecki's "Wisdom of the Crowds," and Glen Reynolds, [http//www.amazon.com/exex/obidos/ASIN/0385721706/ossnet-20 "An Army of Davids."]

Today the single most important source on the collective intelligence movement is the Co-Intelligence Institute founded by Tom Atlee, who is also the author of "The Tao of Democracy: Using Co-Intelligence to Create a World that Works for All.

Within the secret world of intelligence, where other nations are often smarter about how to collect intelligence inexpensively, but where the USA, because of its massive budget, is the only power whose wisdom, if applied, could impact in very positive manner on the rest of the world, the OSINT war began in 1988. As with most major innovations seeking to reform massive bureaucracies, this will be a 25-year war with victory by the public expected in 2012-2013.

In June 1988 Mr. Robert David Steele, under the direction of Col Walter Breede, III, USMC USNA '68, established the Marine Corps Intelligence Command (then Center). By September 1988 it was clear to the Marine Corps that 80% of what it needed to produce third world and low-intensity conflict intelligence was not secret, not in English, not online, and not known to anybody in the Washington, D.C. area. After the U.S. Intelligence Community refused to listen to Marine Corps officil appeals to the National Foreign Intelligence Board and the Military Intelligence Board, then Commandant of the Marine Corps General Alfred M. Gray published an article,"Global Intelligence Challenges of the 1990's," in the American Intelligence Journal (Winter 1988-1989) calling for the establishment of a national open source intelligence capability, and more attention to revolutionary threats including terrorism. Mr. Steele worked from 1988-1992 attempting to gain official understanding of this need. When the bureaucracy refused to listen, he resigned from the Marine Corps and established an annual international conference, Open Source Solutions (now IOP for IO, OSINT, and Peacekeeping Intelligence). This event has brought together over 7,500 international professionals, most of them military, and there are at least seven countries that as a result are light years ahead of the United States in this critical arena, including China, India, Norway, Russia, Sweden, South Africa, and Singapore. Over 600 authorities on OSINT have presented papers, all of which are provided free to the public at OSS.Net.

In Fall 1992 Senator David Boren, then Chairman of the Senate Select Committee on Intelligence, sponsored the National Security Act of 1992, attempting to achieve modest reform in the U.S. Intelligence Community. His counterpart on the House Permanent Select Committee on Intelligence, Congressman McCurdy, aided by Staff Director Mark Lowenthal, who led a masterful study called [www.access.gpo.gov/congress/house/intel/ic21/index.html "IC21: The Intelligence Community in the 21st Century"] included a separate Open Source organization in the bill (proposed legislation). This legislation was defeated by a combination of opposition from Senator John Warner of Virginia, who feared that reform would reduce intelligence jobs in Virginia (this fear is misplaced--every classified job that is eliminated can fund two or more unclassified positions), and a letter from then Secretary of Defense Dick Cheney that severely mis-represented the dangers to our national security of transferring the three *national* intelligence agencies out of the Department of Defense to a truly independent intelligence community. Within the past several years, LtGen Dr. Brent Scowcraft studied this issue again and repeated the recommendation that the three national agencies be removed from the Department of Defense, only to be dismissed by now Vice President Dick Cheney.

In August 1995 the Aspin-Brown Commission sponsored a benchmark exercise on Burundi in which a private sector party, with six telehone calls, overnight, completely crushed the entire U.S. Intelligence Community, producing the following:

-- From the Institute of Scientific Information (ISI), the top 100 international experts on Burundi available for immediate debriefing;

-- From LEXIS-NEXIS, the top international reporters on Burundi, available for immediate debriefing.

-- From Oxford Analytica, twenty two-page reports on the political-military significance of Burundi at the Presidential, Prime Ministerial, and CEO levels;

-- From Jane's Information Group, one paragraph summaries of every article ever published about Burundi, and new, tailor-made tribal orders of battle created overnight for this need

-- From East View Cartographic, 1:50,000 meter combat charts with contour lines from Russia (the Americans do not have combat charts for 90% of the world--they rely too heavily on secret imagery satellites_.

-- From SPOT Image in France, commercial imagery for Burundi, 100% coverage, cloud-free, less than three years old, in the archive and inexpensively available.

The CIA (which sponsors the Open Source Center today) had a small map of Africa and a regional economic study with flawed (Western) premises. There was no clandestine human intelligence, no imagery intelligence, and no signals intelligence, and of course no open source intelligence either.

Several DCI's in succession refused to implement the Aspin-Brown Commission recommendations, including the recommendation in 1996 that stated that US access to open sources was "severely deficient" and that this should be a "top priority" for both funding and DCI attention. Consequently, the US entered the new century completely ignorant of all of the open source information in all languages relevant to identifying and containing Al Qaeda, which has been active, publicly, since 1988.

In July 1997 then DCI George Tenet received the report, "The Challenge of Global Coverage" conducted by Senior Intelligence Service Boyd Sutton. After interviewing virtually all of the Assistant Secretaries of Defense and State, and the heads of the varied intelligence organizations, the report recommended that the U.S. Intelligence Community, which obsesses on seven "hard targets" and continues to ignore the Third World and lower tier topics including the ten topics identified by the United Nations report of 2004, "Creating a More Secure World," spend $1.5 billion dollars a year on OSINT--$10M for each of 150 lower tier countries or topics where most of the instability was located, but which did not, to the denied area mindsets at the time, represent cataclysmic threats to the USA.

Two years after 9-11, through the heroic efforts of a Senior Intelligence Service officer serving on detail to the 9-11 Commission, and because The Honorable Lee Hamilton has witnessed the ignominious defeat of the U.S. Intelligence Community on the topic of Burundi, page 413 of the 9-11 Commission Report included a recommended Open Source Agency completely independent of the Central Intelligence Agency.

In December 2005, the Director of National Intelligence appointed Eliot A. Jardines as the Assistant Deputy Director of National Intelligence for Open Source to serve as the Intelligence Community's senior intelligence officer for open source and to provide strategy, guidance and oversight for the Community's open source activities. In November 2005, the Director of National Intelligence (DNI) announced the creation of the Open Source Center. The OSC’s functions include the collection, analysis and research, and training to facilitate government-wide access and use of OSIF. The OSC builds on the expertise of the CIA’s Foreign Broadcast Information Service (FBIS), which has provided the U.S. Government a broad range of open source products and services since 1941 (including world media content available to the public through World News Connection).

Unfortunately, Mr. Jardines has no program authority, no funding outside of $5M a year (another $20M a year funds the Open Source Center's Large Scale Internet Exploitation (LSIE) initiative), and he has no staff other than two rotational employees. While he oversees roughly $250M a year in "old" capabilities such as the Foreign Broadcast Information System (FBIS), he has not been given the authorities or funds needed to actually create a competent national open source intelligence capability. Until such time as there is an international Open Source Agency and network funded at no less than $1.5 billion a year (including $600M for commercial imagery raw source procurement), the U.S. Intelligence Community should be considered completely ineffective in this arena.

Several other nations maintain open source intelligence gathering operations, for example Australia's Office of National Assessments (ONA), the UK's BBC Monitoring Service (BBCM) and the Swiss Army GeneralStab. Collection and analysis tasks are allocated in different ways by different agencies: for example, the ONA is responsible for analysis and is an Australian government intelligence agency, while the BBCM is government-funded civilian organization concerned solely with collection and aggregation of news sources, and which employs civilian journalists. The "allied" nations tend to subordinate and disrespect OSINT, while the Third World nations have learned to rely on it, partly because that is all they can afford.

In a speech at the Council on Foreign Relations in February 2006, Secretary of Defense Donald Rumsfeld seems to have acknowledged the importance of open media as a component of national security in the information age. Secretary Rumsfeld's Undersecretary of Defense for Intelligence, Dr. Stephen Cambone, told the Security Affairs Support Association (SASA) in / January 2004 that he required nothing less than universal coverage in all languages, 24/7. It is Dr. Cambone, rather than the CIA, that has been forcefully addressing the urgent needs of the military and other agencies for OSINT in support of defense policy-making, defense acquisition, and defense operations.

See also

• Category:Intelligence websites

References

NATO Open Source Intelligence Handbook (2001)

NATO Open Source Intelligence Reader (2002)

NATO Intelligence Exploitation of the Internet (2002)

OSS.net - from Robert D. Steele, largest and most comprehensive privately owned OSINT network and website, updated continuously.

One Page Link Guide to OSINT References - Covers history, context, process, practice, and tools

Terms of Reference for Future of Intelligence - provides matrix for the war between secret and open intelligence proponents regarding funding, priorities, and methods.

Multinational Information Operations Center - Briefing on Moving to the Next Level

Information Operations: The Book - Briefing on Book that Takes IO to the Next Level

Annual IO/OSINT/PKI Conference - Briefing on Annual Conference (since 1992) for Forward Thinkers

IO: Putting the I Back Into Dime - Monograph from the Strategic Studies Institute of the US Army War College

The New Craft of Intelligence - Monograph from the Strategic Studies Institute of the US Army War College

Dr. John Nomikos, The Role of Open Sources in Intelligence, ISRIA PDF, December 31, 2005.

Google News Most Recent Hits

Google Web Most Popular Hits

Map of OSINT Sites (Search for Open Source Intelligence OSINT)

External links (Vendors)

• BBC Monitoring Subscription service from the BBC.
• C4I.org- Private open-source intelligence clearinghouse.
• Dialog is the worldwide leader in providing online-based information services to organizations seeking competitive advantages in such fields as business, science, engineering, finance and law.
• EastView Since 1989, East View has been the world's leader in providing high-quality information services from Russia and the NIS to the widest variety of customers.
• Factiva a Dow Jones & Reuters Company, provides world class global content, including Dow Jones and Reuters Newswires and The Wall Street Journal.
• Infosphere AB- Corporate OSINT from Commercial Intelligence & Knowledge Strategy consultancy Infosphere AB (Sweden).
• Lexis Nexis provides authoritative legal, news, public records and business information; including tax and regulatory publications in online, print or CD-ROM formats
• MarketResearch.com is the leading provider of global market intelligence products and services.
• Open Sources Center Subscription-based OSINT from ISRIA.
• The OSINT Center Geopolitical, Tourism and Energy OSINT from Alan Simpson
• SENTINEL- Government and private sector OSINT services from Elsag Solutions AG (Switzerland).
• Silobreaker- integrated sources and analytical webservice created by and for OSINT and CI users- Intelligence for Everyone (2005)
• World News Connection - global news from Federal Broadcast Information Service (FBIS is run by CIA, now under the DNI Open Source Center(OSC)).