Open Source Intelligence

author: Clive Best, Joint Research Centre
published: Dec. 3, 2007,   recorded: September 2007,   views: 32543


Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.

 Watch videos:   (click on thumbnail to launch)

Watch Part 1
Part 1 1:06:01
Watch Part 2
Part 2 23:47


Open Source Intelligence can be defined as the retrieval, extraction and analysis of information from publicly available sources. Each of these three processes is the subject of ongoing research resulting in specialised techniques. Today the largest source of open source information is the Internet. Most newspapers and news agencies have web sites with live updates on unfolding events, opinions and perspectives on world events are published. Most governments monitor news reports to feel the pulse of public opinion, and for early warning and current awareness of emerging crises. The phenomenal growth in knowledge, data and opinions published on the Internet requires advanced software tools which allow analysts to cope with the overflow of information. Malicious use of the Internet has also grown rapidly particularly on-line fraud, illegal content, virtual stalking, and various scams. These are all creating major challenges to security and law enforcement agencies. The alarming increase in the use of the Internet by extremist and Terrorist groups has emerged. The number of terrorist linked websites has grown from about 15 in 1998 to some 4500 today. These sites use slick multimedia to distil propaganda whose main purpose is to 1) enthuse and stir up rebellion in embedded communities 2) instill fear in the “enemy” and fight psychological warfare. Anonymous communication between terrorist cells via bulletin boards, chat rooms and email is also prevalent. The Joint Research Centre has developed significant experience in Internet content monitoring through its work on media monitoring (EMM) for the European Commission. EMM forms the core of the Commissions daily press monitoring service, and has also been adopted by the European Council Situation Centre for their ODIN system. A new research topic at the JRC is Web mining and open source intelligence. This applies EMM technology to the wider Internet and not just to news sites. This applies advanced multi-lingual search techniques to identify potential web resources and the extraction and download of all the textual content. This is then followed by automatic change detection, the recognition of places, names and relationships, and further analysis of the resultant large bodies of text. These tools help analysts to process large amounts of documents and derive structured data easier to analyse.

This talk will review 4 main topics:

• Internet trends and the rapid rise of Web 2.0 user generated content • Information retrieval: Live content monitoring of multilingual news reports. Web scraping & RSS feed generation, Web Mining and content monitoring • Information Extraction: Topic filtering, Topic Clustering, multilingual named entity extraction, geocoding and geolocating text, event extraction, opinion mining. • Information Analysis: Social Network derivation, geospatial indexing and analysis, incident tracking databases, statistical trend analysis, threat monitoring and assessment.

See Also:

Download slides icon Download slides: mmdss07_best_osi_01.pdf (11.5 MB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Reviews and comments:

Comment1 valery, November 19, 2009 at 7:29 p.m.:

This video lecrure on open source intelligence is very useful for me but it is also regretable that it is imposible to copy it to my PC. Could you help me in any way to do it ?
With regards,
Valery eschenko, kiev,ukraine

Comment2 Jonathan Painter, September 7, 2016 at 8:02 a.m.:

This is perhaps one of the worst briefings that I’ve ever seen. It is a disjointed 90-minute verbal briefing of perhaps 20 minutes of information. While perhaps technically proficient, the briefer appears to lack the ability to effectively convey key concepts. The briefer is not comfortable speaking to the subject matter and appears to simply be reading PowerPoint slides.

Comment3 Herbert R. Reed, April 5, 2018 at 10:35 p.m.:

I was unable to access the Clive Best video but I did gain access to the slides. My software is up to date to support the video, but for some reason, it would not load. I have been having difficulties ever since the new program has come online here at APUS. I had to change from Safari to Google Chrome and still, there are glitches in the programs.

Herbert R. Reed

Comment4 Jonathan Wallace, October 9, 2020 at 1:39 a.m.:

I tried installing adobe flash player and copied the pasted the link in other browsers and still the video will not play. Request this video gets pushed to youtube or some other player that can allow the student to watch the video.

Write your own review or comment:

make sure you have javascript enabled or clear this field: