Advanced Analytics + National Security, Part One
We look at the intersection of advanced analytics, open-source data and National Security.
Whether you are in the national security community or not, we believe that there are useful lessons here for leaders in many industries. And, while big data may have been born in the business world, the size and scope of data in the USG – especially in the national security agencies — may mean the next big thing in big data will start in government and migrate into the commercial space.
When it comes to big data, there is nothing bigger than the U.S. government. It produces, collects, stores, and analyzes more data than any other entity, public or private, in the world. And within the USG, the challenges and opportunities loom the largest in the national security community given the volume and the complexity of data – from sensors and imagers to human intelligence to social media and more – these agencies deal with every day. While late to the game by many measures, the U.S. Government launched a “Big Data Research and Development Initiative” last spring, investing $200 million in big data among federal agencies. In addition, the Pentagon alone has already been spending $250 million annually on big data innovation.
In a recent survey of IT leaders in public-sector organizations, co-sponsored by SAP and TechAmerica, federal IT managers identified Defense and Intelligence community as the “gold standard” in the use of big data. Survey respondents cited the tremendous scale of information that these agencies produce, collect, and analyze. The survey, however, did not paint a full picture of the unprecedented analytical challenges these communities face – combining big & small data and reconciling data collected first-hand with second-hand data shared by partners, and third-party open source data that is often unstructured. (In a recent Wall Street Journal blog post, Tom Davenport discusses similar challenges from a private sector perspective, in what he calls Analytics 3.0).
Why is this so complex?
It is largely because of the break neck speed at which open source data is increasing and digital content is diversifying. Over the past three years the number of Internet users has soared from 1.6 billion (less than 25% of the world’s population) to 2.28 billion (over 30% of the world’s population). During this time, we have also witnessed a stunning increase in social media content and “location aware” data, increasing the richness of open source intelligence. Yet, because only a fraction of the Web is indexed, we use only a narrow slice of this data.
What are the challenges?
The rapidly growing volume and complexity of digital data makes it difficult to harness and exploit effectively. Analysts pull from only a small subset of available data to avoid being overwhelmed. A shocking amount of information is “left on the table.” Hidden in this unexploited data are national security trends, threats, and opportunities. For example, communications among terrorists can go undetected, analysts miss threats and lives are endangered. Also, analysts can miss trends hiding in the data that could point to social or political upheavals – as they did in the weeks and months leading up the Arab Spring.
What are the opportunities?
If harnessed and contextualized properly, the explosion in data presents analysts and planners with new opportunities to translate information into insight and offer high-level decision support not only to the warfighter but to senior policy makers as well. At a tactical or operational level, if an analyst were able to “see through the data” he or she would be able to detect threats before they unfold. However, the benefits of exploiting large and complex data sets would not stop there; there is potential strategic value as well. Advanced analytical techniques, such as simulation and predictive analytics, would also allow senior decision-makers to evaluate a range of policy options before investing or implementing them. This type of analysis would paint a more complete picture for government leaders to understand the potential implications of national security decisions.
Next in this series of posts will focus on advanced analytical techniques that work in the national security realm and could work for you too…