Big Data - the Big Mistakes (part four)

“Last week I posted about the connections between big data sources and military intelligence categories. This week I’m looking at big data analytics and the intelligence process”

5. Isn’t Big Data Analytics a Data-to-Decision Process?

Alarm bells always start ringing when I see Big Data architecture presentations which show the analytics process as a conveyer-belt from data to decisions. They crop up far too frequently (just Google images for: “big data process” if you want to see examples). So let’s get the main message over right away:

Big Data Analytics is a Closed Loop, Data and Decision Cycle

We can learn a lot about the desirable elements of a Big Data analytics architecture by looking back to a military analogue – namely the intelligence process (sometimes called the intelligence cycle). Organisation of military intelligence began in earnest in the 1940’s and grew into an established discipline during the 50’s and 60’s when the basis of the modern intelligence process was established.

The modern intelligence process comprises of six elements arranged in a loop, I’ll start with the two that are usually omitted from Big Data analytics architectures:

Data Requirements – defining the data needing to be collected so that the gaps in existing knowledge, relevant to the unfolding situation, can be efficiently filled. There are a few key words in this description:

• Gaps in existing knowledge: knowing what you don’t know is important; Donald Rumsfeld famously said that there are “known unknowns” and “unknown unknowns”. The “known unknowns” should be on our data requirements list – the “unknown unknowns” are a field in themselves and the subject of much study in complex systems;
• Relevant to the unfolding situation: it’s neither practical nor necessary to know everything about the whole world so some degree of selection based on expectations about what is likely to happen next is essential;
• Efficiently filled: as I said in an earlier post, data collection and processing involves cost – don’t do either unless they increase your decision making ability.

Planning and Direction – determining how the data requirements can be met and developing a robust plan for acquisition. The plan needs to address some or all of these aspects:

• Availability: can the required data be obtained from a trustworthy and reliable source; and is the availability consistent with providers’ terms and conditions, legal constraints and ethical considerations?
• Cost: what is the lifetime cost of the data and how often will it be required? How difficult is the data to manipulate or process into a form which is actually useful?
• Timeliness: can the data be acquired and processed quickly enough that it will make a sensible difference to the resulting analytics output?
• Precision: does the data have the required degree of precision, or, can it be processed to extract enough precision to meet the requirements?
• Change control: how quickly will the data source change: are the protocols, formats and access controls stable, what overhead will be involved in maintaining this data source?

Data Collection – this may be one of the most expensive (and sometimes dangerous) aspects of military intelligence but it’s where Big Data is all at – the easy availability of large amounts of data. The main consideration here is to let the data collection be guided by the planning and direction phase.

Data Processing – both this stage and the next aim to convert raw data into actionable intelligence. The difference lies in the view of the data: data at this stage is seen through a microscope and aims to perform the following tasks:

• Extraction: retrieval of the wanted data from the data stream, for example by fields or parsing; 
• Normalization: conversion to the required format and scale, for example language translation, word stemming or metric scaling;
• Provenance: tagging of the data with its source so that decisions can be audited if necessary;
• Evaluation: assessment of the data reliability or precision.

Information Analysis – as the level of abstraction of the data increases then so does the degree to which relationships between data become increasingly important:

• Medium Angle Processing: information extraction based largely on experience with, or statistical analysis of, previous examples of the same data stream; including statistical pattern recognition and machine intelligence;
• Wide Angle Processing: use of multiple data streams, together with wider “world-view” and historical contexts, to provide a knowledge rich assessment of the unfolding situation.

Dissemination and Decision Support – The final, feedback, element in the cycle where decisions can be facilitated, actions taken and new data requirements, planning and direction be initiated.

Putting these elements together into a Big Data analytics architecture has made a lot of sense for us and enabled us to streamline our whole process. What’s more it’s re-usable and process oriented, so it becomes easier each time.

We place emphasis on handling uncertainty so we use a probabilistic approach allowing us to incorporate priors, integrate multiple sources of information and produce decision support and data requirements, planning and direction in a principled manner. We have found this fits this architecture extremely well.