Non-technologists may have noticed that ‘big data’ is the most recent addition to our ever-expanding lexicon of webtwopointwhateverspeak.
Big data refers to datasets that are beyond the means of ordinary software and processing power to analyse, owing to their sheer scale and complexity. An obvious example is Facebook; the London Data Store is another.
Commercial organisations have been collecting vast amounts of data for years; Anyone that has regularly used Gmail, a supermarket loyalty card, or shopped at Amazon, will have at least an inkling of how an organisation can i) collect data and ii) use it to target them with personalised actions.
What is new, is that in many instances the supply of data that companies and government now collect or access vastly overshadows their own ability to actually process it into useful information. It’s not only computer-processing power that is lacking; a recent report by Deloitte points to a massive shortage in skilled labour. These are however short-term barriers that will be overcome by the larger organisations, either by outsourcing data analysis to countries with a surplus of quant talent, or by simply importing that skilled labour directly.
Traditional critics of data collection have made their arguments on the grounds of individual privacy. However the era of ‘big data’ has other, potentially more sinister implications. Writing recently for The Atlantic, Alexander Furnas of the Oxford Internet Institute believes we have yet to fully appreciate the macro-implications of the information age:
“Rather than caring about what they know about me, we should care about what they know about us. Detailed knowledge of individuals and their behavior coupled with the aggregate data on human behavior now available at unprecedented scale grants incredible power. Knowing about all of us – how we behave, how our behavior has changed over time, under what conditions our behavior is subject to change, and what factors are likely to impact our decision-making under various conditions – provides a roadmap for designing persuasive technologies.”
Taken in conjunction with the popularity of behavioural economics within policy-making circles (consider the UK government’s “Nudge Unit” as a case in point) the potential applications of ‘big data’ for public policy are considerable, and deserve closer scrutiny.