Size matters

Non-technologists may have noticed that ‘big data’ is the most recent addition to our ever-expanding lexicon of webtwopointwhateverspeak.

Big data refers to datasets that are beyond the means of ordinary software and processing power to analyse, owing to their sheer scale and complexity.  An obvious example is Facebook; the London Data Store is another.

Commercial organisations have been collecting vast amounts of data for years; Anyone that has regularly used Gmail, a supermarket loyalty card, or shopped at Amazon, will have at least an inkling of how an organisation can i) collect data and ii) use it to target them with personalised actions.

What is new, is that in many instances the supply of data that companies and government now collect or access vastly overshadows their own ability to actually process it into useful information. It’s not only computer-processing power that is lacking; a recent report by Deloitte points to a massive shortage in skilled labour. These are however short-term barriers that will be overcome by the larger organisations, either by outsourcing data analysis to countries with a surplus of quant talent, or by simply importing that skilled labour directly.

Traditional critics of data collection have made their arguments on the grounds of individual privacy. However the era of ‘big data’ has other, potentially more sinister implications. Writing recently for The Atlantic, Alexander Furnas of the Oxford Internet Institute believes we have yet to fully appreciate the macro-implications of the information age:

“Rather than caring about what they know about me, we should care about what they know about us. Detailed knowledge of individuals and their behavior coupled with the aggregate data on human behavior now available at unprecedented scale grants incredible power. Knowing about all of us – how we behave, how our behavior has changed over time, under what conditions our behavior is subject to change, and what factors are likely to impact our decision-making under various conditions – provides a roadmap for designing persuasive technologies.”

Taken in conjunction with the popularity of behavioural economics within policy-making circles (consider the UK government’s “Nudge Unit” as a case in point) the potential applications of ‘big data’ for public policy are considerable, and deserve closer scrutiny.

One thought on “Size matters

  1. My spouse and i got very pefaceul Raymond managed to complete his web research while using the precious recommendations he had when using the weblog. It is now and again perplexing just to choose to be giving away concepts which often people may have been making money from. So we consider we have you to thank because of that. The type of illustrations you have made, the straightforward blog navigation, the friendships your site aid to instill it is most excellent, and it’s really letting our son and us feel that this concept is satisfying, which is certainly especially serious. Thanks for everything!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s