The Sexy Dark Art of Turning Data into Better Decisions

We all want to make better decisions, right? In the past, most decisions had to be made on gut-feel. Today, we have data to support our decision-making. We can look at real facts and figures on almost anything today before we leap to conclusions and there is definitely no shortage of data. Yet, few managers have mastered the art of using data to support the their most crucial decisions.

This is not helped by the fact that we have now entered the realm of ‘big data’, which is so vast that traditional information management systems can’t cope with it any more. We are generating data at a frightening rate. In the past we had a little snowball of data but this little snowball is now rolling down the mountain gathering pace and size by the second. Today an ever-growing avalanche of data is opening up never-seen opportunities for business, science and society as a hole.

Under this backdrop it is not surprising that data science – the ability to turn this data into useful decisions - was hailed in a Harvard Business Review article as the sexiest jobs of the 21 century. Data scientist are able to deal with the avalanche of data. A bit like avalanche rescue teams that use sniffer dogs, probes and sensors to find survivors that are hidden under the massive piles of snow, data scientists are able to uncover the hidden gems of insights from your data. The problem is that is it not only seen as sexy but as a ‘dark art’ that only few really understand.

I thought it might be a good idea to shine a bit more light onto this dark sexy art of finding value in big data. A first step is to understand and classify the ways companies use big data to derive value. Below is a list of the top 5 big data use cases based on some work by IBM. Let’s look at what IBM believes are the top five ‘high value’ use cases for big data:

1. Big Data Exploration

Find, visualize, understand all big data to improve decision-making. Big data exploration addresses the challenge that every large organization faces: information is stored in many different systems and silos and people need access to that data to do their day-to-day work and make important decisions.

I would like to add: This is a ‘one-size-fits-all’ category that could include anything. The key point is that companies can delve into existing data repositories and transactions using big data techniques. This would also enable them to bring together data from different systems such as financial transactions, operational quality data, HR data, supplier information, etc. that is stored in different places or organizational silos. It enables companies to create a more complete picture and gain new insights from looking at all the available data. One example is corporate email and newsletter provider Constant Contact – they are sending out over 35 billion emails for their clients per year and analyzing the performance of these emails e.g. when to send them, how often, what subject lines work best, etc. gives the company great insights which it can us to optimize performance and to provide feedback to their clients.

2. Enhanced 360º View of the Customer

Extend existing customer views by incorporating additional internal and external information sources. Gain a full understanding of customers—what makes them tick, why they buy, how they prefer to shop, why they switch, what they’ll buy next, and what factors lead them to recommend a company to others.

I would like to add: Here companies use big data analytics to understand and better engage with customers. Examples would include telecom companies that use the data from phone records as well as social media behavior to create better pictures of customers. Some have started to predict churn and loyalty behaviors simply by classifying customers based on their call and social media patterns.

3. Security/Intelligence Extension

Lower risk, detect fraud and monitor cyber security in real time. Augment and enhance cyber security and intelligence analysis platforms with big data technologies to process and analyze new types (e.g. social media, emails, sensors, Telco) and sources of under-leveraged data to significantly improve intelligence, security and law enforcement insight.

I would like to add: Big data analytics allow us to detect fraud by analyzing credit card transactions in real time with the ability to shut down transactions that are suspicious or not feasible e.g. purchasing something in New York City at 2pm and in New Deli at 3pm. Big data analytics are also used to detect terrorist activity and cyber security attracts by constantly monitoring and processing data including phone conversations, social media messages, emails as well as sensor and machine data.

4. Operations Analysis

Analyze a variety of machine and operational data for improved business results. The abundance and growth of machine data, which can include anything from IT machines to sensors and meters and GPS devices requires complex analysis and correlation across different types of data sets. By using big data for operations analysis, organizations can gain real-time visibility into operations, customer experience, transactions and behavior.

I would like to add: The ‘Internet of Things’ is generating new data by the second. Smart Watches, Smart TVs, Smart Phones and even Smart Diapers are contributing to this new data stream. Companies can use this data to improve their own performance and even sell the data and insights to others. One example comes from a pizza delivery company which tracks their drivers using the GPS sensors in their smart phones. This gives the company new insights into how to optimize delivery routes. Another example comes from Energy Smart Meters. Collecting and using this new type of data gives us, for the first time, a real-time understanding the energy usages. This level of understanding allows energy companies to improve grid reliability and performance.

5. Data Warehouse Augmentation

Integrate big data and data warehouse capabilities to increase operational efficiency. Optimize your data warehouse to enable new types of analysis. Use big data technologies to set up a staging area or landing zone for your new data before determining what data should be moved to the data warehouse. Offload infrequently accessed or aged data from warehouse and application databases using information integration software and tools.

I would like to add: I am not really sure that this deserves it’s own ‘high value use case’ category. I would see it more as the technical end of doing big data analytics well. Anyway, you can obviously use big data techniques to optimize your data warehouse. Especially if you want to extend your data warehouse with social media data or break up or off-load part of the processing or analysis to improve overall data warehouse performance. The one use-case I would have added instead is health analytics where big data is used to predict diseases and find new ways to cure illness.

I hope this gives you a better feel for this “sexy dark art” of turning data (especially big data) into better decision-making. More on all of this soon.