The Power Of Big Data

One of my close friends recently joined Microsoft in Seattle in their highly acclaimed data analysis team. I asked him what was his first assignment. He said his boss gave him 1 Petabyte of data related to Microsoft products and its sales and asked him to find insights. That’s it. That was the task. His boss did not say what to find or look for in the data. It’s my friend’s job to find information from the huge volume of data. So I asked him when will he know whether he found something valuable. His answer was unique. He said “I will continue to analyze the data until I find valuable insights which surprises me and my boss” .

While I was listening to my friend talking about the exciting things he gets to do on a day to day basis, my mind started wandering about the surprises I encountered when analysing data with big data systems and the fascinating stories I heard about data analysis from other data scientists. Of all the fascinating data analysis stories I heard, the one I read from the book “The Power of Habit by Charles Duhigg” really stands out. The Power of Habit is not a Big Data or a technology book but rather a fascinating work on how habits form and how we can create a new habit or change an existing one. It’s a great, quick and interesting read and I encourage you to read it.

In one of the chapters, the author talks about how companies use data they collect to predict and manipulate customer’s habits. The title of the chapter is “How Target Knows What You Want Before You Do”. Interesting title, isn’t? The story is even more interesting.

In the chapter, the author explains how Target collects and keep track of customer data. Target tries to associate every sale with a Guest ID (customer number). But how does Target know it is you who bought a six pack beer last week? Very simple. If you used Target issued credit/debit card, any coupons sent to your email or home address, frequent-buyer tag etc., Target can put a name to the sale.

What does Target do with the data?

In the book, the author talks about, Andrew Pole, a data expert at Target who was tasked to find out which of Target’s customers were pregnant based on their buying patterns. Why find pregnant women you ask? Here is the answer from the book –

“Pregnant women and new parents, after all, the holy grail of retail. there is almost no more profitable, product-hungry, price-insensitive group in existence. It’s not just diapers and wipes. People with infants are so tired that they’ll by everything they need – juice and toilet paper, socks and magazines – wherever they purchase their bottles and formula. What’s more, if a new parent starts shopping at Target, they’ll keep coming back for years.

Figuring out who was pregnant, in other words, could make Target millions of dollars”

By looking at the customer’s shopping patterns and shopping history, Andrew created an algorithm/program that can look at mountains of data and can assign a “pregnancy prediction” score to any woman in Target’s database. More importantly, Pole’s algorithm can predict (or guess) the trimester and the due date – so Target could send targeted or pregnancy related coupons and promotions early in the pregnancy.

Data can tell things about you which you don’t even know yourself

Here is another excerpt from chapter to explain how others with your data can find more information about you from the data than you know about yourself.

“About a year after Pole created his pregnancy prediction model, a man walked into a Minnesota Target and demanded to see the manager. He was clutching an advertisement. He was very angry.

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

The manager didn’t have any idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture, and pictures of smiling infants gazing into their mothers’ eyes.

The manager apologized profusely, and then called, a few days later, to apologize again.

The father was somewhat abashed.

“I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of.” He took a deep breath. “She’s due in August. I owe you an apology”. ”

The chapter goes on explaining customer’s (negative) reaction when they got pregnancy targeted coupons and how Target and companies alike found a solution to the negative reactions.

Wow. That’s some predictive analysis, isn’t? It certainly demonstrates the power of data and what you can do with it. It’s an excellent use case revealing reasons why companies like Target are interested in investing in Big Data solutions.

Do you have any interesting stories of how data is or can be used in finding valuable and surprising insights? If you do, please comment below.