Special vs. Common Reactions To Big Data

by Jay Deragon on 02/25/2013

common causeIt is entertaining and discouraging to watch the media’s reaction to events. Entertaining in that they react to common cause as if it were special and special cause as if it were common. Discouraging in that people and organizations react in the same way.

Common cause are events, ocurrences and points of reference that are considered normal variation. Special cause are unusual events that don’t fall into the category of common cause. The reaction to common vs. special cause are different otherwise we increase more of each and that is costly.

Lets say you drive the same road to work every day and on average it takes you 30 minutes to get to work. Then one day it takes you one hour to get to work because of an accident on the road. The common cause is normal traffic flows. The special cause is abnormal traffic flows as a result of an accident. Would you permanently change the route you drive to work because of one accident that ocurred on one day? Not likely.

Yet every day  government and organizations change the rules because they think special cause are common cause and vis versa. In doing so they simply add cost and reduce productivity because of an over reaction to special causes of variation.

Now Big Data Will Create More Reactions

The dialog around Big Data is growing and the reactions vary as expected.

Because of the internet more data has been created in the last ten years then since the beginning of time. The quest to create meaning from the data has increased the demand for minds that understand data.  Consider:

  1. 30 Billion pieces of content shared on Facebook every month
  2. 40% projected growth in global data generated per year vs. 5% growth in global IT spending
  3. 235 terabytes data collected by the US Library of Congress by April 2011
  4. 15 out of 17 sectors in the United States have more data stored per company than the US Library of Congress 
  5. $300 Billion potential annual value to US health care—more than double the total annual health care spending in Spain 
  6. 60% potential increase in retailers’ operating margins possible with big data
  7. 140,000–190,000 more deep analytical talent positions needed in the US today
  8. 1.5 million more data-savvy managers needed in the United States by 2015

There is an old saying “Data lies and liars use data“. Data can be manipulated to mean anything but not everything has meaningful value. To gain meaningful value from data there must be a purpose. The purpose usually is to improve value. Value comes from reducing the friction created by common causes and to understand and reduce the reactions greated by special cause. Otherwise all this BIG DATA will only create BIGGER PROBLEMS.


Jay Deragon February 25, 2013 at 7:27 am


Thanks for the question, it is a good one.

I think the only solution is education of the leaders. Otherwise leaders will continue, as they have in the past, chasing special cause as if it is common cause. When leaders react everyone reacts and the media doesn’t help because they never address the root cause of the problems.

Your thoughts?

Ted Bibbes February 25, 2013 at 6:54 am

So What? Jay, your article starts off great. The topic is relevant and poignant. We often over-react to the big variations in data, and accept the small incremental variations which are more pervasive and detrimental. Yet, your conclusion is nothing more than “heads up” call. What should or could be done about this? Do you have suggested courses of action we can employ today? Can you cite some exemplar companies and their methods for overcoming this problem, and how it helps? I like the article and agree with the premise, but came away wanting more.

Comments on this entry are closed.

Previous post:

Next post: