An age of reason

We expect companies that were born digital, like Amazon, to accomplish things business executives could only dream of a generation ago. But, in fact, the use of big data has the potential to transform traditional businesses as well. It may offer them even greater opportunities for competitive advantage (online businesses have always known they were competing on how well they understood their data).

As the tools and philosophies of big data spread, they will change long-standing ideas about the value of experience, the nature of expertise and the practice of management. Smart leaders across industries will see using big data for what it is: a management revolution. But as with any other major change in business, the challenges of becoming a big data enabled organisation can be enormous and require hands-on – or in some cases hands-off – leadership. Nevertheless, it’s a transition that executives need to engage with today.

Data-driven performance
The business press is rife with anecdotes and case studies that supposedly demonstrate the value of being data-driven. But the truth, we realised recently, is that nobody was tackling that question rigorously. To address this embarrassing gap, we led a team at the Massachusetts Institute of Technology Center for Digital Business, working in partnership with McKinsey’s business technology office and with our colleague Lorin Hitt at Wharton and the MIT doctoral student Heekyung Kim. We set out to test the hypothesis that data-driven companies would be better performers. We conducted structured interviews with executives at 330 public US companies about their organisational and technology management practices, and gathered performance data from their annual reports and independent sources.

One relationship stood out: the more companies characterised themselves as data-driven, the better they performed on objective measures of financial and operational results. In particular, companies in the top third of their industry in the use of data-driven decision-making were, on average, five percent more productive and six percent more profitable than their competitors. This performance difference remained robust after accounting for the contributions of labour, capital, purchased services and traditional information technology investment. It was statistically significant and economically important and was reflected in measurable increases in stock market valuations.

New culture
The technical challenges of using big data are very real, but the managerial challenges are even greater: starting with the role of the senior executive team.

Muting the Hippo: One of the most critical aspects of big data is its impact on how decisions are made and who gets to make them. When data is scarce, expensive to obtain or not available in digital form, it makes sense to let well-placed people make decisions – which they do on the basis of experience.

For particularly important decisions, these people are typically high up in the organisation, or they’re expensive outsiders brought in because of their expertise and track records. Many in the big data community maintain companies often make most of their important decisions by relying on the Highest-Paid Person’s Opinion (Hippo).
We believe that throughout the business world, people rely too much on experience and intuition and not enough on data. For our research, we constructed a five-point composite scale that captured the overall extent to which a company was data-driven. Fully 32 percent of our respondents rated their companies at or below three on this scale.
New roles: Executives interested in leading a big data transition can start with two simple techniques. First, they can get into the habit of asking: “What does the data say?” when faced with an important decision and following up with more-specific questions such as “Where did the data come from?” and “What kinds of analyses were conducted?” Second, they can allow themselves to be overruled by the data; few things are more powerful for changing a decision-making culture than seeing a senior executive concede when data has disproved a hunch.

Five management challenges
Companies won’t reap the full benefits of a transition to using big data unless they’re able to manage change effectively. We have identified five areas which are of particular importance in that process:

Leadership: Companies succeed in the big data era not simply because they have more or better data, but because they have leadership teams that set clear goals, define what success looks like and ask the right questions. Big data’s power doesn’t erase the need for vision or human insight. We must still have business leaders who can spot a great opportunity, articulate a compelling vision, persuade people to embrace it and work hard to realise it.

Talent management: As data becomes cheaper, the complements to data become more valuable. Some of the most crucial of these are data scientists and other professionals skilled at working with large quantities of information. Statistics are important, but many of the key techniques for using big data are rarely taught in traditional statistics courses. Along with the data scientists, a new generation of computer scientists are bringing to bear techniques for working with very large data sets.

Technology: The tools available to handle the volume, velocity and variety of big data have improved greatly in recent years. In general, these technologies are not prohibitively expensive. However, they do require a skill set that is new to most IT departments. They will need to work hard to integrate all the relevant internal and external sources of data.

Decision-making: An effective organisation puts information and the relevant decision rights in the same location. In the big data era, information is created and transferred, and expertise is often not where it used to be. The artful leader will create an organisation flexible enough to minimise the “not invented here” syndrome while at the same time maximising cross-functional cooperation.

Company culture: The first question a data-driven organisation asks itself is not “What do we think?” but “What do we know?” This requires a move away from acting solely on instinct. It also requires breaking a bad habit we’ve noticed in many organisations: pretending to be more data-driven than they actually are. Too often, we saw executives who spiced up their reports with lots of data that supported decisions they had already made using the traditional Hippo approach.

The evidence from our research is clear: data-driven decisions tend to be better decisions. Business leaders will either embrace this fact or be replaced by others who do. In sector after sector, companies that figure out how to combine domain expertise with data science will pull away from their rivals.