Business Analytics: Why Not Experiment?

I recently came across a couple good articles on business experimentation from my meanderings of more than 30 LinkedIn group discussions. Randomized experiments or related quasi-experimental designs for measurement are now part and parcel of evidence-based business. Indeed, whether you refer to the focus as data-driven, super-crunching or big data analytics, central to them all are large data sets, statistics/machine learning algorithms and sophisticated designs.

Writing in the European Business Review, Harvard Business School professor Stefan Thomke derides what he sees as the current obsession with cost-side optimization and standardization — “Six Sigmaing” — which he feels may ultimately come back to haunt practicing companies:

“Such organizations leave themselves vulnerable to underinvesting in experimentation and variation, which are the lifeblood of innovation … And it is only through such experimentation, which might include structured cause-and-effect tests, informal trial-and-error experiments, and rigorous randomized field trials, that companies can unlock their true capacity for innovation.”

The poster children for the innovation problem, according to Thomke, are 3M and now-departed CEO W. James McNerney Jr., who built a Six Sigma reputation at General Electric. Under McNerney’s leadership and process improvement prioritization, 3M started losing its innovation edge, with new product revenue dropping from 33 to 25 percent of sales.

If Six Sigma is preoccupied with the cost side of P&L, then innovation and experimentation are more concerned with revenue. Thomke argues that in a downturn such as the world economy has recently experienced, it’s important to step on the innovation accelerator — “those companies that maintain their experimentation when business is slow will be all the more prepared when the market eventually picks up.”

For Thomke, innovation requires experimentation, since in the complex business world, passive observation and exploration are generally not enough to discern complex environments and linkages among variables. Properly designed and implemented experiments, on the other hand, can shed light on important causal relationships. For example, “randomized field trials can help companies determine whether specific changes, such as a new layout for a chain of retail stores, will lead to improved performance (a significant bump in sales).” And medical researchers can determine the efficacy of a new treatment by executing a trial that randomly assigns patients to treatment/control groups and comparing outcome measures.

Even properly executed experiments — those with high internal validity — can still suffer from external validity shortcomings if the sample isn’t representative of the larger population or the findings don’t generalize beyond the specific experimental settings.

The good news for business innovators is that experimentation has never been cheaper. Many B2C companies design, implement and analyze thousands of experiments weekly, fueled by Internet access to customers and inexpensive technologies. Amazon and Google users participate in trials almost every time they log in.

Thomke's doctrine appears not to have been lost on 3M replacement CEO George Buckley, who countermanded many of his predecessor’s dictums, increasing the R&D budget and unshackling the research scientists, tellingly stating: “Invention is by its very nature a disorderly process ... You can’t put a Six Sigma process into that area and say, well, I’m getting behind on invention, so I’m going to schedule myself for three good ideas on Wednesday and two on Friday. That’s not how creativity works.”

Behavioral economist Dan Ariely is frustrated with business reluctance to experiment as well: “I’ve often tried to help companies do experiments, and usually I fail spectacularly,” he notes, citing personal unfulfilled consultations in marketing and HR.

Ariely offers two psychological explanations for organizational apprehension over experimentation. First, adoption of the experimental method mandates that business trade off short-term losses for the potential of longer-term gains. And that’s only the potential for gain. The experimental mentality means accepting that you’ll often fail – and of course retain management support in the process. In the what-have-you-done-for-me-this-quarter mentality, that accommodation may not fly.  

Second is the sense of security – or maybe just cognitive dissonance – that derives from having highly-paid experts make strategic choices. Historically, companies have been much more comfortable adopting the recommendations of experts over those of cold analytics. I suspect experts have been more comfortable with that situation as well!

But that’s changing. Some of the business consultants I’ve worked with in the past and watched pooh-pooh analytics are now on the bandwagon. The major strategy and business consultancies are aggressively establishing big data and analytics practices. And the notion that analytics trumps experts for many predictive decisions is almost universally accepted now, even if grudgingly by some. This is assuredly good news for business.

This story originally appeared at Information Management.

Readers are encouraged to respond to Steve using the “Add Your Comments” box below. 

This blog may not be reposted or reused without permission from Insurance Networking News.

The opinions of bloggers on www.insurancenetworking.com do not necessarily reflect those of Insurance Networking News.

For reprint and licensing requests for this article, click here.
Analytics Data and information management Policy adminstration
MORE FROM DIGITAL INSURANCE