A Deeper Truth: The New Science of Innovation, Human Choice and Societal Scale Behavior

Conclusion

Introduction - A compulsion to ask

and answer Why?

When I was 13, my advanced math teacher gave the class a formula saying use this to get the right answer. I asked why the formula worked. That math is way beyond this class, just use the formula, he replied.

As crazy as it may sound I couldn’t make myself do it. I had to understand why the formula worked before I would use it. I continued trying to solve the problems, but failed the test. Prior to this I had never received a grade on a math test lower than B.

My teacher asked me why I had done so poorly, and became very annoyed when I explained that I just wasn’t comfortable using the formula unless I understood why it worked. In hindsight he probably thought I was challenging him, but I just needed to know. After two more failed exams, parental involvement, a transfer to the same class taught by another teacher, and that teacher actually taking the time to explain the proof, I went back to getting good grades.

I’m sharing this story to demonstrate an aspect of my nature; I have a compulsion to ask, and understand, "Why?" I am also someone who has repeatedly worked to develop new and innovative products (some successfully, some not), it bothered me immensely that there was no solid answer to the age-old question of why some products succeed, and yet so many others fail. None of the explanations available stood to scrutiny. To make matters worse, new explanations seemed to rise and fall faster than fashion trends and every new product I worked on – success or failure - magnified my need for an answer.

In the same year as my math struggles, either looking for something to impress a girl or compelled by an English assignment, I was flicking through a copy of my mother’s Norton Anthology of English Literature¹ and stumbled upon Percy Shelley’s Ozymandias:

I met a traveller from an antique land

Who said: Two vast and trunkless legs of stone

Stand in the desert. Near them, on the sand,

Half sunk, a shattered visage lies, whose frown,

And wrinkled lip, and sneer of cold command,

Tell that its sculptor well those passions read

Which yet survive, stamped on these lifeless things,

The hand that mocked them and the heart that fed:

And on the pedestal these words appear:

‘My name is Ozymandias, king of kings:

Look on my works, ye Mighty, and despair!’

Nothing beside remains. Round the decay

Of that colossal wreck, boundless and bare

The lone and level sands stretch far away.

A few years later in my first real job my boss asked me, why has every great and powerful civilization – every single one – despite every advantage, eventually collapsed? Ozymandias immediately came to mind and my compulsion to understand why kicked in. Once again, no available answer could stand to scrutiny. Throughout the decades that followed, I continued to seek one out.

Quite unexpectedly my quest for answers to these seemingly unrelated questions came together. I read Driven by Paul Lawrence and Nitin Nohria from Harvard and The Moral Animal by Robert Wright, just after re-reading The Rise and Fall of Great Powers by Paul Kennedy. All had great value to offer, but both Driven and The Rise and Fall had internal inconsistencies or contradictions, were not consistent with my experiences, and had too many elements that were seemingly refuted by other robust research.

The three together, however, provided a whole new direction to explore. Rather than looking for answers in the particulars of specific civilizations, events, or new products, I would look for answers in the one thing they all shared; the medium of human beings.

Rather than two discrete questions, I would ask just one. I would ask a question that would be in some ways simpler and in some more complex. Why? Or more specifically, Why do sane people do the things they do and make the choices they do - including the crazy, irrational, and seemingly inexplicable ones?

Even if you don’t have my compulsion, most of you will share my need for an answer. As a business leader you need to know in order to motivate and direct your staff. As an entrepreneur or product manager you need to know in order to predict the success of your product. As a teacher you need to be able to influence your students. In any role, with any project, you need to know in order to improve your chances of success. As any normal citizen, you need to know simply so you can make better decisions and plan your future? Why Trump? Why terrorism? Why are some things so widely adopted they become disruptive and yet others are rejected seemingly without reason? Why do people do the things they do and make the choices they make?

Science can finally provide an answer, thanks to inputs from dozens of disciplines ranging from economics and neurology to behavioral developmental psychology.² While no single discipline of study provides the logical proof or complete chain of evidence, each contributes a piece of the puzzle. Collectively, they make clear an answer and prove the explanation rests in the universal aspects of human nature.

In short, our brains are comprised of a collection of evolved, purpose specific modules. Amongst other purposes, these modules motivate us to pursue and resist the loss of seemingly abstract outcomes such as relative status, belonging, mastery, and novelty. It is common for these outcomes to be pursued at the expense of rational self-interest or preservation. In addition, these modules, as well as the rest of our brain, are built almost entirely from a single standard neural building block. This universal building block requires nearly all information to be stored in the form of patterns of stimuli and for these patterns to be created by the association of new patterns with ones that already exist. These two realities, the non-monetary forms of value defined by our drive traits combined with the pattern and association bounded nature of all stored information, fundamentally influence our perception and our choices. This is Motivational Drive Trait Theory.

These genetically dictated traits, rather than overriding free will, in many cases prompt it. Despite this, they so strongly influence our perception, choices, and behaviors in context of other people that they dictate readily applicable and formulaic rules for behavior in groups and between groups. These rules in turn enable an explanation and prediction of our choices to adopt new things, a variety of societal scale phenomena, and the interaction of groups with each other. They also offer a pragmatic new approach to leading positive and successful change.

What I propose in this work has what is most lacking in all other explanations: consilience. Meaning it provides a single, consistent explanation across all arenas of human activity and their associated disciplines of study: from economics and the diffusion of technology to neuroscience and from political science to evolutionary developmental psychology.

Despite the strong support consilience represents, and its satisfying nature, I am aware that despite everything I have read and studied, I may have missed something. I have hunted to find every piece of the puzzle and have put those pieces together for you here. I hope this picture is complete, and uplifts your understanding of why we are the way we are, make the decisions we make, and do the things we do. I earnestly hope you find this book worthwhile and thank you in advance for reading onwards.

1 Totally unrelated, another absolutely magnificent poem I stumbled across at the time and remember to this day is Richard Lovelace’s To Althea from Prison.

2 I group books together on my bookshelf that I’ve read in support of this project. I’ve counted these along with the relevant eBooks on my iPad. Together they total 210, but that count only includes the books I own. I’ve also pulled a file count on the research papers I have on my computer associated with the project (1,272). I’m not going to count all of those I have in print. Though I will say my wife would be thrilled if I’d put them all in the recycle bin and free up the entire lower shelf that runs along the length of our front room.

• Chapter 1 •

A Host of Puzzling Questions and What We Think We Know

It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.

–Mark Twain

How is it that suicide becomes a wilfully embraced behavior in so many cultures, religions and political movements? From Vikings volunteering for human sacrifice and kamikazes during World War II to Islamic extremists today, culturally endorsed and wilfully embraced suicide is commonplace throughout history.

Why have intelligent people perpetually sought opportunities to pay absurdly high prices for all manner of things of little or no actual value? From shares in pre-revenue Internet companies in 2000, or tulip bulbs during the mania of 1637, to pet rocks³ in 1975 or Birkin handbags⁴ today (these retail for $12,000 and have sold for as much as $222,912).⁵ Why would people pay substantial sums of money, sometimes as much as the cost of a family home, for something worth so little?

Why do people make obviously irrational choices on a societal scale? What could explain the ridiculous wigs worn by the royal courts of Europe throughout the 17th and 18th century or people wearing gangster jeans literally sagging so below the waist they often fall off?

Photographer: Jo Farrell Photography

Photograph: Matrix Media Ltd.

How is it possible that an entire nation would adopt the crippling practice of foot-binding, a practice popular in China into the twentieth century of breaking their daughter’s feet, folding the broken bones and toes underneath, then binding the mangled mess simply so the feet would appear small?⁶

What causes a whole generation of adolescents to say that’s sick! instead of that’s awesome! to describe something we used to describe as cool!? Why do millions rabidly pursue likes on Facebook and so passionately support specific sporting teams?⁷ And how could anyone have supported or currently support the horrors of slavery, Nazism, the Khmer Rouge, or ISIS?

As a species we seem to perpetually pursue irrational outcomes, and make entirely irrational decisions, on a group and societal scale. The behaviour of both individuals and of whole groups or societies is regularly contrary to what would be predicted by the pursuit of self-preservation, monetary or economic self-interest, or a logical assessment of personal need and health benefits.

Directly on the topic of innovation, why do 90% to 95% of new products, innovation initiatives, and organizational change initiatives fail despite countless experiments, the constant use of new methods, the investment of vast sums of shareholder money or venture capital, and the efforts of many of humanity’s best and brightest? And what could Vikings, foot-binding, and gangster jeans possibly have to do with it? Why are the answers to these seemingly unrelated questions fundamental to solving the puzzle of innovation and new product success?

This new theory from the intersection of economics, neuroscience, genetics, and psychology offers a fascinating and compelling answer backed by hard evidence and research. It delivers a single common explanation not only to the above questions but also for a host of other societal scale phenomena. The story’s most compelling contribution, however, may be what this new interdisciplinary science tells us about how we humans make decisions, how we can make better decisions, and how we can actually predict behavior.

A New Explanation for Disruption is Required

In February 2012 Kodak, which once employed 145,000 people, shut down. In the same year Blockbuster Video, which once employed 60,000 people across 9,000 stores, announced it would close down half of the mere 600 stores that remained. A few months later it announced it would close the rest. After decades, more than 100 years in the case of Kodak, tens of thousands of workers lost their jobs, franchisors hoping to retire had to make new plans, and investors lost any value that remained in what were blue chip stock only a few years earlier. Today, while data shows that most C.E.Os. are optimistic, it also shows that they are living in fear. This fear is no longer about corporate downsizing but about total disruptive obsolescence.

The challenge, despite all of the experts and consultants, the new methods, the investment of billions in innovation labs, agile development programs, and minimum viable products, the vast majority continue to fail. And the participants in these programs just don’t know why. Worse, many have given up on the idea that it is possible to know why. Instead they are adopting fail-fast fail-cheap, not as a method for directed experimentation, but simply as low-cost trial and error.

For several years now, the annual Price Waterhouse Coopers C.E.O. survey has reported half to three quarters of all C.E.Os. consider the rate of technological change and disruption as one of their primary concerns.⁸ Research done by others, including McKinsey and Company and the Economists Intelligence Unit, report that a majority of C.E.Os. feel that the rate of change is accelerating, they expect their business to be disrupted, or that their industry is currently being disrupted. ⁹ Disruptive innovation is the buzz-phrase of the decade and billions of dollars are being invested in consultant fees, hack days, minimum viable products, recruitment, and more.

Despite these efforts, the success or failure rate for new products, innovation initiatives, and start-ups has not improved. If considered in scientific terms, since 2005 alone, hundreds of thousands of experiments in the form of new products and new start-ups have been conducted and less than 10% have yielded the expected result.

Sure, business is a complex arena. But if any scientist conducted a series of experiments and 9 out of every 10 attempts produced radically different results from what their hypothesis predicted, they would conclude that their underlying ideas and hypothesis were wrong. This is what we must do here. To successfully innovate, lead change within our organizations, or succeed in changing the way things are, we must re-examine our underlying beliefs and assumptions about why new products or businesses succeed, why people make the decisions they do, and the causes of widespread or disruptive success.

Clayton Christensen’s Theory of Disruptive Innovation

There are several accepted explanations for highly successful, innovative, and disruptive new products. The leading candidate is Clayton Christensen’s theory of Disruptive Innovation. Put forward in 1995 it achieved near ubiquitous acceptance within a decade.

For those unfamiliar with his theory, Christensen summarises it as follows:

"Disruptive innovation describes a process by which a product or service takes root initially in simple applications at the bottom of a market and then relentlessly moves up market, eventually displacing established competitors.

"As companies tend to innovate faster than their customers’ needs evolve, most organizations eventually end up producing products or services that are actually too sophisticated, too expensive, and too complicated for many customers in their market.

Companies pursue these sustaining innovations" at the higher tiers of their markets because this is what has historically helped them succeed: by charging the highest prices to their most demanding and sophisticated customers at the top of the market, companies will achieve the greatest profitability.

However, by doing so, companies unwittingly open the door to disruptive innovations" at the bottom of the market. An innovation that is disruptive allows a whole new population of consumers at the bottom of a market access to a product or service that was historically only accessible to consumers with a lot of money or a lot of skill.

Characteristics of disruptive businesses, at least in their initial stages, can include: lower gross margins, smaller target markets, and simpler products and services that may not appear as attractive as existing solutions when compared against traditional performance metrics. Because these lower tiers of the market offer lower gross margins, they are unattractive to other firms moving upward in the market, creating space at the bottom of the market for new disruptive competitors to emerge.¹⁰

Today, C.E.Os. continue to relentlessly use the terms innovate and disrupt, investing heavily in projects with that aim. A recent Bloomberg article referred to Disruptive Innovation Theory as possibly the sexiest thing to have emerged from American business schools - ever.¹¹

Yet, as the theory moves toward the level of accepted fact, a number of questions are emerging. Contrary to the years of headlines and the endless predictions of imminent disruption for nearly every industry, most large companies just keep chugging along. The pervasive media hype coupled with the fad-like misuse of the term have produced growing scepticism. More problematically, only a handful of the teams who set out to disrupt markets succeed in doing so, despite the efforts of many of the world’s best and brightest, and the investment of billions of dollars.

Amplifying the situation, academic challenges have been raised by respected professors questioning the veracity of both the theory and its supporting case studies.¹² According to a review in the MIT Sloan Management Review, less than 10% of the examples cited by Clayton Christensen in his own work actually conform to his own model. ¹³ Clayton Christensen has even stated that the theory breaks down under a variety of conditions. He regularly reminds people that it was specifically defined as applicable to only a select set of circumstances. And, both he and his colleagues have openly and repeatedly expressed concerns that most projects intended to disrupt will fail because the theory is so widely misunderstood and misapplied.

While Disruptive Innovation Theory clearly explains events some of the time (and his book The Innovator’s Dilemma is a must-read for any serious innovator, product manager, or senior executive), the research clearly shows it does not explain any more than a relatively small percentage of wildly successful new products. Further, while it clearly offers insight into some decisions made by management teams, it does not explain the widespread resistance to new products so frequently observed or why rational executives so consistently make irrational choices such as Kodak’s continued decisions to ignore digital photography long after it was clearly the future.

Given the evidence and the fact that even its most ardent critics acknowledge that Disruptive Innovation Theory is correct in specific situations, it must be a component of any answer. But there must also be more to the story.

Gladwell’s Tipping Point

Another popular explanation for wildly successful products is the idea of a viral Tipping Point. Malcolm Gladwell’s book of that title sold millions of copies. Whether the result of different types of people who promote and convince others to adopt new things, or that of a psychological contagion, Gladwell points out that successful products spread like viruses. The success of the book, coupled with the success of products like Hotmail and the emergence of seemingly successful viral marketing campaigns, has generated widespread acceptance of the desirability of viral characteristics in product design and marketing.

Once again, however, making something viral has proven exceedingly difficult and virtually impossible to predict. For every success attributed to viral-by-design characteristics such as those used by DropBox’s referral program, the Blendtec YouTube Can it blend video campaign, or LinkedIn - there have been a multitude of failures such as the Cheeto’s