Where Good Ideas Come From: The Natural History of Innovation

"[A] rich, integrated and often sparkling book. Mr. Johnson, who knows a thing or two about the history of science, is a first-rate storyteller."--"The New York Times"

"A vision of innovation and ideas that is resolutely social, dynamic and material...Fluidly written, entertaining and smart without being arcane."--"Los Angeles Times"

"A magical mystery tour of the history and architecture of innovation."--"The Oregonian"

"A rapid-fire tour of 'spaces' large, small, mental, physical, and otherwise... Where Good Ideas Come From may be the ultimate distillation of his thinking on these issues... One admires the intellectual athleticism of Johnson's maneuvers here."--"Boston Globe"

Steven Johnson is the bestselling author of Future Perfect, Where Good Ideas Come From, The Invention of Air, The Ghost Map, and Everything Bad is Good for You, and is the editor of The Innovator's Cookbook. He is the founder of a variety of influential websites and writes for Time, Wired, The New York Times, and The Wall Street Journal. He lives in Marin County, California, with his wife and three sons.

Introduction

REEF, CITY, WEB
 

—SHAKESPEARE, A Midsummer Night’s Dream, V.i.14-17

Darwin’s Paradox

April 4, 1836. Over the eastern expanse of the Indian Ocean, the reliable northeast winds of monsoon season have begun to give way to the serene days of summer. On the Keeling Islands, two small atolls composed of twenty-seven coral islands six hundred miles west of Sumatra, the emerald waters are invitingly placid and warm, their hue enhanced by the brilliant white sand of disintegrated coral. On one stretch of shore usually guarded by stronger surf, the water is so calm that Charles Darwin wades out, under the vast blue sky of the tropics, to the edge of the live coral reef that rings the island.

For hours he stands and paddles among the crowded pageantry of the reef. Twenty-seven years old, seven thousand miles from London, Darwin is on the precipice, standing on an underwater peak ascending over an unfathomable sea. He is on the edge of an idea about the forces that built that peak, an idea that will prove to be the first great scientific insight of his career. And he has just begun exploring another hunch, still hazy and unformed, that will eventually lead to the intellectual summit of the nineteenth century.

Around him, the crowds of the coral ecosystem dart and shimmer. The sheer variety dazzles: butterflyfish, damselfish, parrotfish, Napoleon fish, angelfish; golden anthias feeding on plankton above the cauliflower blooms of the coral; the spikes and tentacles of sea urchins and anemones. The tableau delights Darwin’s eye, but already his mind is reaching behind the surface display to a more profound mystery. In his account of the Beagle’s voyage, published four years later, Darwin would write: “It is excusable to grow enthusiastic over the infinite numbers of organic beings with which the sea of the tropics, so prodigal of life, teems; yet I must confess I think those naturalists who have described, in well-known words, the submarine grottoes decked with a thousand beauties, have indulged in rather exuberant language.”

What lingers in the back of Darwin’s mind, in the days and weeks to come, is not the beauty of the submarine grotto but rather the “infinite numbers” of organic beings. On land, the flora and fauna of the Keeling Islands are paltry at best. Among the plants, there is little but “cocoa-nut” trees, lichen, and weeds. “The list of land animals,” he writes, “is even poorer than that of the plants”: a handful of lizards, almost no true land birds, and those recent immigrants from European ships, rats. “The island has no domestic quadruped excepting the pig,” Darwin notes with disdain.

Yet just a few feet away from this desolate habitat, in the coral reef waters, an epic diversity, rivaled only by that of the rain forests, thrives. This is a true mystery. Why should the waters at the edge of an atoll support so many different livelihoods? Extract ten thousand cubic feet of water from just about anywhere in the Indian Ocean and do a full inventory on the life you find there: the list would be about as “poor” as Darwin’s account of the land animals of the Keelings. You might find a dozen fish if you were lucky. On the reef, you would be guaranteed a thousand. In Darwin’s own words, stumbling across the ecosystem of a coral reef in the middle of an ocean was like encountering a swarming oasis in the middle of a desert. We now call this phenomenon Darwin’s Paradox: so many different life forms, occupying such a vast array of ecological niches, inhabiting waters that are otherwise remarkably nutrient-poor. Coral reefs make up about one-tenth of one percent of the earth’s surface, and yet roughly a quarter of the known species of marine life make their homes there. Darwin doesn’t have those statistics available to him, standing in the lagoon in 1836, but he has seen enough of the world over the preceding four years on the Beagle to know there is something peculiar in the crowded waters of the reef.

The next day, Darwin ventures to the windward side of the atoll with the Beagle’s captain, Vice Admiral James FitzRoy, and there they watch massive waves crash against the coral’s white barrier. An ordinary European spectator, accustomed to the calmer waters of the English Channel or the Mediterranean, would be naturally drawn to the impressive crest of the surf. (The breakers, Darwin observes, are almost “equal in force [to] those during a gale of wind in the temperate regions, and never cease to rage.”) But Darwin has his eye on something else—not the violent surge of water but the force that resists it: the tiny organisms that have built the reef itself.

The ocean throwing its waters over the broad reef appears an invincible, all-powerful enemy; yet we see it resisted, and even conquered, by means which at first seem most weak and inefficient. It is not that the ocean spares the rock of coral; the great fragments scattered over the reef, and heaped on the beach, whence the tall cocoa-nut springs, plainly bespeak the unrelenting power of the waves . . . Yet these low, insignificant coral-islets stand and are victorious: for here another power, as an antagonist, takes part in the contest. The organic forces separate the atoms of carbonate of lime, one by one, from the foaming breakers, and unite them into a symmetrical structure. Let the hurricane tear up its thousand huge fragments; yet what will that tell against the accumulated labour of myriads of architects at work night and day, month after month?

Darwin is drawn to those minuscule architects because he believes they are the key to solving the mystery that has brought the Beagle to the Keeling Islands. In the Admiralty’s memorandum authorizing the ship’s five-year journey, one of the principal scientific directives is the investigation of atoll formation. Darwin’s mentor, the brilliant geologist Charles Lyell, had recently proposed that atolls are created by undersea volcanoes that have been driven upward by powerful movements in the earth’s crust. In Lyell’s theory, the distinctive circular shape of an atoll emerges as coral colonies construct reefs along the circumference of the volcanic crater. Darwin’s mind had been profoundly shaped by Lyell’s understanding of the deep time of geological transformation, but standing on the beach, watching the breakers crash against the coral, he knows that his mentor is wrong about the origin of the atolls. It is not a story of simple geology, he realizes. It is a story about the innovative persistence of life. And as he mulls the thought, there is a hint of something else in his mind, a larger, more encompassing theory that might account for the vast scope of life’s innovations. The forms of things unknown are turning, slowly, into shapes.

Days later, back on the Beagle, Darwin pulls out his journal and reflects on that mesmerizing clash between surf and coral. Presaging a line he would publish thirty years later in the most famous passage from On the Origin of Species, Darwin writes, “I can hardly explain the reason, but there is to my mind much grandeur in the view of the outer shores of these lagoon-islands.” In time, the reason would come to him.

The Superlinear City

From an early age, the Swiss scientist Max Kleiber had a knack for testing the edges of convention. As an undergraduate in Zurich in the 1910s, he roamed the streets dressed in sandals and an open collar, shocking attire for the day. During his tenure in the Swiss army, he discovered that his superiors had been trading information with the Germans, despite the official Swiss position of neutrality in World War I. Appalled, he simply failed to appear at his next call-up, and was ultimately jailed for several months. By the time he had settled on a career in agricultural science, he had had enough of the restrictions of Zurich society. And so Max Kleiber charted a path that would be followed by countless sandal-wearing, nonconformist war protesters in the decades to come. He moved to California.

Kleiber set up shop at the agricultural college run by the University of California at Davis, in the heart of the fertile Central Valley. His research initially focused on cattle, measuring the impact body size had on their metabolic rates, the speed with which an organism burns through energy. Estimating metabolic rates had great practical value for the cattle industry, because it enabled farmers to predict with reasonable accuracy both how much food their livestock would require, and how much meat they would ultimately produce after slaughter. Shortly after his arrival at Davis, Kleiber stumbled across a mysterious pattern in his research, a mathematical oddity that soon brought a much more diverse array of creatures to be measured in his lab: rats, ring doves, pigeons, dogs, even humans.

Scientists and animal lovers had long observed that as life gets bigger, it slows down. Flies live for hours or days; elephants live for half-centuries. The hearts of birds and small mammals pump blood much faster than those of giraffes and blue whales. But the relationship between size and speed didn’t seem to be a linear one. A horse might be five hundred times heavier than a rabbit, yet its pulse certainly wasn’t five hundred times slower than the rabbit’s. After a formidable series of measurements in his Davis lab, Kleiber discovered that this scaling phenomenon stuck to an unvarying mathematical script called “negative quarter-power scaling.” If you plotted mass versus metabolism on a logarithmic grid, the result was a perfectly straight line that led from rats and pigeons all the way up to bulls and hippopotami.

Physicists were used to discovering beautiful equations like this lurking in the phenomena they studied, but mathematical elegance was a rarity in the comparatively messy world of biology. But the more species Kleiber and his peers analyzed, the clearer the equation became: metabolism scales to mass to the negative quarter power. The math is simple enough: you take the square root of 1,000, which is (approximately) 31, and then take the square root of 31, which is (again, approximately) 5.5. This means that a cow, which is roughly a thousand times heavier than a woodchuck, will, on average, live 5.5 times longer, and have a heart rate that is 5.5 times slower than the woodchuck’s. As the science writer George Johnson once observed, one lovely consequence of Kleiber’s law is that the number of heartbeats per lifetime tends to be stable from species to species. Bigger animals just take longer to use up their quota.

Over the ensuing decades, Kleiber’s law was extended down to the microscopic scale of bacteria and cell metabolism; even plants were found to obey negative quarter-power scaling in their patterns of growth. Wherever life appeared, whenever an organism had to figure out a way to consume and distribute energy through a body, negative quarter-power scaling governed the patterns of its development.

Several years ago, the theoretical physicist Geoffrey West decided to investigate whether Kleiber’s law applied to one of life’s largest creations: the superorganisms of human-built cities. Did the “metabolism” of urban life slow down as cities grew in size? Was there an underlying pattern to the growth and pace of life of metropolitan systems? Working out of the legendary Santa Fe Institute, where he served as president until 2009, West assembled an international team of researchers and advisers to collect data on dozens of cities around the world, measuring everything from crime to household electrical consumption, from new patents to gasoline sales.

When they finally crunched the numbers, West and his team were delighted to discover that Kleiber’s negative quarter-power scaling governed the energy and transportation growth of city living. The number of gasoline stations, gasoline sales, road surface area, the length of electrical cables: all these factors follow the exact same power law that governs the speed with which energy is expended in biological organisms. If an elephant was just a scaled-up mouse, then, from an energy perspective, a city was just a scaled-up elephant.

But the most fascinating discovery in West’s research came from the data that didn’t turn out to obey Kleiber’s law. West and his team discovered another power law lurking in their immense database of urban statistics. Every datapoint that involved creativity and innovation—patents, R&D budgets, “supercreative” professions, inventors—also followed a quarter-power law, in a way that was every bit as predictable as Kleiber’s law. But there was one fundamental difference: the quarter-power law governing innovation was positive, not negative. A city that was ten times larger than its neighbor wasn’t ten times more innovative; it was seventeen times more innovative. A metropolis fifty times bigger than a town was 130 times more innovative.

Kleiber’s law proved that as life gets bigger, it slows down. But West’s model demonstrated one crucial way in which human-built cities broke from the patterns of biological life: as cities get bigger, they generate ideas at a faster clip. This is what we call “superlinear scaling”: if creativity scaled with size in a straight, linear fashion, you would of course find more patents and inventions in a larger city, but the number of patents and inventions per capita would be stable. West’s power laws suggested something far more provocative: that despite all the noise and crowding and distraction, the average resident of a metropolis with a population of five million people was almost three times more creative than the average resident of a town of a hundred thousand. “Great cities are not like towns only larger,” Jane Jacobs wrote nearly fifty years ago. West’s positive quarter-power law gave that insight a mathematical foundation. Something about the environment of a big city was making its residents significantly more innovative than residents of smaller towns. But what was it?

The 10/10 Rule

The first national broadcast of a color television program took place on January 1, 1954, when NBC aired an hour-long telecast of the Tournament of Roses parade, and distributed it to twenty-two cities across the country. For those lucky enough to see the program, the effect of a moving color image on a small screen seems to have been mesmerizing. The New York Times, in typical language, called it a “veritable bevy of hues and depth.” “To concentrate so much color information within the frame of a small screen,” the Times wrote, “would be difficult for even the most gifted artist doing a ‘still’ painting. To do it with constantly moving pictures seemed pure wizardry.” Alas, the Rose Parade “broadcast” turned out to be not all that broad, given that it was visible only on prototype televisions in RCA showrooms. Color programming would not become standard on prime-time shows until the late 1960s. After the advent of color, the basic conventions that defined the television image would go unchanged for decades. The delivery mechanisms began to diversify with the introduction of VCRs and cable in the late 1970s. But the image remained the same.

In the mid-1980s, a number of influential media and technology executives, along with a few visionary politicians, had the eminently good idea that it was time to upgrade the video quality of broadcast television. Speeches were delivered, committees formed, experimental prototypes built, but it wasn’t until July 23, 1996, that a Raleigh, North Carolina, CBS affiliate initiated the first public transmission of an HDTV signal. Like the Tournament of Roses footage, though, there were no ordinary consumers with sets capable of displaying its “wizardry.”1 A handful of broadcasters began transmitting HDTV signals in 1999, but HD television didn’t become a mainstream consumer phenomenon for another five years. Even after the FCC mandated that all television stations cease broadcasting the old analog standard on June 12, 2009, more than 10 percent of U.S. households had televisions that went dark that day.

It is one of the great truisms of our time that we live in an age of technological acceleration; the new paradigms keep rolling in, and the intervals between them keep shortening. This acceleration reflects not only the flood of new products, but also our growing willingness to embrace these strange new devices, and put them to use. The waves roll in at ever-increasing frequencies, and more and more of us are becoming trained surfers, paddling out to meet them the second they start to crest. But the HDTV story suggests that this acceleration is hardly a universal law. If you measure how quickly a new technology progresses from an original idea to mass adoption, then it turns out that HDTV was traveling at the exact same speed that color television had traveled four decades earlier. It took ten years for color TV to go from the fringes to the mainstream; two generations later, it took HDTV just as long to achieve mass success.

In fact, if you look at the entirety of the twentieth century, the most important developments in mass, one-to-many communications clock in at the same social innovation rate with an eerie regularity. Call it the 10/10 rule: a decade to build the new platform, and a decade for it to find a mass audience. The technology standard of amplitude-modulated radio—what we now call AM radio—evolved in the first decade of the twentieth century. The first commercial AM station began broadcasting in 1920, but it wasn’t until the late 1920s that radios became a fixture in American households. Sony inaugurated research into the first consumer videocassette recorder in 1969, but didn’t ship its first Betamax for another seven years, and VCRs didn’t become a household necessity until the mid-eighties. The DVD player didn’t statistically replace the VCR in American households until 2006, nine years after the first players went on the market. Cell phones, personal computers, GPS navigation devices—all took a similar time frame to go from innovation to mass adoption.

Consider, as an alternate scenario, the story of Chad Hurley, Steve Chen, and Jawed Karim, three former employees of the online payment site PayPal, who decided in early 2005 that the Web was ripe for an upgrade in the way it handled video and sound. Video, of course, was not native to the Web, which had begun its life fifteen years before as a platform for academics to share hypertext documents. But over the years, video clips had begun to trickle their way online, thanks to new video standards that emerged, such as Quick-Time, Flash, or Windows Media Player. But the mechanisms that allowed people to upload and share their own videos were too challenging for most ordinary users. So Hurley, Chen, and Karim cobbled together a rough beta for a service that would correct these deficiencies, raised less than $10 million in venture capital, hired about two dozen people, and launched YouTube, a website that utterly transformed the way video information is shared online. Within sixteen months of the company’s founding, the service was streaming more than 30 million videos a day. Within two years, YouTube was one of the top-ten most visited sites on the Web. Before Hurley, Chen, and Karim hit upon their idea for a start-up, video on the Web was as common as subtitles on television. The Web was about doing things with text, and uploading the occasional photo. YouTube brought Web video into the mainstream.

Now compare the way these two ideas—HDTV and YouTube— changed the basic rules of engagement for their respective platforms. Going from analog television to HDTV is a change in degree, not in kind: there are more pixels; the sound is more immersive; the colors are sharper. But consumers watch HDTV the exact same way they watched old-fashioned analog TV. They choose a channel, and sit back and watch. YouTube, on the other hand, radically altered the basic rules of the medium. For starters, it made watching video on the Web a mass phenomenon. But with YouTube you weren’t limited to sitting and watching a show, television-style; you could also upload your own clips, recommend or rate other clips, get into a conversation about them. With just a few easy keystrokes, you could take a clip running on someone else’s site, and drop a copy of it onto your own site. The technology allowed ordinary enthusiasts to effectively program their own private television networks, stitching together video clips from all across the planet.

Some will say that this is merely a matter of software, which is intrinsically more adaptable than hardware like televisions or cellular phones. But before the Web became mainstream in the mid-1990s, the pace of software innovation followed the exact same 10/10 pattern of development that we saw in the spread of other twentieth-century technologies. The graphical user interface, for instance, dates back to a famous technology demo given by pioneering computer scientist Doug Engelbart in 1968. During the 1970s, many of its core elements—like the now ubiquitous desktop metaphor—were developed by researchers at Xerox-PARC. But the first commercial product with a fully realized graphical user interface didn’t ship until 1981, in the form of the Xerox Star workstation, followed by the Macintosh in 1984, the first graphical user interface to reach a mainstream, if niche, audience. But it wasn’t until the release of Windows 3.0 in 1990—almost exactly ten years after the Xerox Star hit the market—that graphical user interfaces became the norm. The same pattern occurs in the developmental history of other software genres, such as word processors, spreadsheets, or e-mail clients. They were all built out of bits, not atoms, but they took just as long to go from idea to mass success as HDTV did.

There are many ways to measure innovation, but perhaps the most elemental yardstick, at least where technology is concerned, revolves around the job that the technology in question lets you do. All other things being equal, a breakthrough that lets you execute two jobs that were impossible before is twice as innovative as a breakthrough that lets you do only one new thing. By that measure, YouTube was significantly more innovative than HDTV, despite the fact that HDTV was a more complicated technical problem. YouTube let you publish, share, rate, discuss, and watch video more efficiently than ever before. HDTV let you watch more pixels than ever before. But even with all those extra layers of innovation, YouTube went from idea to mass adoption in less than two years. Something about the Web environment had enabled Hurley, Chen, and Karim to unleash a good idea on the world with astonishing speed. They took the 10/10 rule and made it 1/1.