It took years for the Internet to reach its first 100 computers. Today, 100 new ones join each second. And running deep within the silicon souls of most of these machines is the work of a technical wizard of remarkable power, a man described as a genius and a bully, a spiritual leader and a benevolent dictator.
Linus Torvalds - who in person could be mistaken for just another a paunchy, middle-aged suburban dad who happens to have a curiously large collection of stuffed penguin dolls - looms over the future of computing much as Bill Gates and the late Steve Jobs loom over its past and present. For Linux, the operating system that Torvalds created and named after himself, has come to dominate the exploding online world, making it more popular overall than rivals from Microsoft or Apple.
But while Linux is fast, flexible and free, a growing chorus of critics warn that it has security weaknesses that could be fixed but haven't been. Worse, as Internet security has surged as a subject of international concern, Torvalds has engaged in an occasionally profane standoff with experts on the subject. One group he has dismissed as "masturbating monkeys." In blasting the security features produced by another group, he said in a public post, "Please just kill yourself now. The world would be a better place."
There are legitimate philosophical differences amid the harsh words. Linux has thrived in part because of Torvalds's relentless focus on performance and reliability, both of which could suffer if more security features were added. Linux works on almost any chip in the world and is famously stable as it manages the demands of many programs at once, allowing computers to hum along for years at a time without rebooting.
Yet even among Linux's many fans there is growing unease about vulnerabilities in the operating system's most basic, foundational elements - housed in something called "the kernel" - which Torvalds has personally managed since its creation in 1991. Even more so, there is concern that Torvalds's approach to security is too passive, bordering on indifferent.
"Linus doesn't take security seriously, it's yet another concern in his mind, and he's surrounded himself with people who share those views," said Daniel Micay, a Toronto-based security researcher whose company, Copperhead, is developing a hardened version of the Android mobile operating system, which is based on Linux. "There are a lot of kernel developers who do really care about security, but they're not the ones making the calls."
The rift between Torvalds and security experts is a particular source of worry for those who see Linux becoming the dominant operating system at a time when technology is blurring the borders between the online and offline worlds. Much as Windows long was the standard for personal computers, Linux runs on most of the Internet's servers. It also operates on medical equipment, sensitive databases and computers on many kinds of vehicles, including tiny drones and warships.
"If you don't treat security like a religious fanatic, you are going to be hurt like you can't imagine. And Linus never took seriously the religious fanaticism around security," said Dave Aitel, a former National Security Agency research scientist and founder of Immunity, a Florida-based security company.
Torvalds - who despite his history of blistering online exchanges is genial in person, often smiling from behind round-framed glasses - indeed appears to be the opposite of a religious fanatic as he zips around his adopted home town of Portland, Oregon, in a yellow Mercedes convertible. The license plate is "DAD OF3," but it's the plate holder that better captures his sly sense of humor, somehow mixing self-confidence with self-mockery. "MR. LINUX," it reads, "KING OF GEEKS."
Over several hours of conversation, Torvalds, 45, disputed suggestions that security isn't important to him or to Linux, but he acknowledged being "at odds" with some security experts. His broader message was this: Security of any system can never be perfect. So it always must be weighed against other priorities - such as speed, flexibility, ease of use - in a series of inherently nuanced trade-offs. This is a process, Torvalds suggested, poorly understood by his critics.
"The people who care most about this stuff are completely crazy. They are very black and white," he said, speaking with a slight Nordic accent from his native Finland. "Security in itself is useless. . . . The upside is always somewhere else. The security is never the thing that you really care about."
When the interviewer asked whether Linux - designed in an era before hacking had become a major criminal enterprise, a tool of war and constant threat to the privacy of billions of people - was due for a security overhaul after 24 years, Torvalds replied, "You're making sense, and you may even be right."
But what followed was a bracing example of why Torvalds believed the interviewer was wrong: Imagine, Torvalds said, that terrorists exploited a flaw in the Linux kernel to cause a meltdown at a nuclear power plant, killing millions of people.
"There is no way in hell the problem there is the kernel," Torvalds said. "If you run a nuclear power plant that can kill millions of people, you don't connect it to the Internet."
Or if you do, he continued, you build robust defenses such as firewalls and other protections beyond the operating system so that a bug in the Linux kernel isn't enough to create a catastrophe.
"If I have to worry about that kind of scenario happening," Torvalds added with a wry grin, "I won't get any work done."
Even without a potential nuclear disaster, the stakes are high. Operating system kernels are the most essential code on any computer, allowing hardware to work smoothly with multiple pieces of software. This makes kernels uniquely powerful - they can override the safeguards on any other program, meaning nothing on a computer is truly secure if the operating system kernel is not.
Now, consider this: The Linux kernel runs on the New York Stock Exchange, every Android smartphone and nearly all of the world's supercomputers. Most of the rapidly expanding universe of connected devices use Linux, as do many of the world's biggest companies, including Google, Facebook and Amazon.com. The tech-heavy US economy, many would argue, also depends on the smooth functioning of Linux.
Even more broadly, the battle over Linux security is a battle over the future of the online world. At a time when leading computer scientists are debating whether the Internet is so broken that it needs to be replaced, the network is expanding faster than ever, layering flaw upon flaw in an ever-expanding web of insecurity. Perhaps the best hope for fixing this, some experts argue, lies in changing the operating system that - more than any other - controls these machines.
But first, they have to change the mind of Linus Torvalds.
Stories about tech titans tend toward pat narratives: the blazing discovery, the shrewd business moves, the thrilling triumph after years of struggle. The story of Torvalds, and by extension Linux, is almost the opposite. He was a shy, brainy college student who built something with no obvious market - a new operating system in a world that already had Windows, Mac OS and Unix - and gave it away. It wasn't a business. It was a hobby.
There is a telling moment in his autobiography, "Just for Fun," written with journalist David Diamond, that captures this spirit of naive experimentation. In early 1992, about six months after announcing the creation of Linux, Torvalds posted an online message asking anyone using the operating system to send him a postcard.
Soon, his mailbox in Helsinki overflowed with hundreds of postcards from the United States, New Zealand, Japan and beyond. It was the first time his sister and mother, with whom Torvalds shared an apartment, realized he was up to something big. Torvalds had told them little about what he was doing in his bedroom, perched over his computer, all hours of the day and night.
This diffuse and ever-growing community of users proved to be the magic that powered Linux. The operating system had its inherent virtues - it was simple and clean; tech enthusiasts worldwide fell in love with its elegance - but more important it was an "open-source" project. That meant anybody could use it, alter it and even make a new version without paying a cent, without even asking permission. Linux soon became, in a phrase from Torvalds's autobiography, the "world's largest collaborative project," with contributors numbering in the hundreds of thousands. They drove the growth of Linux long after Torvalds might have lost interest.
"In 1992," he said, "I was like, 'Wow, it does everything I wanted it to do. What now?' "
Torvalds had little choice but to become the general of an unruly volunteer army. As the kernel grew from 10,000 lines of code to 19 million, Torvalds created an elaborate and remarkably functional system that, every couple of months, offered a free update of the Linux kernel to anyone who wanted it.
Based on the kernel, others then tailored the operating systems to their own tastes and purposes, adding even more lines of code that collectively became fully fledged "distributions" of Linux that ran on various types of computers. The price of admission to this elaborate process was faith in Torvalds, although some went the extra step of making some kind of offering to their hero: free computer gear, company T-shirts or penguin dolls (because a squat, cheerful-looking aquatic waterfowl - usually sitting lazily on its butt - was the symbol of Linux).
Years of spinning such devotion into well-honed computer code has shaped a development process that is gradual and evolutionary. The goal is to fix problems and adapt to new hardware, while never causing malfunctions. This idea is enshrined, somewhat antiseptically, in Torvalds's often-stated prohibition against what he calls "breaking user space" - essentially, causing something that a user depends on to stop working. But there is nothing antiseptic about his reaction when somebody violates this cardinal rule.
One notorious exchange came in December 2012, when Torvalds publicly raged to a regular Linux contributor who had proposed a flawed patch: "WE DO NOT BREAK USERSPACE! Seriously. How hard is this rule to understand? We particularly don't break user space with TOTAL CRAP. I'm angry, because your whole email was so _horribly_ wrong, and the patch that broke things was so obviously crap."
Torvalds sometimes expresses regret about his rhetorical excesses, but the emotion that boils up in these moments is unmistakably real, fueled by his fierce sense of guardianship over Linux.
The effect of Torvalds's approach to managing the kernel - defensive, gradualist, sometimes cranky - chilled debate about the security of Linux even as it became a bigger, richer target for hackers. The result, critics argue, is that while Linux in its early days was widely considered a safer choice than Windows or other commercial operating systems, the edge has dwindled and perhaps disappeared.
"While I don't think that the Linux kernel has a terrible track record, it's certainly much worse than a lot of people would like it to be," said Matthew Garrett, principal security engineer for CoreOS, a San Francisco company that produces an operating system based on Linux. At a time when research into protecting software has grown increasingly sophisticated, Garrett said, "very little of that research has been incorporated into Linux."
Versions of Linux have proved vulnerable to some of the most serious bugs in recent years, including Heartbleed and Shellshock. AshleyMadison.com, the site that facilitates extramarital affairs and suffered an embarrassing data breach in July, was reportedly running Linux on its servers, as do many companies.
Those problems did not involve the kernel itself, but experts say the kernel has become a popular target for hackers building "botnets," giant networks of computers that can be organized to attack targets. Experts also say that government spies - and the companies that sell them surveillance tools - have turned their attention to the kernel as Linux has spread.
The Security Intelligence Response Team for Akamai, a leading Internet content delivery company, spoke bluntly on the rising vulnerability of Linux in September when it announced the discovery of a massive botnet that attacked up to 20 targets worldwide each day.
"A decade ago, Linux was seen as the more secure alternative to Windows environments, which suffered the lion's share of attacks at the time," Akamai's security team wrote. But the sharply rising popularity of Linux has meant "the potential opportunity and rewards for criminals has also grown. Attackers will continue to evolve their tactics and tools and security professionals should continue to harden their Linux based systems accordingly."
But harden how?
Even if Torvalds originally considered Linux just a hobby, others saw gold. Red Hat, a North Carolina company, released a version that became widely deployed across corporate America and at many government agencies. A South African businessman released Ubuntu, a popular desktop version of Linux, in 2004. Traditional tech giants - IBM, Intel, Oracle - also made big bets on Linux.
As Linux took off, Torvalds took something of a detour, leaving Finland with his wife and first child in 1997 to work for a Silicon Valley start-up. But he never gave up control of Linux and, in 2003, Torvalds joined an Oregon-based nonprofit group that later merged with another group to become the Linux Foundation, which promotes the overall development of the operating system.
(Torvalds also was granted stock options by Red Hat and one other company selling Linux products, making him comfortable enough to pay cash for a new house but not nearly as rich as Gates or other top tech executives.)
The rising popularity of the operating system sparked efforts to toughen its defenses. Companies that sold versions of Linux had security teams add protections. Even the US government, which has adopted Linux on many of its computers, had the NSA develop advanced security features, called SELinux, making the operating system more suitable for sensitive work. (This was a defensive effort, say security experts, not part of the NSA's spying mission.)
The problem, as critics pointed out, was that these protections relied on building walls around the operating system that, however high or thick, could not possibly stop all comers. Those who penetrated gained control of the Linux kernel itself, meaning the hackers could make a compromised computer do anything they wanted - even if every other piece of software on the machine was flawlessly protected. According to veteran security engineer Kees Cook, this made the Linux kernel "the ultimate attack surface."
"Vulnerabilities in the kernel generally meant that an attacker with access to a flawed kernel interface" - meaning a bug in the code - "could bypass nearly every other security policy in place and take total control of the system," said Cook, who from 2006 to 2011 worked for Canonical, which supported the Ubuntu version of Linux, and later joined Google to work on kernel security.
Another expert, Brad Spengler of Grsecurity, used satire to make a similar point in 2009, circulating a spoof of an illustration that had been used in promotional material for SELinux. The original version showed the kernel wrapped in protective layers that repelled attacks, but the spoof overlaid images of "Sesame Street" characters happily getting through these layers to menace the kernel. Ernie, Bert, Elmo, Oscar the Grouch and the Cookie Monster represented "Blackhats with kernel exploits," the text read, meaning that malicious hackers armed with the computer bugs that offered a way past even the heaviest defenses.
Spengler later acknowledged that the spoof was "childish" but said it "at least was more accurate" than the original diagram. To drive the point home, he soon demonstrated how nearly a dozen known Linux coding bugs could be exploited by malicious hackers to defeat external defenses and take control of the kernel.
The response from Torvalds to such concerns did little to calm Spengler or other critics. In an era when software makers increasingly were candid about security flaws, issuing alerts that detailed problems and explicitly urged people to install safer updates, Torvalds had a different approach. In messages that accompanied each new version of the Linux kernel, he described various improvements but would not call attention to the ones that fixed security problems.
This frustrated security experts who saw transparency as a key part of their mission. They reasoned that if a software maker knew about a bug, then malicious hackers almost certainly did, too, and had been exploiting it for months or even years. Failing to warn users directly and forcefully made it harder for them to protect themselves.
Torvalds, however, has held his ground on this issue. He knew there were countless versions of Linux running across the world and that weeks or months often passed before updates reached individual machines. Publicly revealing details about computer bugs - even if fixed in the latest release - gave an edge to malicious hackers until the software fixes arrived, he believed.
Torvalds also resisted suggestions that security deserved a special place in the hierarchy of concerns faced by software makers. All flaws, in his view, were equally serious. This attitude was enshrined in a public posting in July 2008 that said: "I personally consider security bugs to be just 'normal bugs.' I don't cover them up, but I also don't have any reason what-so-ever to think it's a good idea to track them and announce them as something special."
This comment - often recalled in shorthand as Torvalds's declaration that "bugs are just bugs" - is the line most often quoted by his critics as they seek to explain what they consider a persistent, almost willful tone-deafness on security. These experts say that although most bugs are mere glitches that might cause a function to fail or a program to crash, others are far more serious, offering malicious hacker an opening to take total control of computers.
Those who specialize in security think in terms of categories of bugs. Each one is a cousin of others, some known, some not yet discovered, based on what functions they exploit. By studying each new one carefully, these experts believe it is possible to defeat entire classes of bugs with a single fix.
But in his recent interview with The Washington Post, Torvalds rejected the notion that bugs could be usefully sorted into categories. "I refuse to waste a second of my life or any other developer's life trying to classify something that can't be classified," he said.
Rather than trying to create protections against "classes" of bugs, Torvalds seeks to inspire better coding in general. "Well-written code just doesn't have a lot of special cases. It just does the right thing. . . . It just works in all situations."
As for the exceptions, Torvalds shrugs: "Sometimes reality bites you in the ass. Sometimes it's just bad coding."
There has been a recurring subplot in the history of the online world: For every advance, every thrilling new vista of possibility, there are those who warn of dangers lurking in shadows ahead. To borrow from Greek mythology, they are the Cassandras - often right in their prophecies, yet generally ignored until disaster actually arrives.
The leading Cassandra in the Linux story has been Spengler, whose critique of SELinux featured malevolent "Sesame Street" characters in 2009. He and a pair of collaborators, who worked for an affiliated project called the PaX Team, had over several years developed patches that dramatically hardened Linux. The best known of these techniques, called address space layout randomization, reshuffled each computer's memory regularly. So even when hackers found their way to the kernel, they became so disoriented that it was difficult to steal files or implant malicious code.
Despite providing a steady supply of defensive innovations, Spengler did not become a popular figure within the upper reaches of the Linux community, among whom he was seen as extreme in his views and sometimes brittle in his manner. Plus the Grsecurity and PaX patches, though universally regarded as cutting-edge security measures, can slow computer performance. Some also caused some features to not work as well, violating Torvalds's cardinal rule against "breaking user space."
Torvalds said recently of Spengler, "He's one of the crazy security people, no doubt about it, and so we've butted heads."
He added that Spengler "is somebody I respect from a technical standpoint," but a split emerged that was philosophical and, eventually, personal. Torvalds was happy to let Spengler's project toil on the fringes of a sprawling Linux empire, but Torvalds showed little interest in overhauling the kernel itself to address complaints from the security community, especially if that meant exacting a significant price in operating system performance.
"The market for that is pretty small in the end," he later said of Spengler's project. "Most people don't want the Grsecurity system."
The limited consumer demand for security was not news to anybody who worked in the field. Spengler often lamented how, as Linux spawned a multibillion-dollar industry, he and his colleagues struggled to raise enough in donations to underwrite their work.
"People don't really care that much," Spengler later said. "All of the incentives are totally backward, and the money isn't going where it's supposed to. The problem is just going to perpetuate itself."
Because the Linux kernel is not produced by a business, it doesn't respond to market conditions in a conventional way, but it is unquestionably shaped by incentives - and most of all, by Torvalds's priorities.
To carry out this vision, Torvalds has surrounded himself with dozens of code "maintainers," each of whom help manage different elements of the operating system. Anyone with an idea for improving Linux can craft the relevant code and submit it to a maintainer, who vets each proposal before sending the best ones upward to Torvalds himself.
From his home office above a three-car garage, Torvalds then approves - and occasionally rejects - the changes submitted by the maintainers and consolidates them before releasing the next version. Each new release typically affects hundreds of thousands of lines of code, and each change carries the risk of creating new bugs.
Though once largely a volunteer effort, top maintainers today typically have day jobs with tech companies that have a stake in the growth of the operating system and pay salaries to developers to support that common goal. But the Linux development process remains decentralized, relying heavily on the individual interests and initiative.
Even many Linux enthusiasts see a problem with this from a security perspective: There is no systemic mechanism for identifying and remedying problems before hackers discover them, or for incorporating the latest advances in defensive technologies. And there is no chief security officer for the Linux kernel.
"Security is an easy problem to ignore, and maybe everyone thinks somebody else should do it," said Andrew Lutomirski, a maintainer for part of the Linux kernel and an advocate for introducing better defenses overall. "There certainly are people who have security as a much higher priority than Linus Torvalds does."
Spengler's quest to improve overall Linux security peaked in 2010, when he spoke at a Linux conference in Boston. He prepared an extensive presentation titled "Linux Security in 10 Years" that detailed a range of ideas for keeping the kernel safe even when hacks inevitably happened.
The proposals seemed so urgent to Spengler that he expected to see top Linux maintainers, and possibly even Torvalds himself, in the audience. But when he looked out across the half-empty room, Spengler saw none of them. They were all off at other meetings.
"These guys are just working on things that they're interested in, and, for most of them, what they're interested in is not security," Spengler said recently. "My feeling with Linux is that they still treat security as a kind of nuisance thing."
In the years since Spengler and others began warning about the security of Linux, it has triumphed in the marketplace. Google released its first version of the Android mobile operating system, which is based on Linux, in 2007, allowing Torvalds's work to reach hundreds of millions of smartphones each year. Google also made the kernel the basis of Chrome OS, used in an increasingly popular category of cloud-based computers called Chromebooks.
Companies building the so-called Internet of Things - a massive universe including objects as diverse as online thermostats, heart-rate monitors and in-flight entertainment systems - also came to prefer Linux, which requires no fees that might drain away profits.
Those worried about security arguably have bigger problems than Linux, at least for now. Hackers are more likely to prey upon Oracle's Java and Adobe's Flash and Acrobat. But while many older, vulnerable pieces of software are being phased out, Linux is conquering new computing worlds.
As the operating system explodes in popularity, the debate over security has begun drawing attention beyond the world of Linux insiders. Sergey Bratus, as associate professor of computer science at Dartmouth College, argues that the kernel should be overhauled to streamline the code and to integrate the type of security features long advocated for by Spengler and other critics - even if the features slow computers down.
"In a device that I trust my life to, I would prefer this," Bratus said.
The most famous overhaul in software history came in 2002, when Gates ordered engineers at Microsoft to make security their top priority, a process that took several years and helped the famously hackable staples of that company's lineup get considerably safer.
The security situation with Linux is not nearly so dire as it was for Microsoft in 2002. It's also harder to see how such an overhaul could happen for an open-source project.
"Linux cannot just be turned around by a memo from Linus. He's not Bill Gates," Bratus said. "But a culture change is definitely needed before we start relying on these systems for everything."
The Linux Foundation did suffer an embarrassing hack in 2011. More recently, in 2014, Linux devotees were unhappy to discover that an Italian surveillance company, called Hacking Team, had swiftly turned a Linux exploit called "towelroot" into a skeleton key capable of gaining access to hundreds of millions of Android phones. This allowed Hacking Team to turn Android devices into powerful spying tools - capable of tracking targets, recording their conversations, rifling through their files, even taking pictures of them - on behalf of customers that included some of the world's most repressive governments.
"It works :)," wrote one Hacking Team developer to another in an e-mail about towelroot, according to a trove published by WikiLeaks. "Good job, thanks!"
The security stakes for the tech industry were underscored in the keynote address at an August summit on Linux security that pointedly compared the blinkered attitude of software makers today with that of the automobile industry in the 1960s, when cars functioned well but failed to protect people during unforeseen events such as crashes - leading directly to unnecessarily suffering and death.
"Let's not take 50 years to get to the point where computing is fun, powerful and a lot less likely to maim you when you make a mistake," concluded the keynote speaker, Konstantin Ryabitsev, who manages computer systems for the Linux Foundation.
'Dodo birds had it coming'
The Cassandra myth reached its tragic climax when she warned the Trojans that a giant wooden horse on their shores - supposedly a gift of surrender after a long siege - actually was filled with Greek warriors who soon would emerge to destroy Troy. The Trojans laughed and ridiculed Cassandra. They realized their error when it was too late.
In the days after Ryabitsev gave his August keynote address suggesting that software makers should rethink how they approach security, several Linux maintainers exchanged messages on a public mailing list about the possibility of revisiting some of the issues long raised by Spengler and other critics.
"We have some measures in place, although we are really not doing everything we can," wrote James Morris, maintainer of Linux's exterior defenses against attackers. As evidence of his concern, Morris cited occasions when bugs are discovered that are thwarted by Grsecurity - Spengler's patches - but not the main kernel released by Torvalds.
Spengler's name soon came up explicitly in the discussion, although participants correctly guessed that he had little interest in participating in such an effort now. ("I already did it in 2010," he said in an interview afterward. "It's kind of annoying that nothing came of it at the time. . . . I feel it would be better if they came up with their own ideas.")
Among those who were part of the discussion was Kees Cook, the Linux security engineer who now works for Google. He, too, recalled Spengler's call to action in 2010. Cook said there have been improvements since then - what he called "the low-hanging fruit" - but not enough.
"We're five years into that list, and we've only scratched the surface," said Cook, who in addition to his work for Google is a maintainer for Linux and part of a kernel security response team. "There is not the cultural shift I'd like to see."
Yet Cook and others say the chances of a major reconsideration of kernel security may now be better than ever. Edward Snowden's revelations about the extent of government spying - and about how the NSA took advantage of security weaknesses that experts often knew about but had failed to get fixed - have alarmed many in the tech community. So have the recent rash of high-profile hacks, such as the massive pilfering of personal data from the US government computers at the Office of Personnel Management.
"Given some of the evidence of the widespread security problems, it's a little easier to introduce the topic again," Morris said in an interview. "Now that we're looking at literally billions of Linux systems out there, I think people are starting to wake up."
The online discussion sparked by Morris in August has produced at least one tangible result: At the annual Linux Kernel Summit in Seoul last week, he and Cook gave a presentation that echoed many of Spengler's points from 2010 - only the list of problems needing serious attention had doubled, from six to 12. And this time, Torvalds and some of his top deputies were there.
There was a revealing moment, however, when Cook raised the possibility of adding an especially intrusive feature long offered by Grsecurity. Torvalds immediately spoke up, saying this was "the kind of idea that makes security people look crazy," according to LWN.net, a site that follows Linux issues.
Torvalds has often said - and reiterated after the meeting in Seoul - that he is open to new kernel defenses if the cost in performance is reasonable. But, of course, debate remains about what qualifies as "reasonable."
Torvalds himself still instinctively resists anything smacking of a dramatic overhaul, asking the world to trust the Linux development model's gradualist, evolutionary approach in which problems - and the trouble that often results - lead to computer code continually improving.
"I don't think you have an alternative," Torvalds said in the interview with The Post. "I don't think you can design things better than they evolve. . . . It really is working very well."
And what, he was asked, of the inevitable costs of evolution? The entire species, like the dodo bird, that have died off? Must progress come at such a price?
Torvalds smiles again: "Dodo birds had it coming."
But dodo birds, driven from existence after the arrival of humans ruined their native island habitat, had little chance to protect themselves from doom. What about the Trojans?
© 2015 The Washington Post