CHAPTER 1 Welcome to the Most Important Conversation of Our Time
Bacteria, on the other hand, aren’t great learners. Their DNA specifies not only the design of their hardware, such as sugar sensors and flagella, but also the design of their software. They never learn to swim toward sugar; instead, that algorithm was hard-coded into their DNA from the start. There was of course a learning process of sorts, but it didn’t take place during the lifetime of that particular bacterium. Rather, it occurred during the preceding evolution of that species of bacteria, through a slow trial-and-error process spanning many generations, where natural selection favored those random DNA mutations that improved sugar consumption. (Location 504)
Tags: bacteria
Note: .bacteria bacteria evolved through natural selection rather than learned
bacteria are an example of what I’ll call “Life 1.0”: life where both the hardware and software are evolved rather than designed. (Location 510)
You and I, on the other hand, are examples of “Life 2.0”: life whose hardware is evolved, but whose software is largely designed. (Location 511)
Perhaps your school allows you to select a foreign language: Do you want to install a software module into your brain that enables you to speak French, or one that enables you to speak Spanish? Do you want to learn to play tennis or chess? Do you want to study to become a chef, a lawyer or a pharmacist? (Location 516)
Tags: favorite, learning
Note: .learning view learning as the installation of software module is enhance capabilities is a select area
This ability of Life 2.0 to design its software enables it to be much smarter than Life 1.0. High intelligence requires both lots of hardware (made of atoms) and lots of software (made of bits). The fact that most of our human hardware is added after birth (through growth) is useful, since our ultimate size isn’t limited by the width of our mom’s birth canal. (Location 519)
Tags: learning
Note: .learning most human software is added after birth. Our learning potential isnt limit by our software at birth
despite the most powerful technologies we have today, all life forms we know of remain fundamentally limited by their biological hardware. None can live for a million years, memorize all of Wikipedia, understand all known science or enjoy spaceflight without a spacecraft. ... requires life to undergo a final upgrade, to Life 3.0, which can design not only its software but also its hardware. In other words, Life 3.0 is the master of its own destiny, finally fully free from its evolutionary shackles. (Location 542)
Note: we are limited by our inability to upgrade our bodies
In summary, we can divide the development of life into three stages, distinguished by life’s ability to design itself: Life 1.0 (biological stage): evolves its hardware and software Life 2.0 (cultural stage): evolves its hardware, designs much of its software Life 3.0 (technological stage): designs its hardware and software (Location 552)
The fear of machines turning evil is another red herring. The real worry isn’t malevolence, but competence. A superintelligent AI is by definition very good at attaining its goals, whatever they may be, so we need to ensure that its goals are aligned with ours. You’re probably not an ant hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green energy project and there’s an anthill in the region to be flooded, too bad for the ants. The beneficial-AI movement wants to avoid placing humanity in the position of those ants. (Location 789)
Tags: favorite, goals, ai
Note: .ai we need to ensure the goals of ai are aligned with ours
boring pseudo-controversies caused by misunderstandings. For example, never waste time arguing about “life,” “intelligence,” or “consciousness” before ensuring that you and your protagonist are using these words to mean the same thing! (Location 853)
Tags: language, words
Note: .words .language a common definition of words is key for fruitful conversation
CHAPTER 2 Matter Turns Intelligent
Hydrogen …, given enough time, turns into people. (Location 868)
Tags: time
Note: .time the possibilities are endless on a large timescale
intelligence = ability to accomplish complex goals (Location 881)
Tags: goals, intelligence
Note: .intelligence there are many definitions for intelligence. Here is is defined as the ability to achiev complex goals
The memory in your brain works very differently from computer memory, not only in terms of how it’s built, but also in terms of how it’s used. Whereas you retrieve memories from a computer or hard drive by specifying where it’s stored, you retrieve memories from your brain by specifying something about what is stored. (Location 1034)
Tags: memory
Note: .memory we retrieve memories by spcifying something about them, like a google search
computation is a pattern in the spacetime arrangement of particles, and it’s not the particles but the pattern that really matters! Matter doesn’t matter. In other words, the hardware is the matter and the software is the pattern. This substrate independence of computation implies that AI is possible: intelligence doesn’t require flesh, blood or carbon atoms. (Location 1141)
Once technology gets twice as powerful, it can often be used to design and build technology that’s twice as powerful in turn, triggering repeated capability doubling in the spirit of Moore’s law. The cost of information technology has now halved roughly every two years for about a century, enabling the information age. (Location 1376)
Tags: progress, moores law
Note: once tech gets twice as good it can then be used to design tech twice as good again
CHAPTER 3 The Near Future: Breakthroughs, Bugs, Laws, Weapons and Jobs
If we don’t change direction soon, we’ll end up where we’re going. Irwin Corey (Location 1384)
Tags: quote
Note: .quote
Throughout human history, we’ve relied on the same tried-and-true approach to keeping our technology beneficial: learning from mistakes. We invented fire, repeatedly messed up, and then invented the fire extinguisher, fire exit, fire alarm and fire department. We invented the automobile, repeatedly crashed, and then invented seat belts, air bags and self-driving cars. Up until now, our technologies have typically caused sufficiently few and limited accidents for their harm to be outweighed by their benefits. (Location 1570)
Tags: trial and error, mistakes, cars, fire
Note: .fire .cars humanity has progressed through trial and error. We may not be able to take such an approach with ai
as technology grows more powerful, we should rely less on the trial-and-error approach to safety engineering. In other words, we should become more proactive than reactive, investing in safety (Location 1577)
Tags: trial and error, ai
Note: .ai
Car accidents alone took over 1.2 million lives in 2015, (Location 1658)
Tags: cars
Note: .cars
In the United States, with its high safety standards, motor vehicle accidents killed about 35,000 people last year—seven (Location 1659)
Tags: cars
Note: .cars
“Digital Athens.” The reason that the Athenian citizens of antiquity had lives of leisure where they could enjoy democracy, art and games was mainly that they had slaves to do much of the work. But why not replace the slaves with AI-powered robots, creating a digital utopia that everyone can enjoy? Erik’s AI-driven economy would not only eliminate stress and drudgery and produce an abundance of everything we want today, but it would also supply a bounty of wonderful new products and services that today’s consumers haven’t yet realized that they want. (Location 2003)
the combined wealth of the bottom half of the world’s population (over 3.6 billion people) is the same as that of the world’s eight richest people (Location 2017)
Tags: inequality, wealth, poor
Note: .poor .wealth .inequality
digital technology drives inequality in three different ways. First, by replacing old jobs with ones requiring more skills, technology has rewarded the educated: since the mid-1970s, salaries rose about 25% for those with graduate degrees while the average high school dropout took a 30% pay cut.45 Second, they claim that since the year 2000, an ever-larger share of corporate income has gone to those who own the companies as opposed to those who work there—and that as long as automation continues, we should expect those who own the machines to take a growing fraction of the pie. (Location 2025)
Tags: inequality, jobs
No-longer-needed horses were slaughtered and not replaced, causing the U.S. equine population to collapse from about 26 million in 1915 to about 3 million in 1960.55 As mechanical muscles made horses redundant, will mechanical minds do the same to humans? (Location 2127)
Tags: horses, ai
Note: .ai .horses equine population dropped from 26 to 3 million over 50 years
Jobs can provide people with more than just money. Voltaire wrote in 1759 that “work keeps at bay three great evils: boredom, vice and need.” (Location 2170)
Tags: work
Note: .work
The greater our breakout paranoia, the less AI-invented technology we can use. To play it safe, as the Omegas did in the prelude, we humans can only use AI-invented technology that we ourselves are able to understand and build. A drawback of the enslaved-god scenario is therefore that it’s more low-tech than those with free superintelligence. (Location 3018)
The Dreaded Comparison: Human and Animal Slavery, Marjorie Spiegel argues that like human slaves, non-human animals are subjected to branding, restraints, beatings, auctions, the separation of offspring from their parents, and forced voyages. Moreover, despite the animal-rights movement, we keep treating our ever-smarter machines as slaves without a second thought, and talk of a robot-rights movement is met with chuckles. Why? (Location 3049)
Tags: animals
Note: .animals
How would a conqueror AI eliminate us? Probably by a method that we wouldn’t even understand, at least not until it was too late. Imagine a group of elephants 100,000 years ago discussing whether those recently evolved humans might one day use their intelligence to kill their entire species. “We don’t threaten humans, so why would they kill us?” they might wonder. Would they ever guess that we would smuggle tusks across Earth and carve them into status symbols for sale, even though functionally superior plastic materials are much cheaper? (Location 3099)
Tags: ai
Note: It's impossible to predict how AI might conquer humans. Elephants would never have guessed that humans would cut off their tusks and sell them
although we may feel very differently about the descendant and conqueror scenarios, the two are actually remarkably similar in the grand scheme of things: during the billions of years ahead of us, the only difference lies in how the last human generation(s) are treated: how happy they feel about their lives and what they think will happen once they’re gone. (Location 3169)
Reversion Wouldn’t it be tempting to escape the perils of technology without succumbing to stagnant totalitarianism? Let’s explore a scenario where this was accomplished by reverting to primitive technology, inspired by the Amish. After the Omegas took over the world as in the opening of the book, a massive global propaganda campaign was launched that romanticized the simple farming life of 1,500 years ago. Earth’s population was reduced to about 100 million people by an engineered pandemic blamed on terrorists. The pandemic was secretly targeted to ensure that nobody who knew anything about science or technology survived. With the excuse of eliminating the infection hazard of large concentrations of people, Prometheus-controlled robots emptied and razed all cities. Survivors were given large tracts of (suddenly available) land and educated in sustainable farming, fishing and hunting practices using only early medieval technology. In the meantime, armies of robots systematically removed all traces of modern technology (including cities, factories, power lines and paved roads), and thwarted all human attempts to document or re-create any such technology. Once the technology was globally forgotten, robots helped dismantle other robots until there were almost none left. The very last robots were deliberately vaporized together with Prometheus itself in a large thermonuclear explosion. There was no longer any need to (Location 3233)
the story of nuclear radiation, nuclear EMP and nuclear winter all demonstrate that the greatest hazards may be ones we haven’t even thought of yet. It’s incredibly difficult to foresee all aspects of the aftermath, and how nuclear winter, infrastructure collapse, elevated mutation levels and desperate armed hordes might interact with other problems such as new pandemics, ecosystem collapse and effects we haven’t yet imagined. (Location 3312)
Tags: nuclear
Note: .nuclear drop in global temperatires,knock out electricity grid and radiation
THE BOTTOM LINE: The current race toward AGI can end in a fascinatingly broad range of aftermath scenarios for upcoming millennia. Superintelligence can peacefully coexist with humans either because it’s forced to (enslaved-god scenario) or because it’s “friendly AI” that wants to (libertarian-utopia, protector-god, benevolent-dictator and zookeeper scenarios). Superintelligence can be prevented by an AI (gatekeeper scenario) or by humans (1984 scenario), by deliberately forgetting the technology (reversion scenario) or by lack of incentives to build it (egalitarian-utopia scenario). Humanity can go extinct and get replaced by AIs (conqueror and descendant scenarios) or by nothing (self-destruction scenario). (Location 3352)
Tags: ai
Note: .ai
CHAPTER 6 Our Cosmic Endowment: The Next Billion Years and Beyond
To me, the most inspiring scientific discovery ever is that we’ve dramatically underestimated life’s future potential. Our dreams and aspirations need not be limited to century-long life spans marred by disease, poverty and confusion. Rather, aided by technology, life has the potential to flourish for billions of years, not merely here in our Solar System, but also throughout a cosmos far more grand and inspiring than our ancestors imagined. Not even the sky is the limit. (Location 3366)
Tags: immortality
sneaky Hail Mary approach to expanding even faster than any of the above methods will permit: using Hans Moravec’s “cosmic spam” scam from chapter 4. By broadcasting a message that tricks naive freshly evolved civilizations into building a superintelligent machine that hijacks them, a civilization can expand essentially at the speed of light, the speed at which their seductive siren song spreads through the cosmos. Since this may be the only way for advanced civilizations to reach most of the galaxies within their future light cone and they have little incentive not to try it, we should be highly suspicious of any transmissions from extraterrestrials! In Carl Sagan’s book Contact, we Earthlings used blueprints from aliens to build a machine we didn’t understand—I don’t recommend doing this … (Location 3741)
Tags: ai, favorite, cosmic spam
“the history of human technological civilization is measured in centuries—and it may be only one or two more centuries before humans are overtaken or transcended by inorganic intelligence, which will then persist, continuing to evolve, for billions of years. … We would be most unlikely to ‘catch’ it in the brief sliver of time when it took organic form.” (Location 4057)
Tags: humans
Note: .humans there is high probability that other species in the galaxy would have advanced beyond biological bodies
we saw how intergalactic travel becomes much easier once people and other intelligent entities can be transmitted in digital form, potentially making us masters of our own destiny not only in our Solar System or the Milky Way Galaxy, but also in the cosmos. (Location 4080)
Tags: travel
Note: .travel
Above we considered the very real possibility that we’re the only high-tech civilization in our Universe. Let’s spend the rest of this chapter exploring this scenario, and the huge moral responsibility it entails. This means that after 13.8 billion years, life in our Universe has reached a fork in the road, facing a choice between flourishing throughout the cosmos or going extinct. If we don’t keep improving our technology, the question isn’t whether humanity will go extinct, but how. What will get us first—an asteroid, a supervolcano, the burning heat of the aging Sun, or some other calamity (Location 4082)
Tags: humanity
Note: .humanity if we are the only civilisation in the galaxy the embracing tech will be the way to avoid inevitable extinction
If we don’t improve our technology, the question isn’t whether humanity will go extinct, but merely how: will an asteroid, a supervolcano, the burning heat of the aging Sun or some other calamity get us first? (Location 4116)
Tags: humans
Note: .humans
Life is a journey, not a destination. Ralph Waldo Emerson (Location 4125)
Tags: life
Note: .life
If a life form copies itself and the copies do the same, then the total number will keep doubling at regular intervals until the population size bumps up against resource limitations or other problems. Repeated doubling soon produces huge numbers: if you start with one and double just three hundred times, you get a quantity exceeding the number of particles in our Universe. This means that not long after the first primitive life form appeared, huge quantities of matter had come alive. Sometimes the copying wasn’t perfect, so soon there were many different life forms trying to copy themselves, competing for the same finite resources. Darwinian evolution had begun. (Location 4191)
Charles Darwin elegantly explained why: since the most efficient copiers outcompete and dominate the others, before long any random life form you look at will be highly optimized for the goal of replication. (Location 4198)
Tags: reproduction, humans
Note: .humans .reproduction
In other words, the real risk with AGI isn’t malice but competence. A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. (Location 4309)
Tags: ai
Note: .ai
people don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants. Most researchers therefore argue that if we ever end up creating superintelligence, then we should make sure it’s what AI-safety pioneer Eliezer Yudkowsky has termed “friendly AI”: AI whose goals are aligned with ours. (Location 4311)
Tags: animals, ai
Note: .ai humans dont care about flooding ant hills to create a dam. Its important our goals align with that of ai
However, the point is that it will resist being shut down if you give it any goal that it needs to remain operational to accomplish—and this covers almost all goals! (Location 4405)
Utilitarianism: Positive conscious experiences should be maximized and suffering should be minimized. Diversity: A diverse set of positive experiences is better than many repetitions of the same experience, even if the latter has been identified as the most positive experience possible. Autonomy: Conscious entities/societies should have the freedom to pursue their own goals unless this conflicts with an overriding principle. Legacy: Compatibility with scenarios that most humans today would view as happy, incompatibility with scenarios that essentially all humans today would view as terrible. (Location 4493)
Traditionally, utilitarianism is taken to mean “the greatest happiness for the greatest number of people,” (Location 4498)
If animals are conscious, then what are predators supposed to eat? (Location 4532)
Tags: animals, vegans
Note: .vegans
Intelligence is the ability to accomplish complex goals. (Location 4636)
Tags: intelligence
Note: .intelligence
Aligning machine goals with our own involves three unsolved problems: making machines learn them, adopt them and retain them. (Location 4641)
when people ask about the meaning of life as if it were the job of our cosmos to give meaning to our existence, they’re getting it backward: It’s not our Universe giving meaning to conscious beings, but conscious beings giving meaning to our Universe. (Location 5194)
Tags: life
Note: .life
I made a New Year’s resolution for 2014 that I was no longer allowed to complain about anything without putting some serious thought into what I could personally do about it, (Location 5259)
Tags: complain
Note: .complain dont complain unless you will do something about it
“Chatham House Rule,” which prohibits participants from subsequently revealing who said what. (Location 5345)
Investments in AI should be accompanied by funding for research on ensuring its beneficial use, including thorny questions in computer science, economics, law, ethics, and social studies, such as: (a) How can we make future AI systems highly robust, so that they do what we want without malfunctioning or getting hacked? (b) How can we grow our prosperity through automation while maintaining people’s resources and purpose? (c) How can we update our legal systems to be more fair and efficient, to keep pace with AI, and to manage the risks associated with AI? (d) What set of values should AI be aligned with, and what legal and ethical status should it have? (Location 5461)
Tags: legal, ethics, ai
while avoiding pitfalls. Erik pointed out that according to game theory, positive visions form the foundation of a large fraction of all collaboration in the world, from marriages and corporate mergers to the decision of independent states to form the USA. (Location 5540)