Nick Bostrom Quotes & Sayings

Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred. Contrast this with shorter timescales: most technologies that will have a big impact on the world in five or ten years from now are already in limited use, while technologies that will reshape the world in less than fifteen years probably exist as laboratory prototypes. Twenty — Nick Bostrom

Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound. — Nick Bostrom

We do not need to plug a fiber optic cable into our brains in order to access the Internet. Not only can the human retina transmit data at an impressive rate of nearly 10 million bits per second, but it comes pre-packaged with a massive amount of dedicated wetware, the visual cortex, that is highly adapted to extracting meaning from this information torrent and to interfacing with other brain areas for further processing. — Nick Bostrom

Whenever an observation is made that rules out some possible worlds, we remove the sand from the corresponding areas of the paper and redistribute it evenly over the areas that remain in play. Thus, the total amount of sand on the sheet never changes, it just gets concentrated into fewer areas as observational evidence accumulates. This is a picture of learning in its purest form. (To — Nick Bostrom

Had Mother Nature been a real parent, she would have been in jail for child abuse and murder. — Nick Bostrom

The Internet is a big boon to academic research. Gone are the days spent in dusty library stacks digging for journal articles. Many articles are available free to the public in open-access journal or as preprints on the authors' website. — Nick Bostrom

The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. — Nick Bostrom

There are some problems that technology can't solve. — Nick Bostrom

We would want the solution to the safety problem before somebody figures out the solution to the AI problem. — Nick Bostrom

Solving the value-loading problem is a research challenge worthy of some of the next generation's best mathematical talent. — Nick Bostrom

[D]umb evolutionary processes have dramatically amplified the intelligence in the human lineage even compared with our close relatives the great apes and our own humanoid ancestors; and there is no reason to suppose Homo sapiens to have reached the apex of cognitive effectiveness attainable in a biological system. — Nick Bostrom

Human individuals and human organizations typically have preferences over resources that are not well represented by an "unbounded aggregative utility function". A human will typically not wager all her capital for a fifty-fifty chance of doubling it. A state will typically not risk losing all its territory for a ten percent chance of a tenfold expansion. [T]he same need not hold for AIs. An AI might therefore be more likely to pursue a risky course of action that has some chance of giving it control of the world. — Nick Bostrom

Human working memory is able to hold no more than some four or five chunks of information at any given time. — Nick Bostrom

Table 2 When will human-level machine intelligence be attained?81 — Nick Bostrom

We find ourselves in a thicket of strategic complexity, surrounded by a dense mist of uncertainty. — Nick Bostrom

We should not be confident in our ability to keep a super-intelligent genie locked up in its bottle forever. — Nick Bostrom

There are anthropogenic state risks at the existential level as well: the longer we live in an internationally anarchic system, the greater the cumulative chance of a thermonuclear Armageddon or of a great war fought with other kinds of weapons of mass destruction, laying waste to civilization. — Nick Bostrom

The gap between a dumb and a clever person may appear large from an anthropocentric perspective, yet in a less parochial view the two have nearly indistinguishable minds. — Nick Bostrom

Are you living in a computer simulation? — Nick Bostrom

Once a discovery has been published, there is no way of un-publishing it. — Nick Bostrom

Our approach to existential risks cannot be one of trial-and-error. There is no opportunity to learn from errors. The reactive approach - see what happens, limit damages, and learn from experience - is unworkable. Rather, we must take a proactive approach. This requires foresight to anticipate new types of threats and a willingness to take decisive preventive action and to bear the costs (moral and economic) of such actions. — Nick Bostrom

Discovering traces of life on Mars would be of tremendous scientific significance: The first time that any signs of extraterrestrial life had ever been detected. Many people would also find it heartening to learn that we're not entirely alone in this vast, cold cosmos. — Nick Bostrom

The existence of birds demonstrated that heavier-than-air flight was physically possible and prompted efforts to build flying machines. Yet the first functioning airplanes did not flap their wings. The jury is out on whether machine intelligence will be like flight, which humans achieved through an artificial mechanism, or like combustion, which we initially mastered by copying naturally occurring fires. — Nick Bostrom

Machine intelligence is the last invention that humanity will ever need to make. — Nick Bostrom

When we are headed the wrong way, the last thing we need is progress. — Nick Bostrom

Human nature is a work in progress. — Nick Bostrom

Our demise may instead result from the habitat destruction that ensues when the AI begins massive global construction projects using nanotech factories and assemblers - construction — Nick Bostrom

Knowledge about limitations of your data collection process affects what inferences you can draw from the data. — Nick Bostrom

three conclusions: (1) at least weak forms of superintelligence are achievable by means of biotechnological enhancements; (2) the feasibility of cognitively enhanced humans adds to the plausibility that advanced forms of machine intelligence are feasible - because even if we were fundamentally unable to create machine intelligence (which there is no reason to suppose), machine intelligence might still be within reach of cognitively enhanced humans; and (3) when we consider scenarios stretching significantly into the second half of this century and beyond, we must take into account the probable emergence of a generation of genetically enhanced populations - voters, inventors, scientists - with the magnitude of enhancement escalating rapidly over subsequent decades. — Nick Bostrom

Another lesson is that smart professionals might give an instruction to a program based on a sensible-seeming and normally sound assumption (e.g. that trading volume is a good measure of market liquidity), and that this can produce catastrophic results when the program continues to act on the instruction with iron-clad logical consistency even in the unanticipated situation where the assumption turns out to be invalid. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. This is a theme that we will encounter again. — Nick Bostrom

Any death prior to the heat death of the universe is premature if your life is good. — Nick Bostrom

The bouillon cubes of discrete human-like intellects thus melt into an algorithmic soup. — Nick Bostrom

the orthogonality thesis speaks not of rationality or reason, but of intelligence. — Nick Bostrom

A few hundred thousand years ago, in early human (or hominid) prehistory, growth was so slow that it took on the order of one million years for human productive capacity to increase sufficiently to sustain an additional one million individuals living at subsistence level. By 5000 BC, following the Agricultural Revolution, the rate of growth had increased to the point where the same amount of growth took just two centuries. Today, following the Industrial Revolution, the world economy grows on average by that amount every ninety minutes. — Nick Bostrom

One can speculate that the tardiness and wobbliness of humanity's progress on many of the "eternal problems" of philosophy are due to the unsuitability of the human cortex for philosophical work. On this view, our most celebrated philosophers are like dogs walking on their hind legs - just barely attaining the treshold level of performance required for engaging in the activity at all. — Nick Bostrom

It's unlikely that any of those natural hazards will do us in within the next 100 years if we've already survived 100,000. By contrast, we are introducing, through human activity, entirely new types of dangers by developing powerful new technologies. We have no record of surviving those. — Nick Bostrom

(On one estimate, the adult human brain stores about one billion bits - a couple of orders of magnitude less than a low-end smartphone. — Nick Bostrom

Such an AI might also be able to produce a detailed blueprint for how to bootstrap from existing technology (such as biotechnology and protein engineering) to the constructor capabilities needed for high-throughput atomically precise manufacturing that would allow inexpensive fabrication of a much wider range of nanomechanical structures. — Nick Bostrom

A full-throttled deployment of the practices of strategic communication would kill candor and leave truth bereft to fend for herself in the backstabbing night of political bogeys. — Nick Bostrom

Think of a "discovery" as an act that moves the arrival of information from a later point in time to an earlier time. The discovery's value does not equal the value of the information discovered but rather the value of having the information available earlier than it otherwise would have been. A scientist or a mathematician may show great skill by being the first to find a solution that has eluded many others; yet if the problem would soon have been solved anyway, then the work probably has not much benefited the world [unless having a solution even slightly sooner is immensely valuable or enables further important and urgent work]. — Nick Bostrom

For healthy adult people, the really big thing we can foresee are ways of intervening in the ageing process, either by slowing or reversing it. — Nick Bostrom

The image of evolution as a process that reliably produces benign effects is difficult to reconcile with the enormous suffering that we see in both the human and the natural world. Those who cherish evolution's achievements may do so more from an aesthetic than an ethical perspective. Yet the pertinent question is not what kind of future it would be fascinating to read about in a science fiction novel or to see depicted in a nature documentary, but what kind of future it would be good to live in: two very different matters. — Nick Bostrom

Perhaps we ought not to welcome small catastrophes in case they increase our vigilance to the point of making us prevent the medium-scale catastrophes that would have been needed to make us take the strong precautions necessary to prevent existential catastrophes? (And of course, just as with biological immune systems, we also need to be concerned with over-reactions, analogous to allergies and autoimmune disorders.) — Nick Bostrom

There is more scholarly work on the life-habits of the dung fly than on existential risks [to humanity]. — Nick Bostrom

related type of argument is that we ought - rather callously - to welcome small and medium-scale catastrophes on grounds that they make us aware of our vulnerabilities and spur us into taking precautions that reduce the probability of an existential catastrophe. The idea is that a small or medium-scale catastrophe acts like an inoculation, challenging civilization with a relatively survivable form of a threat and stimulating an immune response that readies the world to deal with the existential variety of the threat.15 — Nick Bostrom

no simple mechanism could do the job as well or better. It might simply be that nobody has yet found the simpler alternative. The Ptolemaic system (with the Earth in the center, orbited by the Sun, the Moon, planets, and stars) represented the state of the art in astronomy for over a thousand years, and its predictive accuracy was improved over the centuries by progressively complicating the model: adding epicycles upon epicycles to the postulated celestial motions. Then the entire system was overthrown by the heliocentric theory of Copernicus, which was simpler and - though only after further elaboration by Kepler - more predictively accurate.63 Artificial intelligence methods are now used in more areas than it would make sense to review here, but mentioning a sampling of them will give an idea of the breadth of applications. Aside from the game AIs — Nick Bostrom

Nanotechnology has been moving a little faster than I expected, virtual reality a little slower. — Nick Bostrom

Some little idiot is bound to press the ignite button just to see what happens. — Nick Bostrom

Choosing the criteria for choosing — Nick Bostrom

The challenge presented by the prospect of superintelligence, and how we might best respond is quite possibly the most important and most daunting challenge humanity has ever faced. And-whether we succeed or fail-it is probably the last challenge we will ever face. — Nick Bostrom

An emulation operating at a speed of ten thousand times that of a biological brain would be able to read a book in a few seconds and write a PhD thesis in an afternoon. — Nick Bostrom

Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization - a niche we filled because we got there first, not because we are in any sense optimally adapted to it. — Nick Bostrom

Traits acquired during one's lifetime - muscles built up in the gym, for example - cannot be passed on to the next generation. Now with technology, as it happens, we might indeed be able to transfer some of our acquired traits on to our selected offspring by genetic engineering. — Nick Bostrom

The cognitive functioning of a human brain depends on a delicate orchestration of many factors, especially during the critical stages of embryo development-and it is much more likely that this self-organizing structure, to be enhanced, needs to be carefully balanced, tuned, and cultivated rather than simply flooded with some extraneous potion. — Nick Bostrom

In reality, it is impossible to build such an agent because it is computationally intractable to perform the requisite calculations. Any attempt to do so succumbs to a combinatorial explosion just like the one described in our discussion of GOFAI. To see why this is so, consider one tiny subset of all possible worlds: those that consist of a single computer monitor floating in an endless vacuum. The monitor has 1, 000 x 1, 000 pixels, each of which is perpetually either on or off. Even this subset of possible worlds is enormously large: the 2(1,000 x 1,000) possible monitor states outnumber all the computations expected ever to take place in the observable universe. Thus, we could not even enumerate all the possible worlds in this tiny subset of all possible worlds, let alone perform more elaborate computations on each of them individually. — Nick Bostrom

Granted, there is still that picture of the Terminator jeering over practically every journalistic attempt to engage with the subject. — Nick Bostrom

Expert chess playing, for example, was once thought to epitomize human intellection. In the view of several experts in the late fifties: "If one could devise a successful chess machine, one would seem to have penetrated to the core of human intellectual endeavor. — Nick Bostrom

I personally don't think of myself as either an optimist or a pessimist. — Nick Bostrom

But what of the seemingly more fanciful idea that the internet might one day "wake up"? Could the internet become something more than just the backbone of a loosely integrated collective superintelligence - something more like a virtual skull housing an emerging unified super-intellect? (This — Nick Bostrom

The first period of excitement, which began with the Dartmouth meeting, was later described by John McCarthy (the event's main organizer) as the "Look, Ma, no hands!" era. — Nick Bostrom

Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~ 2 GHz). — Nick Bostrom

A recent canvass of professional philosophers found the percentage of respondents who "accept or leans toward" various positions. On normative ethics, the results were deontology 25.9%; consequentialism 23.6%; virtue ethics 18.2%. On metaethics, results were moral realism 56.4%; moral anti-realism 27.7%. On moral judgment: cognitivism 65.7%; non-cognitivism 17.0% (Bourget and Chalmers 2009). — Nick Bostrom

We know that blind evolutionary processes can produce human-level general intelligence, since they have already done so at least once. Evolutionary processes with foresight - that is, genetic programs designed and guided by an intelligent human programmer - should be able to achieve a similar outcome with far greater efficiency. — Nick Bostrom

The ground for preferring superintelligence to come before other potentially dangerous technologies, such as nanotechnology, is that superintelligence would reduce the existential risks from nanotechnology but not vice versa.4 Hence, if we create superintelligence first, we will face only those existential risks that are associated with superintelligence; whereas if we create nanotechnology first, we will face the risks of nanotechnology and then, additionally, the risks of superintelligence. — Nick Bostrom