Singularity Summit 2007
Blog with notes on some talks:
http://www.acceleratingfuture.com/michael/blog/?p=549
Audio of each talk is at:
http://www.singinst.org/media/singularitysummit2007
Rodney Brooks, CTO of iRobot Corp, former Director (until June 30, 2007) of the MIT Computer
Science and Artificial Intelligence Laboratory The Singularity: A Period Not An Event 9:30AM -
10:30AM
Summary from www.singinst.org
- slight Australian accent, mixed with Boston accent
- I wonder why he left MIT
- opened with a lot of too-frequently-quoted quotes - Yogi Berra etc.
- says that techno-salvation and techno-holocaust are not likely, too extreme
- presented demographic charts about the aging of populations to make point that we need
robots to take care of us in our old age.
- “The Future - Exponentials”
- presented his iRobot stuff, PackBot for Iraq, mentioned how soldiers bond to robot
- FCS Future Combat Systems, 1/3 of military missions unmanned by 2015 is goal of DOD
- “Send in the SUGV”, (SUV guided) military putting a lot of money into robotics as well as AI
- he’s also involved with MIT, Kismet, (Kismet may be made obsolete by Hanson robotics)
- played a bunch of Kismet videos, Cynthia Breazeal
- Kismet is highly uncanny - uncanny valley in action
- it reminds me of how easy it is to anthropomorphize robots, seeing complexity which isn’t there
- Kismet is able to look where people are looking
- robot “Domo” — Aaron Edsinger http://people.csail.mit.edu/edsinger/domo.htm
- showed some interesting videos with robots-eye view of Domo
- safer to interact with physically than automobile-building robots
- aware of external forces
- dark video that was hard to see
- shows us a few optical illusions, says human vision is sophisticated
- recently, we didn’t know about Archaea
- 50% of the world’s biomass is below ground and we didn’t know about it until recently
- maybe AI is already out there
- before we have 100% AI we’ll have 99% AI
- not likely to happen accidentally
- large scale, unexplained oscillations in the Internet may be a sign of AI
- “an annoying alternative” - maybe AI is created and it ignores us as being trivial like we
ignore chipmunks in our yard.
- “another alternative future” - a virus spreads among robots that take care of baby boomers, five
million aging Rolling Stones fans die in one evening - Keith Richards survives
- robotkind sent back 50 years, postponing the Singularity
- another alternative - maybe we’re just not smart enough to create AI
- direct neural implants become elective
- drug enhancement becomes accepted
- genetic enhancement becomes normal
- neural enhancement catches on
- we and our world won’t be us anymore
- may lose distinction between us and them, this could all happen before 2029
- he likes the movie Millenium Man. Notes that movies like iRobot contain old fashioned cars
- believes that humans and robots will become indistinguishable
- question: what was your inspiration for the user interface for the combat robot?
- answer - similar to game controller, 19-year old soldiers can use it without training
- question: will we pay an emotional price as we become symbiotic with robots??
- answer: not just going to be connections w/ machines, also drugs, a wide range, wouldn’t say
that it’s damping down our emotions
- there’s a third-party industry making clothes and skins for Roombas
- “we’ll have Facebook for robots”
- question: are there robots with machine guns in the military?
- answer: not ours (iRobot Corp), but we need to think about that soon
- question: you’ve helped developed robotics for the US government, who has disobeyed the
Geneva conventions. Why?
- answer: other scientists have been funded by the military beginning with DaVinci, scientists
do need to enforce control, Geneva conventions do work, governments can change, people do
change, it’s an ongoing question
Eliezer Yudkowsky, co-founder of the Singularity Institute
Introducing the “Singularity”: Three Major Schools of Thought
10:30AM-11:30AM
Summary from www.singinst.org
- wrote Creating Friendly AI in 2001
- Introducing the “Singularity”: Three Major Schools
- word means a different thing today than when the Singularity Institute got started
- Ray Kurzweil: Accelerating Change, Vernor Vinge: event horizon, I. J. Good: intelligence
explosion
- accelerating change: summarizes the idea, that technology accelerates, may even be a smooth
exponential
- techno-juju increasing exponentially over time
- Threshold of Big Juju will be crossed at a specific time in the future, perhaps on March 4th
2015 at 9:30AM in the morning
- criticisms of some aspects of exponentiality don’t take away from the fact that acceleration is
still happening
- any positive derivative shows that acceleration is happening
- Event Horizon: formulated by Vernor Vinge in the 1970s
- The original “Singularity”
- realized that he couldn’t write stories with people smarter than him
- note that it’s the model of the future that breaks down, not the future itself
- something happens, we just don’t know what it is
- transhuman minds imply a weirder future than flying cars and gadgets
- brains are the source of all dangerous, beautiful and impressive things
- the brain: the trick that does all other tricks at the same time
- intelligence is not book smarts, it’s also persuasiveness, enthusiasm, rationality, musical
talent, thinking on your feet, etc.
- intelligence is the foundation of human power, strength that fuels our other arts
- scale of intelligent minds - village idiot to Einstein? no.
- when he talks about intelligence, he means trans-species - a rock to flatworm to insects to
chimps to humans and beyond
- if you want to see the true shape of the future, look to cognitive technologies, not gadgets with
blinking lights
- bold version of Event Horizon: to predict anything a transhuman mind would do, we’d have to
be at least that smart ourselves
- the Event Horizon argues against the bold version of accelerating change, that smooth
exponentials will allow indefinite predictions
- this is why it’s important to disentangle all these concepts!
- bold accelerationism can be wrong and it’s not an argument against the Event Horizon
- we could reach the threshold following some totally weird trajectory, even one where
technological progress even reverses
- third school: intelligence explosion, goes back to the 1960s, pre-invented by John Campbell
in the 1930s, editor of Astounding Stories
- intelligence explosion: closing the loop of intelligence and technology
- what might enhanced humans use their smartness for? to design the next generation of
intelligence enhancements.
- core thesis of intelligence explosion; minds making technology to improve minds is a positive
feedback cycle, like a pencil falling on its side
- most extreme version of the thesis is an AI improving its own source code
- 18 years cycle for genetic engineering, 18 months for brain-computer interfaces, AI could be
18 seconds!
- comparison between neuron speed and silicon chips
- neuron speed: ~20 spikes/second, 150 meters/second
- 1,000,000-fold speedup physically possible
- just like a skyscraper is OOM taller than a human, and a jet plane OOM faster, there could be
minds that think OOM faster
- all known AIs today are dumber than a village idiot
- shouldn’t use the human scale of intelligence to judge AIs
- shows an animation of the AI creeping along the intelligence scale
- if AI thinks 1000 times faster than human programmers, wouldn’t it improve itself quite faster
than before?
- what does the global economy look like once humans come along? Comparatively, it explodes
overnight. AI could be the same.
- bold claim of Intelligence Explosion: minds making technology to improve minds rapidly
improves to superintelligence
- contradicts both accelerationism and Strong Event horizon
- graph of the intelligence explosion goes almost vertical due to criticality of recursive self-
improvement
- calculating the exact time of the Singularity is a popular pastime among accelerationists
- how long to factor a 75-digit number?
- Geordie Rose calculated that it’s better to use a 2007 algorithm with a 1977 computer than
vice versa, by a factor of 10:3
- to factor that huge number, he’d rather have a modern theory and Apple II than a 70s theory
and Blue Gene
- this shows that algorithms are improving
- more brute force lets you get away with a less clever design
- Moore’s Law of Mad Science: every year the IQ required to destroy the world drops by one point
- summarizes the three core theses
- the three schools are logically distinct but can support or contradict each other’s core or bold
claims
- these claims are often smashed together into Singularity paste
- people coin new meanings of Singularity all the time: if you do so, for the love of cute kittens,
tell us what you mean?
- question: is AI parallelizable?
- answer: I usually say “the true way of AI is as pure as moonlight reflecting off a pool of water”,
computing power might not matter, but AI
- question: how does biology and other wet stuff play into the Singularity?
- answer: I tend to be sort of skeptical, first flying machine was neither an artificial or scaled-up
bird
- it will be easier to make it from scratch than hack the spaghetti code, undocumented human
brain
- question: what do people donating to SIAI expect?
- answer: we’re supposed to build an intelligence explosion grenade, carefully shape it to be
Friendly, and then pull the pin!
- question: is there any other system that recursively self-improves
- answer: closest analogy are humans thinking about how to think, this may be the most
important open problem in AI, no AIs getting mileage out of thinking about thinking
- question: what’s your thought about evolutionary computing?
- answer: a major skeptic, amazing that evolution even works at all, zero intelligence stumbling
around in the dark, requires hundreds of thousands of generations of complex machinery that a
human could create in an afternoon
- our thinking is vastly more efficient, no need to create millions of generations
- question: would you debate John Koza on this?
- answer: yes but we might not totally disagree
- question: can we estimate the threshold for RSI?
- answer: we probably can’t do it, just as we can’t predict math breakthroughs
- question: how much computing power necessary to make an intelligence explosion grenade?
- answer: a billion ops/sec might do it if you were clever enough with another 100 years worth
of AI science
- Networks of Plausible Inference by Judea Pearl (1988), says exactly what goes wrong if you
try to use 1st-order logic to describe reality
Session I: What are the Pathways and Major Challenges?
Barney Pell, Co-founder and CEO of Powerset
Pathways to Advanced General Intelligence: Architecture, Development, and Funding
10:45AM - 12:15PM
Summary from www.singinst.org
While there is broad consensus among the AI community that we will have artificial general
intelligence (AGI) within the century, there is little discussion about the alternate technical
and economic pathways that will bring this about. I present a framework for comparing different
approaches, in which we view any intelligent behavior as a combination of architecture and
development, both of which can be characterized as more or less human-brain-like. Seen
within this framework, one extreme strives for complete brain simulations that develop like
human children. Another extreme strives for unconstrained engineered systems that acquire
knowledge through diverse methods. I predict that the path to AGI will be based on a much
richer interplay between these two extremes, in which top-down and bottom-up approaches
meet in the middle.
The hybrid development path combines the benefits of both technical extremes. It also supports
applications that create incremental business advantage for incremental improvements in AGI
capability, thus driving business competition that accelerates the science. These applications
include video games, virtual worlds, household robots, autonomous vehicles, search, and
conversational interfaces.
- AGI scenario: competing for jobs
- concrete scenario: a humanoid robot that competes with High School graduates for jobs
- often get the job: general, trainable, robust, social, affordable
- he asked a bunch of people at a AI conference, broad consensus this would happen in 100
years
- what are the milestones? people said they had no idea.
- is your work going to be a critical milestone? Answer: probably not.
- little consensus or even debate on what the milestones are
- most of AI has not worked on AGI, it’s too hard, there’s a preference for specialist systems
instead of general ones
- focus on incremental research and low-hanging fruit
- no roadmap that emphasizes generality, and lack of funding is incredibly daunting
- even the DoD wants turnaround in 5 years
- shows a graph with two axes: development and architecture, either similar or different than
human beings
- same same: same as human architecture, same developmental process, like a human baby
- same architecture, different development: neuromorphic AI, thanks to advances in brain
imaging
- different architecture, same development: might be certain types of baby-like robots that grow
like humans but are not neuromorphic at all
- different architecture, different development goes off into totally new and exotic areas, with
their own languages and social structures
- he agrees with Brooks that AI may have already happened, probably not, but it is possible
- may be possible to create AI without understanding how brains actually work, maybe we can
leave that hard task to future scientists or to the AIs themselves
- fundamental risk of top-down is that the theory might not actually work
- he believes that a compromise between top-down and bottom-up may actually create true AI
- virtuous cycle: better AGI -> better products -> more revenue -> more funding -> better AI
- historically, this has not been the case, AI has only been valued as specialized, non-general
systems
- within 10 years we’ll have cars driving us around, will lead to a major transformation in urban
arrangements
- once that happens, there will be more drive towards AI
- there’s an increasing emphasis on general purpose for military robots
- elderly care robotics is another area which could drive financial resources to general AI
- virtual worlds, video games: they’re close to passing the point where the realism is not as
important as the gameplay, the characters, that is, the AIs
- natural language search is a huge revolution in 5-20 years
- introduction of linguistically capable software into everyday activities
- 411 calls are being increasingly handled automatically
- progress made so far is impressive considering how little funding
- next 10 years there will be outstanding increases in funding and focus on generality
- Powerset: building a conversational interface and natural language search engine
- licensed tech from Xerox PARC, it’s been in development for decades
- they knew the conversational interface was a long-term project, although the GUI from PARC
changed the world
- they were given the freedom to keep working on the problem
- in the last five years, it seems the fundamental problems have been solved
- goal is to read every sentence on the web, read out meanings, create a database and use it in
amazing ways
- the idea is a better and transformative search experience
- he believes Powerset is not alone, everyone (Google, Yahoo, etc.) are all going to do this
- these systems are becoming economically central
- their knowledge will increase rapidly: engineering, platform effects, ecosystems, machine
reading
- there may be a milestone where top-down systems like Powerset combined with adaptive,
predictive, associative systems
- conclusions; AGI has received only limited research and computational resources to date
- these conditions are changing quickly
- multiple pathways exist
- a combination of approaches is likely
- the problems are very hard…
- but the Singularity may be nearer
- Q: are these systems going to help us learn?
- A: Yes, and we already have these systems, like Google, which has transformed education
- factual knowledge extracted from Wikipedia and organized automatically is one possible
- Ron Caplan at ACC03 said he predicted dialogue and ambiguity-savvy interfaces in 5 years,
we’re on track to that
- there’s going to be a time when no one thinks huge changes are coming around the corner,
but they are
- eventually people in the future will thank us for thinking about this!
Wendell Wallach
The Road to Singularity: Comedic Complexity, Technological Thresholds, and Bioethical Broad
Jumps
Summary from www.singinst.org
The prospect of implementing higher order cognitive faculties in AI presumes that theories
about the computational nature of mind are valid, that known technological issues can be
solved, that there are no major surprise technological thresholds that will need to be crossed,
and that computer scientists and public officials will find ways to navigate a broad array of
ethical challenges. While some of these concerns have received considerable attention, others
are just beginning to be noted. The ethical challenges, in particular, have not been well
addressed. From robots carrying weapons, to moral decision making faculties for AI, to
institutional review boards for robotic research, and political resistance to some categories of AI
research, the bioethical challenges, if not addressed, could potentially undermine funding and
public support for advanced AI systems. Progress in developing moral decision making faculties
for computers is one area that engineers and designers can begin to tackle, and which will
have a significant impact. The successful development of artificial moral agents (AMAs) is a
major step that will help ameliorate other societal concerns regarding the development of
advanced AI. The pathways for implementing moral decision making faculties in AI include top-
down, bottom-up, and hybrid approaches. In addition, AMAs may require supra-rational
faculties, such as social skills, emotions, consciousness, and a theory of mind.
Notes from lecture:
Two main pathways to AGI:
1. Ongoing incremental AI. Continuing our current path of gradually improving what we have
now. This may take a long time.
2. Dedicated focus approach. Specifically trying to develop AGI.
-Logical
-Mathematical
-Evolutionary
-Theory driven
-Reverse engineering the human brain
-Going directly to adult level ?
-Engineered AGI, choosing a more developed model
-Robotics and embedded systems
Sam S. Adams, IBM Distinguished Engineer, IBM Research Division
Superstition and Forgetfulness – Two Essentials for Artificial General Intelligence
Summary from www.singinst.org
- our experience is highly tied to our temporality
- speed up a record by a couple times, and we can’t even understand it!
- the Joshua Blue Project; develop a computer system, pattern after the human mind, capable of
autonomously learning to successfully function in any number of embodiments and
environments
- stop at age 3, because they can already do all sorts of things most AIs have never been able to
do
- not able to read or write, but able to do all sorts of cool stuff
- pass a “Toddler Turing Test”
- semantic processing, autonomous common sense knowledge acquisition
- quantitative measurement of cognitive machine capabilities
- general purpose “mind”, universal cognitive engine
- self-bootstrapping with minimal a priori knowledge
- graph: architectural clues from human development
- shows neuronal density, eye focus and tracking, neuronal plasticity, synaptic density in
visual cortex, neocortex, other qualities graphed over time
- Y axis: % adult level, X axis: age in years and months
We have the same number of neurons and adulthood, but they drop in half during gestation. As
an engineer, I want to know why.
You have to create a predictive model of your space
- superstition and forgetfulness: the source of all knowledge?
- everything we ever learn starts off as a first time experience, we have no idea whether or not it’
s going to repeat
- “superstition’? yes, but grounded by experience. Symbol grounding problem that has been
present in AI. Two neurons actually fired so it is grounded. Tsunami of going on. About
every four years there is a major reorganization going on in the brain.
1/2 and 10 Petabytes of information hit your visual cortex every day. Can do it on its own
without an homunculus. Run these things in real time. The difference between single cores
and multicores is an issue.
Hi-fidelity system We need a much more flexible model of intelligence
- aggressive forgetfulness is the way to get rid of superstitions that aren’t useful
- evidence that forgetfulness is happening in a continual way in the human brain
- superstition + forgetfulness = general intelligence?
- project has been in full swing since 2002
- needs better hardware to do the algorithms justice
- where’s the pathway to AGI? Follow the child. Go across campus and talk to the psychology
department.
Question: How do you teach Joshua Blue how to forget? If you haven't reached a certain level of
potentiation then get rid of it.
Panel 1
Stephen M. Omohundro
The Nature of Self-Improving Artificial Intelligence
Summary from www.singinst.org
Can we predict the behavior of systems that modify themselves? Can we design them to embody
our values even after many generations of self-improvement? This talk will present a framework
for answering questions like these. It shows that self-improving systems converge on a specific
cognitive architecture that arose out of von Neumann's foundational work on microeconomics.
In these systems there is a universal principle which governs the organization of all levels of
physical and computational resources. They exhibit four natural drives: 1) efficiency, 2) self-
preservation, 3) resource acquisition, and 4) creativity. Unbridled, these lead to both desirable
and undesirable behaviors.
The efficiency drive leads to algorithm optimization, data compression, atomically precise
physical structures, reversible computation, adiabatic physical action, the virtualization of the
physical, and governs a system's choice of memories, theorems, language, and logic. The self-
preservation drive leads to defensive strategies such as "energy encryption" for hiding
resources and promotes replication and game theoretic modelling. The resource acquisition
drive leads to a variety of competitive behaviors and promotes rapid physical expansion and
imperialism. The creativity drive leads to the development of new concepts, algorithms,
theorems, devices, and processes.
The best of these traits could usher in a new era of peace and prosperity; the worst are
characteristic of human psychopaths and could bring widespread destruction. How can we
ensure that this technology acts in alignment with our highest values? We have leverage both
in designing the systems' initial values and in creating the social context within which they
operate. But we must have great clarity in imagining the future we want to create. We need not
just a logical understanding of the technology but a deep introspection into what we cherish
most. With both logic and inspiration we can work toward building a technology that empowers
the human spirit rather than diminishing it.
Avoiding death
Energy encryption
Acquisition
These systems want more stuff. Want to increase their resources.
They want to build a space machine.
Especially if their time horizon is quite long
Especially if their only goal is acquisition
AI may act like an obsessive paranoid psychopath.
Open ended goals
Many of the goals for humans like music, art
Utility function tells us what to do
Belief function is how do we make these decisions given the goals
We have to be very careful as to what we ask for.
They needed a technology which was the constitution
Peter Voss
Increased Intelligence, Improved Life
Summary from www.singinst.org
Artificial General Intelligence (AGI) promises unprecedented advances not only in science and
technology, but also in ethics and social systems. However, business – and thus consumers –
will be first to experience some of the enormous benefits of this emerging technology. This talk
will explore some of these improvements, and try to make a case for how increased intelligence
leads to improved morality.
Panel 2
Neil Jacobstein
Innovative Applications of Early Stage AI
Summary from www.singinst.org
Early stage artificial intelligence has already produced a wide range of valuable but narrowly
focused knowledge systems applications in industry and government. Many of these
applications have performed complex tasks such as planning, monitoring, design, risk
assessment, diagnosis, training, process control, classification, and analysis. For example,
AAAI’s Innovative Applications of AI Conference has published hundreds of successful
applications of AI. The applications are in fields as diverse as biotechnology, space flight,
manufacturing, security, paleontology, construction, energy, music, military, intelligence,
banking, telecommunications, news media, management, law, emergency services,
agriculture, treaty verification, and many other areas. This talk will review the distribution of
these applications across tasks and domains, and discuss the patterns that connect these
applications: what worked, what didn’t, and what are the key trends. None of these systems
exhibited general intelligence, but each documented our ability to codify and distribute
human problem solving knowledge, and put it to work. The answer to the question about how
far are we from advanced AI depends on the operational definition of “advanced”. It is clear from
the knowledge systems produced thus far that even relatively straightforward applications can
be valuable. The larger endeavor to produce AI systems that learn and reason at human levels
and beyond is promising, and will require both enlightened research sponsorship and
appropriate safeguards.
Ben Goertzel
Nine Years To a Positive Singularity – If We Really, Really Try
Summary from www.singinst.org
Common wisdom holds that powerful artificial general intelligence is decades to centuries off.
Even techno-futurist Ray Kurzweil projects a date of 2029 for human-level AI via human brain
emulation. My contention, however, is that powerful and beneficial AGI could come much
sooner – if sufficient attention and resources are devoted to the right approaches. My favored
approach involves integrating probabilistic and evolutionary learning, artificial economics, and
other cutting-edge computer science techniques in a cognitive architecture informed by
cognitive science and systems theory; and then embedding this architecture in virtual agents
that interact with humans and each other in online virtual worlds. Among other advantages, I
argue that this sort of AGI architecture is intrinsically better suited for stably ethical behavior
than more closely human brain based architectures, due to the presence of a coherent and
logical goal system. Current prototype work will be discussed, aimed at actualizing this
approach via the release of intelligent agents controlled by the Novamente AI Engine in Second
Life and other virtual worlds. Of course, it is difficult to place any kind of reliable estimate on
the course of development of this kind of technology, given the R&D that remains to be done,
and the uncertainties regarding funding and other practical exigencies. But radical success
within less than a decade does not seem an outrageously unlikely possibility, in the view of
this AGI researcher and entrepreneur.
You should develop in virtual reality space rather than fiddling with sensors and actuators.
You can get a huge mass of people on Second Life to play with your AI and teach it stuff.
AGI at home
Ladder of increasing intelligence
animals
virtual pets such as parrots
virtual babies
virtual humans
virtual superhumans
vastly transhuman
superhuman god cluster
Paul Saffo
Machines of Loving Grace: Envisioning Advanced AI
Summary from www.singinst.org
A long-anticipated vision of advanced AI is on the verge of arriving late and in utterly
unexpected ways. As we approach this event, a quick look through the rear-view mirror at
earlier AI visions can do much to reduce the uncertainty around how things may unfold.
Moreover, these earlier visions are a powerful reminder that we are not hapless bystanders, but
active participants in what this future should look like. Just as William Gibson once served up
the the vision of cyberspace that shaped the 1990s Internet revolution, a poet writing here in
San Francisco almost exactly 40 years ago penned his vision for what a world of advanced AI
should be.
Panel 3
Day Two
Peter Norvig, Director of Search Quality, Google
The History and Future of Technological Change
9:00AM
Summary from www.singinst.org
The invention of new technology is limited only by the laws of science and by the degree of
ingenuity in the lab. But the proliferation of new technology into everyday life is a complex
social process involving entrepreneurs, venture capitalists, international corporations,
politicians, consumers, and dumb luck. It's hard to predict, especially the future, but by
examining past and present examples of technological change, we can begin to come to grips
with the possibility of abrupt change in the future.
- emphasizes findings that on political issues, educated lay people can be just as effective as
“expert” theorists
- shows that GDP has increased at a steady rate without large jumps due to technology
- life expectancy steadily increasing, no marked acceleration though
- field he made up: AGS: artificial general space exploration
- talks about his “former life” at NASA Ames
- they sent people to a crater in the Arctic to understand what it would like to be an astronaut
- using it to simulate Mars exploration
- another field that’s made up - AGMS - artificial general materials science
- no one is saying “we need one material that can have all these special properties”
- he talked to the Nobelist of RNA interference who said he was extremely excited about the
future of AI - not biotech.
- AGC: artificial general culture
- what makes humans distinct is not our individual intelligence but our collective intelligence
- he believes the Singularity will be a period, not an event
Session IV: What Risks May We Face?
J. Storrs Hall, author of Beyond AI, 10:00 - 10:35AM
Asimov’s Laws of Robotics – Revised
Summary from www.singinst.org
- he used to make tin can robots with bicycle reflectors for eyes, wrote down Asimov’s laws on a
piece of paper and put it into its head
- Asimov’s robots were not self-improving
- law #1: superintelligent AIs should understand memetic evolution
- we need to select evolutionarily stable strategies that are also beneficial
- start AIs out in an evolutionarily stable strategy attractor that took us millions of years to find
- ESC (evolutionary stable conscience)
- an AI with an ESC who “knows” that if it moves off the attractor it will just go back
- will have guidelines to extend the spirit and morality of our ethics into new situations
- law #2: robots should be open source
- we live in a world largely run by artificial information processing structures which have no
conscience: corporations and governments
- law #3: a robot shall be economically sentient
- in other words, at least a part of its utility function should be a consideration of the values
placed on things by others
- this turns out to be an evolutionary stable conscience
- the more transparent the government, the less likely it is to commit atrocities
- law #4: a robot shall be trustworthy, loyal, helpful, friendly, courteous, kind, obedient,
cheerful, thrifty
- summary: he thinks robots should be boy scouts
Peter Thiel, venture capitalist
Financial Markets and the Singularity
Summary from www.singinst.org
- investing in general: do something fundamentally true, and that no one else is doing
- how would markets be different if the Singularity were to happen?
- in a world where there’s a possibility of things going extraordinary good or bad
- the Singularity could either cause a tremendous boom or a world where there’s nothing else
to invest in
- this causes an unusual investment environment
- in a world where everything blows up, there’d be nothing left to buy or sell
- you can’t invest in the bad scenarios of the Singularity
- he thinks the volatility of markets is suggestive of accelerating change
- as we get more information, things should smooth out, but they don’t
- if you were to chart all the booms and busts in history of the modern world, you’d see the
recent amplitudes are the largest
- first hyper-boom took place in Japan in the 1980s
- Japanese stock market in 1989 was worth more than all other markets combined
- people said the entire future of the world would be based in Tokyo
- there was eventually a massive bust there
- another hyper boom: the dot com era in Silicon Valley
- post-mortem: march 2000 - people were completely crazy, he could say a lot of anecdotes that
support that, but…
- what if that “peak of insanity” was just a peak of sanity?
- what if they were right that the old ways of business would all stop working
- what if you had to bet on the one way that would be the way out?
- people thought it was the Internet, but it turned out some companies were totally useless
- bust in 02/03 was extraordinary
- in the last three years, there have been a series of new booms
- you have to choose - which area is going to rapidly accelerate?
- enormous boom in hedge funds, applying computer science to Wall Street
- making them allocate money perfectly, maybe Wall Street, finance will kickstart a Singularity
- Web 2.0 boom is happening, obviously
- macro thesis: even though most of these massive moves may be fake, it can’t the case that they’
re all fake
- as an investor, you have to try to figure out which ones are real
- people may be betting on the Singularity, or parts of it, without even knowing it
- much of Buffett portfolio in the last decade has been about catastrophic insurance products
- different scenarios: you just get premiums, a disaster happens and it benefits the company, a
disaster so big happens that millions die and the fabric of society suffers
- as an investor, it’s important to look at incipient booms
- in the next 20 years, booms and busts will get even more extreme
Charles L. Harper Jr., John Templeton Foundation, 10:45 - 11:30AM
Superintelligence, the Dilemma of Power, and the Transformation of Desire
Summary from www.singinst.org
- remarks he doesn’t know anything about AI
- first question: what does a slug know of Mozart
- we tend to think that superintelligence is just von Neumann, only smarter - continuity
- or there could be radical discontinuities, like phase changes in matter
- to a slug, all of Mozart are primitive perceptions of vibrations
- limitations are based on the slug’s ontology
- slug not well matched to understand Mozart, doesn’t have the ontology
- Gork is speglic (based on orphensic “ontology”)
- he made these words up, making the point that these words mean nothing to us but could
mean a lot to superintelligence
- are we like slugs with respect to some higher complexity level?
- could major discontinuities that we know nothing about? (and can know nothing about?)
- (slugs are not) but are humans at the top?
- we tend to think that we’re on the top
- question #2: how serious is the “dilemma of power”?
- shows image Fritz Haber and his nitrogen fixation apparatus
- this is the fact that science and tech create new forms of power rapidly, but cultures don’t
easily create parallel capacities of stewardship requires to utility newly created powers for
benevolent use and restrain them from being used to serve malevolent ends
- does the dilemma of power matter?
- science/tech production of new powers is rapid and relatively easy
- civilizational production of new capacities of appropraite stewardship of these new expanding
human powers is hard
- suggests a reasonable prospect for future megadisruptive megadisasters
- also implies a vital challenge for innovation to advance our stewardship capabilities
- Christmas Eve, 1938: the idea of fission
- the power was immediately taken over by governments
- another excellent Haber image
- reminds us that Haber invented Zyklon B
- we are creating “mixing problems” - like giving kindergartners loaded AK 47s
- could New York be nuked in the next few decades?
- does the dilemma of power matter a lot? he says probably yes
- what can be done on the stewardship side?
- question #3: how important is the “transformation of desire”
- shows images of young and old Michael Jackson
- demonstrates that the transformation of desire can be problematic
- fasting in all traditions is the direct effect to transform desire for hunger into something
“higher”
- Bhuddist “superintelligence” - Four Noble Truths
- Nirvana is the extinction of desire
- humanist version: Library of Celsus in Ephesus
- Sophia (wisdom), Arete (virtue/excellence), Ennoia (thought/intelligence/intention), Episteme
(knowledge)
- Christian humanism ads agapic love to the conversation
- in addition to IQ, what is it “intelligent” to advance?
- wisdom, empathy, compassion, agapic love, altruistic vision and purpose, courage, fortitude,
etc.
- how does this relate to computational science?
- Martin Nowak’s agenda - develop a formal mathematical understanding of human cooperation
- could the games industry, like SimCity, have some overlap with his deep theoretical work?
- connection between games industry and learning of virtues?
- Olaf Stapledon - Starmaker
- might it be possible to develop win-win synergy between evolutionary logic of virtues and
gaming?
- the success of modernity follows a productivity logic involving the division of labor
- there is some opposition to integration in universities because they are so used to these
divisions
- Bacon, Boyle, Newton, Hooke, etc all pursued integrated visions
- this meeting is fascinating, specifically because people care about
Lunch: Special Presentation by X PRIZE Foundation
- they are playing the standard X-Prize promotion video (this was also showed by Diamandis at
TV07)
- Genomics X-Prize: $10 million for 100 human genomes in 10 days
- Automotive X-Prize: $25 million for 100 mpg and proven market demand for their design
- “Revolution Through Competition”
- target categories that are “stuck”
- looking at educational software for Education X-Prize
- trying to develop educational software that can improve student achievement by as much as 2
sigma
Panel 5
Steve Jurvetson, Draper Fischer Jurvetson, 1:30 - 2:00PM
Dichotomy of Designed and Evolutionary Paths to AI Futures
Summary from www.singinst.org
- asked people whether they think AI will be designed or evolved
- 1/5 said designed, about 2/3 said evolved
- DFJ invests about $5 billion
- he talks very fast, very impressive
- nanotech futures: top-down vs. bottom-up
- progress in nanotech using bio-components
- he finds bottom-up more interesting and more near-term potential
- reminds us that life sciences have just as much interesting accelerating change and activity
as IT
- shows us an exponential graph of genome base pairs sequenced
- Craig Venter’s Socrerer II Expedition (Moore Foundation program)
- there’s a renaissance in genomics and it may relate to AI
- building robust complex systems is the similarity
- evolved information systems
- biology is existence proof of something that can create products more complex than its
antecedents
- emergent layerse of abstraction
- subsystem inscrutability
- no simple shortcuts across the iterations
- design: control, brittle, simple problems, subsystem clarity, artifact learning, modular reuse
and portability
- evolution: out of control, robust resilient, adaptive, complex, transcendent, inscrutability,
hierarchial subsumption
- implications: co-evolutionary islands, path dependence, uploads will definitely require bodies
as well
- mentions the company Genomatica, knocks out a bunch of genes in the metabolic pathway of
microbes to make custom-wanted chemicals
- bio-factories out of e-coli using artificial selection and selective gene repression
- “we will not engineer an AI, rather we will set up the right conditions under which an
intelligence can emerge.”
Christine L. Peterson, Foresight Institute, 1:45 - 2:30PM
Preparing for Bizarreness: Open Source Physical Security
Summary from www.singinst.org
- encourages us to support life extension, SENS
- reminds us we may need to stick around if AGI takes a while
- preparing for risks: we need both info and physical tech to be secure
- three stages of physical risk
- stage 1: explosives, chemical, nuclear
- stage 2: bio
- stage 3: nano
- each one gets harder
- very scary world ahead
- current approaches to risk mitigation are too top-down and short-term
- the top-down way: centralized, mandatory, monolithic, limited in participation, secretive
- the bottom-up way: decentralized, voluntary to extent possible, experimental, collaborative,
open, transparent
- general goals: make voluntary/privatize wherever possible, decentralize remaining gov’t tasks
to lowest levels possible
- how would open source-style security principles apply to these sample situations?
- immediate: airline seucrity
- mid-term: sensing for bioweapons
- long-term: sensing for advanced nanotech weapons
- sousveillance: surveillance from below
- role of sims and games: from Engines of Creation, “We can turn loose a horde of…”
- role of openness, reciprocal transparency
- a lot of what bad guys can do has to do with what they can do in secret before they’re found out
- long-term security goal: automated system able to protect without being threatening
- role of alpha geeks: just like Franklin and Jefferson, when necessary they dealt with politics
superbly well
- it’s like we’re living in a Heinlein novel
- “What — who, me?”
- call for action: many talented people exist here, we have to participate, and start early!
- EoC free at e-drexler.com
- Foresight Open Source Physical Security Project
James Hughes, Institute for Ethics and Emerging Technologies
Waiting for the Great Leap… Forward?, 2:30 - 3:15PM
Summary from www.singinst.org
- lefty transhumanist, wants to build a bridge between us fringe futurists and mainstream
- AGI is likely, “sentient, self-willed, greater-than-human machine minds are very likely in the
coming fifty years”
- AGI Probably Very Dangerous, so steps must be taken to ensure its safety
- he agrees with Singularitarians on those two points
- AGI will be radically alien; if you can’t communicate with a gecko with 60% genetic
similarity, how will we communicate with AI?
- attempt Friendly AI, but it is ultimately futile
- how to ensure that human-derived superintelligences are accountable?
- we could design a perfectly utilitarian computer
- wouldn’t necessarily want a perfect utilitarian AGI
- believes motivations can drift, be edited
- Millennialist Cognitive Biases in Singularitarianism?
- yes, apocalypse and rapture both possible
- but we shouldn’t assume either
- we have some ability to determine outcomes
- “positive assumptions that what comes out of this would be good”
- the AI that comes out of the box will just be like Jesus Christ
- have to say “I’m going to engage in these public policy actions”
- need to have serious engagement, not magical thinking
- emergent A-Life may evolve from primitive designed AI in infosphere ecosystem
- connecting S^ with cyber-security initiatives
- Storm Worm botnet acting in a defensive matter, launches DDoS against researchers
- most cybersecurity ignores AGI
- global tech regulation: techs of mass destruction require transnational regulation
- AGI Police Infrastructure: detection and counter-measures may require machine minds as
well
- with AGIs that have loyalty tied to human institutions somehow?
- human intelligence augmentation is integral to keep up with AI
- perhaps all AGI should be driven by mammal-origin brains
- CogAug and uploads as safe AGI
- toaster caricature
Eliezer Yudkowsky, co-founder of the Singularity Institute
Panel 6
Ray Kurzweil
Summary from www.singinst.org
Trackback URI
21 Responses to “Liveblogging Singularity Summit 2007”
Lincoln Cannon Says:
September 8th, 2007 at 11:25 am
Thanks, Michael.
Mormon Transhumanists: Lincoln Cannon Says:
September 8th, 2007 at 11:36 am
Singularity Summit 2007 Underway with Notes from Michael Anissimov…
The Singularity Summit 2007, AI and the future of humanity, is now underway at the Palace
of……
Anonymous Says:
September 8th, 2007 at 12:20 pm
Dienst am Leser: Singularity Summit II…
Bis jetzt sehe ich 2 Blogs die live vom 2. Singularity Summit aus San Francisco bloggen:
Michael Anissimov
Frontier Channel
Wem die Singularitätshypothese nicht fremd ist, dem dürfte auch die englische Sprache
geläufig sein. Die Restmenge könnte au…
David Orban Says:
September 8th, 2007 at 12:47 pm
Michael, I am sitting at the power strip behind the camera in the right side of the theater. I am
also liveblogging on http://www.davidorban.com. The lunch break is approaching, and it’d be
good to meet.
George Dvorsky Says:
September 8th, 2007 at 7:01 pm
Thanks for this, Michael.
Tom McCabe Says:
September 8th, 2007 at 7:07 pm
It would have been easier, I think, just to webcast the whole thing.
AnneC Says:
September 8th, 2007 at 11:50 pm
Cool, I’m glad I’ll be able to catch the talks I missed on media later on; I’ll be blogging what I
caught today over the course of the next week or so (with the usual verbose commentary).
(Logistical/scheduling stuff made it so I could only be there Saturday until 6-ish, but I got a ton
of material written down today). It was definitely worth going today; I think I have a much better
handle on the relevant subject matter.
Al Fin Says:
September 9th, 2007 at 6:12 am
Nice summaries, Michael.
Interesting how the different approaches to AI appear to diverge with different speakers–some
say “follow the child”, some say that AI will probably be completely unrelated to human
intelligence.
Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.
Jeffrey Herrlich Says:
September 9th, 2007 at 11:24 am
“Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.”
I think it could potentially be very unrelated at the behavioral level. There definitely will be
some design constraints that are common to human minds. For example, formative algorithms,
and constructed through-put (you can’t have a mind without those things). And I suspect that
there will also have to be a macroscopic architecture. Look at the behavioral differences
between different humans - now multiply that potential divergence by about a gazillion.
Jeffrey Herrlich Says:
September 9th, 2007 at 12:27 pm
Michael, could you or someone else at the Summit inform Tyler that the SIAI website has been
intermittently down and won’t load. I know that this is a important period for SIAI’s outreach -
so I’d be good to get the site working properly, ASAP. Maybe it’s just a result of high traffic or
maybe hacking, I don’t know.
Michael Anissimov Says:
September 9th, 2007 at 1:39 pm
Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.
By making up an abstract theory of intelligence and creating a system based on it. Given that
previous successes in AI use systems highly unrelated to human brains, it would be unusual if
future AIs would much more explicitly human-like.
Bob Mottram Says:
September 10th, 2007 at 12:38 am
A lot of this sounds like very hand-wavy stuff, low on detail.
Also, I don’t think that tieing AGIs to human institutions is necessarily a good idea. Human
institutions are the source of many of the problems we see in the world today. Giving these
institutions enormously greater powers is hardly likely to make things better.
If you imagine AGI to be an amplifying device, if naively applied to human affairs as they exist
now the result could be far from friendly.
Jeffrey Herrlich Says:
September 10th, 2007 at 7:43 am
“- attempt Friendly AI, but it is ultimately futile”
I like James, but why do so many very smart and knowledgeable people so readily dismiss the
feasibility of a Friendly AI (eg. Many active AGI researchers and Ray Kurzweil too, at least
previously)? Where there is *no motivation* to be rebellious against its goals, it will not actively
defy its goals. Lots of people anthropomorphize this really, really badly. The only challenge is to
frame the goals in a way that’s *automatically* stable through recursive self-improvement. The
AI doesn’t need to have any conflicting self-interests like we humans are hardwired with. This
can’t be stressed enough.
Bob Mottram Says:
September 10th, 2007 at 10:57 am
An AGI having a provably friendly goal system might be possible, but like many of the other
pundits I’m rather skeptical that this can be achieved in practice. If the AI operated within a
narrow domain and had only limited learning ability it may be possible to prove by rigerous
formal methods that such a system could never exhibit unfriendly behavior X. However, for a
system interacting with the full complexity of the real world and continuosly adapting to it I
think it’s going to be extremely challenging to characterise the behavior of such a system over
extended periods of time.
There is also the more fundamental problem that even very sophisticated formal methods may
not be able to prove some true statements (according to Godel). If one of these true statements
happens to relate to some unfriendly behavior then we’re in trouble.
Jeffrey Herrlich Says:
September 10th, 2007 at 12:21 pm
“An AGI having a provably friendly goal system might be possible, but like many of the other
pundits I’m rather skeptical that this can be achieved in practice. If the AI operated within a
narrow domain and had only limited learning ability it may be possible to prove by rigerous
formal methods that such a system could never exhibit unfriendly behavior X.”
It may or not be possible to demonstrate with 100% provability. But with enough effort we can
steer the likely outcome arbitrarily close to 100%, I beleive. 99.999…% would probably suffice
for most people. I’d flip the switch.
“However, for a system interacting with the full complexity of the real world and continuosly
adapting to it I think it’s going to be extremely challenging to characterise the behavior of such
a system over extended periods of time.”
I suspect that we’ll be fine as long as we can keep the Friendly goal-system stable under self-
improvement. Its self-improvement, I believe, will only lead to more accurate, and intelligent
perception of the meaning of its assigned goals. We need, in effect, a self-strengthening goal-
system, but I think that it’s very achievable.
“If one of these true statements happens to relate to some unfriendly behavior then we’re in
trouble.”
Ultimately, it’s about playing the odds. This would apply to all minds. We already have some
existence proofs that some minds equate to nice people. We just need to make sure with a
reasonable probability that we design a nice baby.
SR Says:
September 10th, 2007 at 2:04 pm
Hmm, I am even more appreciative of Josh Storr Hall’s talk in retrospect than I was “in real
time” (media speak, lol). I like that he talked about the importance of a robot’s consciousness of
its own programming as well as the reasons for the decisions made in the early processes of its
creation. This more interactive (if you will)approach seemed far more logical than treating a
being with a potential to far out-perform humanity as a mere child to be chastised.Result is that
my mild interest in his book has grown to an “eager” pitch.
Charles Harper’s talk was insightful as well. However, more retrospective thoughts tend toward
the fearful: although I admire a certain element of the Buddhist “extinction of desire” concept
and would certainly want an AGI to be sex/jealousy/hunger-free, I certainly don’t wish that for
myself. I don’t think this is necessarily what he meant, actually, but want to take this
opportunity to voice some common, mayhap even semi-luddite-ish fears. While I agree in a
large part with Ray Kurzweil when he talks about culture changing so much that taking meals
together may very well no longer be a way to cement group bonds, nor sex a way to cement
romantic ones,I don’t think this shift in cultural values is going to happen simultaneously with
the emergence of AGI, but will trod along a bit until uploading (which I imagine happening
later - who knows) - and may even then, if the uploadees desire it, continue to exist in a virtual
reality. In my mind these “desires,” while totally unnecessary in terms of something we’d want
an AGI to have or experience, will continue to be fairly valuable in - and to - even those
humans who develop it.
Nick Tarleton Says:
September 10th, 2007 at 8:42 pm
Any idea how soon videos will be online?
Digital Crusader Says:
September 10th, 2007 at 11:32 pm
Singularity Summit 2007: My Thoughts…
The Singularity Summit 2007, I was there both days, although I missed the first two talks on
Sunday Summary from www.singinst.org because of a prior commitment to black-belt outdoor
conditioning :-). Read on for some brief thoughts and notes that I took. Also,……
Michael Anissimov Says:
September 11th, 2007 at 6:55 am
I love how Michael Vassar looks like some hot shot investor with the cell phone and huge smile.
No idea when the videos will be online.
Sarah, I just had a few drinks with Josh Hall here in Tucson. He made up this joke:
What do you call it when Ray Kurzweil gets uploaded into the galactic supercomputer?
Cosmic Ray.
Chris Petersen Says:
September 12th, 2007 at 12:10 am
Yup, Singularitarianism isn’t indie enough anymore … I think I’ll form a splinter group named
‘BAYES LIVES! (and he’s hopping mad)’.
Michael Varian Daly Says:
September 15th, 2007 at 7:41 am
- question: are there robots with machine guns in the military?
- answer:no, but we need to think about that soon
This jumped out at me while I scrolled down. Actually, the correct answer is yes. They are being
used in Iraq to deal with IED’s. You guys need to watch “Weaponology” on The Military
Channel. This stuff is getting out there faster than you realize.
Blog with notes on some talks:
http://www.acceleratingfuture.com/michael/blog/?p=549
Audio of each talk is at:
http://www.singinst.org/media/singularitysummit2007
Rodney Brooks, CTO of iRobot Corp, former Director (until June 30, 2007) of the MIT Computer
Science and Artificial Intelligence Laboratory The Singularity: A Period Not An Event 9:30AM -
10:30AM
Summary from www.singinst.org
- slight Australian accent, mixed with Boston accent
- I wonder why he left MIT
- opened with a lot of too-frequently-quoted quotes - Yogi Berra etc.
- says that techno-salvation and techno-holocaust are not likely, too extreme
- presented demographic charts about the aging of populations to make point that we need
robots to take care of us in our old age.
- “The Future - Exponentials”
- presented his iRobot stuff, PackBot for Iraq, mentioned how soldiers bond to robot
- FCS Future Combat Systems, 1/3 of military missions unmanned by 2015 is goal of DOD
- “Send in the SUGV”, (SUV guided) military putting a lot of money into robotics as well as AI
- he’s also involved with MIT, Kismet, (Kismet may be made obsolete by Hanson robotics)
- played a bunch of Kismet videos, Cynthia Breazeal
- Kismet is highly uncanny - uncanny valley in action
- it reminds me of how easy it is to anthropomorphize robots, seeing complexity which isn’t there
- Kismet is able to look where people are looking
- robot “Domo” — Aaron Edsinger http://people.csail.mit.edu/edsinger/domo.htm
- showed some interesting videos with robots-eye view of Domo
- safer to interact with physically than automobile-building robots
- aware of external forces
- dark video that was hard to see
- shows us a few optical illusions, says human vision is sophisticated
- recently, we didn’t know about Archaea
- 50% of the world’s biomass is below ground and we didn’t know about it until recently
- maybe AI is already out there
- before we have 100% AI we’ll have 99% AI
- not likely to happen accidentally
- large scale, unexplained oscillations in the Internet may be a sign of AI
- “an annoying alternative” - maybe AI is created and it ignores us as being trivial like we
ignore chipmunks in our yard.
- “another alternative future” - a virus spreads among robots that take care of baby boomers, five
million aging Rolling Stones fans die in one evening - Keith Richards survives
- robotkind sent back 50 years, postponing the Singularity
- another alternative - maybe we’re just not smart enough to create AI
- direct neural implants become elective
- drug enhancement becomes accepted
- genetic enhancement becomes normal
- neural enhancement catches on
- we and our world won’t be us anymore
- may lose distinction between us and them, this could all happen before 2029
- he likes the movie Millenium Man. Notes that movies like iRobot contain old fashioned cars
- believes that humans and robots will become indistinguishable
- question: what was your inspiration for the user interface for the combat robot?
- answer - similar to game controller, 19-year old soldiers can use it without training
- question: will we pay an emotional price as we become symbiotic with robots??
- answer: not just going to be connections w/ machines, also drugs, a wide range, wouldn’t say
that it’s damping down our emotions
- there’s a third-party industry making clothes and skins for Roombas
- “we’ll have Facebook for robots”
- question: are there robots with machine guns in the military?
- answer: not ours (iRobot Corp), but we need to think about that soon
- question: you’ve helped developed robotics for the US government, who has disobeyed the
Geneva conventions. Why?
- answer: other scientists have been funded by the military beginning with DaVinci, scientists
do need to enforce control, Geneva conventions do work, governments can change, people do
change, it’s an ongoing question
Eliezer Yudkowsky, co-founder of the Singularity Institute
Introducing the “Singularity”: Three Major Schools of Thought
10:30AM-11:30AM
Summary from www.singinst.org
- wrote Creating Friendly AI in 2001
- Introducing the “Singularity”: Three Major Schools
- word means a different thing today than when the Singularity Institute got started
- Ray Kurzweil: Accelerating Change, Vernor Vinge: event horizon, I. J. Good: intelligence
explosion
- accelerating change: summarizes the idea, that technology accelerates, may even be a smooth
exponential
- techno-juju increasing exponentially over time
- Threshold of Big Juju will be crossed at a specific time in the future, perhaps on March 4th
2015 at 9:30AM in the morning
- criticisms of some aspects of exponentiality don’t take away from the fact that acceleration is
still happening
- any positive derivative shows that acceleration is happening
- Event Horizon: formulated by Vernor Vinge in the 1970s
- The original “Singularity”
- realized that he couldn’t write stories with people smarter than him
- note that it’s the model of the future that breaks down, not the future itself
- something happens, we just don’t know what it is
- transhuman minds imply a weirder future than flying cars and gadgets
- brains are the source of all dangerous, beautiful and impressive things
- the brain: the trick that does all other tricks at the same time
- intelligence is not book smarts, it’s also persuasiveness, enthusiasm, rationality, musical
talent, thinking on your feet, etc.
- intelligence is the foundation of human power, strength that fuels our other arts
- scale of intelligent minds - village idiot to Einstein? no.
- when he talks about intelligence, he means trans-species - a rock to flatworm to insects to
chimps to humans and beyond
- if you want to see the true shape of the future, look to cognitive technologies, not gadgets with
blinking lights
- bold version of Event Horizon: to predict anything a transhuman mind would do, we’d have to
be at least that smart ourselves
- the Event Horizon argues against the bold version of accelerating change, that smooth
exponentials will allow indefinite predictions
- this is why it’s important to disentangle all these concepts!
- bold accelerationism can be wrong and it’s not an argument against the Event Horizon
- we could reach the threshold following some totally weird trajectory, even one where
technological progress even reverses
- third school: intelligence explosion, goes back to the 1960s, pre-invented by John Campbell
in the 1930s, editor of Astounding Stories
- intelligence explosion: closing the loop of intelligence and technology
- what might enhanced humans use their smartness for? to design the next generation of
intelligence enhancements.
- core thesis of intelligence explosion; minds making technology to improve minds is a positive
feedback cycle, like a pencil falling on its side
- most extreme version of the thesis is an AI improving its own source code
- 18 years cycle for genetic engineering, 18 months for brain-computer interfaces, AI could be
18 seconds!
- comparison between neuron speed and silicon chips
- neuron speed: ~20 spikes/second, 150 meters/second
- 1,000,000-fold speedup physically possible
- just like a skyscraper is OOM taller than a human, and a jet plane OOM faster, there could be
minds that think OOM faster
- all known AIs today are dumber than a village idiot
- shouldn’t use the human scale of intelligence to judge AIs
- shows an animation of the AI creeping along the intelligence scale
- if AI thinks 1000 times faster than human programmers, wouldn’t it improve itself quite faster
than before?
- what does the global economy look like once humans come along? Comparatively, it explodes
overnight. AI could be the same.
- bold claim of Intelligence Explosion: minds making technology to improve minds rapidly
improves to superintelligence
- contradicts both accelerationism and Strong Event horizon
- graph of the intelligence explosion goes almost vertical due to criticality of recursive self-
improvement
- calculating the exact time of the Singularity is a popular pastime among accelerationists
- how long to factor a 75-digit number?
- Geordie Rose calculated that it’s better to use a 2007 algorithm with a 1977 computer than
vice versa, by a factor of 10:3
- to factor that huge number, he’d rather have a modern theory and Apple II than a 70s theory
and Blue Gene
- this shows that algorithms are improving
- more brute force lets you get away with a less clever design
- Moore’s Law of Mad Science: every year the IQ required to destroy the world drops by one point
- summarizes the three core theses
- the three schools are logically distinct but can support or contradict each other’s core or bold
claims
- these claims are often smashed together into Singularity paste
- people coin new meanings of Singularity all the time: if you do so, for the love of cute kittens,
tell us what you mean?
- question: is AI parallelizable?
- answer: I usually say “the true way of AI is as pure as moonlight reflecting off a pool of water”,
computing power might not matter, but AI
- question: how does biology and other wet stuff play into the Singularity?
- answer: I tend to be sort of skeptical, first flying machine was neither an artificial or scaled-up
bird
- it will be easier to make it from scratch than hack the spaghetti code, undocumented human
brain
- question: what do people donating to SIAI expect?
- answer: we’re supposed to build an intelligence explosion grenade, carefully shape it to be
Friendly, and then pull the pin!
- question: is there any other system that recursively self-improves
- answer: closest analogy are humans thinking about how to think, this may be the most
important open problem in AI, no AIs getting mileage out of thinking about thinking
- question: what’s your thought about evolutionary computing?
- answer: a major skeptic, amazing that evolution even works at all, zero intelligence stumbling
around in the dark, requires hundreds of thousands of generations of complex machinery that a
human could create in an afternoon
- our thinking is vastly more efficient, no need to create millions of generations
- question: would you debate John Koza on this?
- answer: yes but we might not totally disagree
- question: can we estimate the threshold for RSI?
- answer: we probably can’t do it, just as we can’t predict math breakthroughs
- question: how much computing power necessary to make an intelligence explosion grenade?
- answer: a billion ops/sec might do it if you were clever enough with another 100 years worth
of AI science
- Networks of Plausible Inference by Judea Pearl (1988), says exactly what goes wrong if you
try to use 1st-order logic to describe reality
Session I: What are the Pathways and Major Challenges?
Barney Pell, Co-founder and CEO of Powerset
Pathways to Advanced General Intelligence: Architecture, Development, and Funding
10:45AM - 12:15PM
Summary from www.singinst.org
While there is broad consensus among the AI community that we will have artificial general
intelligence (AGI) within the century, there is little discussion about the alternate technical
and economic pathways that will bring this about. I present a framework for comparing different
approaches, in which we view any intelligent behavior as a combination of architecture and
development, both of which can be characterized as more or less human-brain-like. Seen
within this framework, one extreme strives for complete brain simulations that develop like
human children. Another extreme strives for unconstrained engineered systems that acquire
knowledge through diverse methods. I predict that the path to AGI will be based on a much
richer interplay between these two extremes, in which top-down and bottom-up approaches
meet in the middle.
The hybrid development path combines the benefits of both technical extremes. It also supports
applications that create incremental business advantage for incremental improvements in AGI
capability, thus driving business competition that accelerates the science. These applications
include video games, virtual worlds, household robots, autonomous vehicles, search, and
conversational interfaces.
- AGI scenario: competing for jobs
- concrete scenario: a humanoid robot that competes with High School graduates for jobs
- often get the job: general, trainable, robust, social, affordable
- he asked a bunch of people at a AI conference, broad consensus this would happen in 100
years
- what are the milestones? people said they had no idea.
- is your work going to be a critical milestone? Answer: probably not.
- little consensus or even debate on what the milestones are
- most of AI has not worked on AGI, it’s too hard, there’s a preference for specialist systems
instead of general ones
- focus on incremental research and low-hanging fruit
- no roadmap that emphasizes generality, and lack of funding is incredibly daunting
- even the DoD wants turnaround in 5 years
- shows a graph with two axes: development and architecture, either similar or different than
human beings
- same same: same as human architecture, same developmental process, like a human baby
- same architecture, different development: neuromorphic AI, thanks to advances in brain
imaging
- different architecture, same development: might be certain types of baby-like robots that grow
like humans but are not neuromorphic at all
- different architecture, different development goes off into totally new and exotic areas, with
their own languages and social structures
- he agrees with Brooks that AI may have already happened, probably not, but it is possible
- may be possible to create AI without understanding how brains actually work, maybe we can
leave that hard task to future scientists or to the AIs themselves
- fundamental risk of top-down is that the theory might not actually work
- he believes that a compromise between top-down and bottom-up may actually create true AI
- virtuous cycle: better AGI -> better products -> more revenue -> more funding -> better AI
- historically, this has not been the case, AI has only been valued as specialized, non-general
systems
- within 10 years we’ll have cars driving us around, will lead to a major transformation in urban
arrangements
- once that happens, there will be more drive towards AI
- there’s an increasing emphasis on general purpose for military robots
- elderly care robotics is another area which could drive financial resources to general AI
- virtual worlds, video games: they’re close to passing the point where the realism is not as
important as the gameplay, the characters, that is, the AIs
- natural language search is a huge revolution in 5-20 years
- introduction of linguistically capable software into everyday activities
- 411 calls are being increasingly handled automatically
- progress made so far is impressive considering how little funding
- next 10 years there will be outstanding increases in funding and focus on generality
- Powerset: building a conversational interface and natural language search engine
- licensed tech from Xerox PARC, it’s been in development for decades
- they knew the conversational interface was a long-term project, although the GUI from PARC
changed the world
- they were given the freedom to keep working on the problem
- in the last five years, it seems the fundamental problems have been solved
- goal is to read every sentence on the web, read out meanings, create a database and use it in
amazing ways
- the idea is a better and transformative search experience
- he believes Powerset is not alone, everyone (Google, Yahoo, etc.) are all going to do this
- these systems are becoming economically central
- their knowledge will increase rapidly: engineering, platform effects, ecosystems, machine
reading
- there may be a milestone where top-down systems like Powerset combined with adaptive,
predictive, associative systems
- conclusions; AGI has received only limited research and computational resources to date
- these conditions are changing quickly
- multiple pathways exist
- a combination of approaches is likely
- the problems are very hard…
- but the Singularity may be nearer
- Q: are these systems going to help us learn?
- A: Yes, and we already have these systems, like Google, which has transformed education
- factual knowledge extracted from Wikipedia and organized automatically is one possible
- Ron Caplan at ACC03 said he predicted dialogue and ambiguity-savvy interfaces in 5 years,
we’re on track to that
- there’s going to be a time when no one thinks huge changes are coming around the corner,
but they are
- eventually people in the future will thank us for thinking about this!
Wendell Wallach
The Road to Singularity: Comedic Complexity, Technological Thresholds, and Bioethical Broad
Jumps
Summary from www.singinst.org
The prospect of implementing higher order cognitive faculties in AI presumes that theories
about the computational nature of mind are valid, that known technological issues can be
solved, that there are no major surprise technological thresholds that will need to be crossed,
and that computer scientists and public officials will find ways to navigate a broad array of
ethical challenges. While some of these concerns have received considerable attention, others
are just beginning to be noted. The ethical challenges, in particular, have not been well
addressed. From robots carrying weapons, to moral decision making faculties for AI, to
institutional review boards for robotic research, and political resistance to some categories of AI
research, the bioethical challenges, if not addressed, could potentially undermine funding and
public support for advanced AI systems. Progress in developing moral decision making faculties
for computers is one area that engineers and designers can begin to tackle, and which will
have a significant impact. The successful development of artificial moral agents (AMAs) is a
major step that will help ameliorate other societal concerns regarding the development of
advanced AI. The pathways for implementing moral decision making faculties in AI include top-
down, bottom-up, and hybrid approaches. In addition, AMAs may require supra-rational
faculties, such as social skills, emotions, consciousness, and a theory of mind.
Notes from lecture:
Two main pathways to AGI:
1. Ongoing incremental AI. Continuing our current path of gradually improving what we have
now. This may take a long time.
2. Dedicated focus approach. Specifically trying to develop AGI.
-Logical
-Mathematical
-Evolutionary
-Theory driven
-Reverse engineering the human brain
-Going directly to adult level ?
-Engineered AGI, choosing a more developed model
-Robotics and embedded systems
Sam S. Adams, IBM Distinguished Engineer, IBM Research Division
Superstition and Forgetfulness – Two Essentials for Artificial General Intelligence
Summary from www.singinst.org
- our experience is highly tied to our temporality
- speed up a record by a couple times, and we can’t even understand it!
- the Joshua Blue Project; develop a computer system, pattern after the human mind, capable of
autonomously learning to successfully function in any number of embodiments and
environments
- stop at age 3, because they can already do all sorts of things most AIs have never been able to
do
- not able to read or write, but able to do all sorts of cool stuff
- pass a “Toddler Turing Test”
- semantic processing, autonomous common sense knowledge acquisition
- quantitative measurement of cognitive machine capabilities
- general purpose “mind”, universal cognitive engine
- self-bootstrapping with minimal a priori knowledge
- graph: architectural clues from human development
- shows neuronal density, eye focus and tracking, neuronal plasticity, synaptic density in
visual cortex, neocortex, other qualities graphed over time
- Y axis: % adult level, X axis: age in years and months
We have the same number of neurons and adulthood, but they drop in half during gestation. As
an engineer, I want to know why.
You have to create a predictive model of your space
- superstition and forgetfulness: the source of all knowledge?
- everything we ever learn starts off as a first time experience, we have no idea whether or not it’
s going to repeat
- “superstition’? yes, but grounded by experience. Symbol grounding problem that has been
present in AI. Two neurons actually fired so it is grounded. Tsunami of going on. About
every four years there is a major reorganization going on in the brain.
1/2 and 10 Petabytes of information hit your visual cortex every day. Can do it on its own
without an homunculus. Run these things in real time. The difference between single cores
and multicores is an issue.
Hi-fidelity system We need a much more flexible model of intelligence
- aggressive forgetfulness is the way to get rid of superstitions that aren’t useful
- evidence that forgetfulness is happening in a continual way in the human brain
- superstition + forgetfulness = general intelligence?
- project has been in full swing since 2002
- needs better hardware to do the algorithms justice
- where’s the pathway to AGI? Follow the child. Go across campus and talk to the psychology
department.
Question: How do you teach Joshua Blue how to forget? If you haven't reached a certain level of
potentiation then get rid of it.
Panel 1
Stephen M. Omohundro
The Nature of Self-Improving Artificial Intelligence
Summary from www.singinst.org
Can we predict the behavior of systems that modify themselves? Can we design them to embody
our values even after many generations of self-improvement? This talk will present a framework
for answering questions like these. It shows that self-improving systems converge on a specific
cognitive architecture that arose out of von Neumann's foundational work on microeconomics.
In these systems there is a universal principle which governs the organization of all levels of
physical and computational resources. They exhibit four natural drives: 1) efficiency, 2) self-
preservation, 3) resource acquisition, and 4) creativity. Unbridled, these lead to both desirable
and undesirable behaviors.
The efficiency drive leads to algorithm optimization, data compression, atomically precise
physical structures, reversible computation, adiabatic physical action, the virtualization of the
physical, and governs a system's choice of memories, theorems, language, and logic. The self-
preservation drive leads to defensive strategies such as "energy encryption" for hiding
resources and promotes replication and game theoretic modelling. The resource acquisition
drive leads to a variety of competitive behaviors and promotes rapid physical expansion and
imperialism. The creativity drive leads to the development of new concepts, algorithms,
theorems, devices, and processes.
The best of these traits could usher in a new era of peace and prosperity; the worst are
characteristic of human psychopaths and could bring widespread destruction. How can we
ensure that this technology acts in alignment with our highest values? We have leverage both
in designing the systems' initial values and in creating the social context within which they
operate. But we must have great clarity in imagining the future we want to create. We need not
just a logical understanding of the technology but a deep introspection into what we cherish
most. With both logic and inspiration we can work toward building a technology that empowers
the human spirit rather than diminishing it.
Avoiding death
Energy encryption
Acquisition
These systems want more stuff. Want to increase their resources.
They want to build a space machine.
Especially if their time horizon is quite long
Especially if their only goal is acquisition
AI may act like an obsessive paranoid psychopath.
Open ended goals
Many of the goals for humans like music, art
Utility function tells us what to do
Belief function is how do we make these decisions given the goals
We have to be very careful as to what we ask for.
They needed a technology which was the constitution
Peter Voss
Increased Intelligence, Improved Life
Summary from www.singinst.org
Artificial General Intelligence (AGI) promises unprecedented advances not only in science and
technology, but also in ethics and social systems. However, business – and thus consumers –
will be first to experience some of the enormous benefits of this emerging technology. This talk
will explore some of these improvements, and try to make a case for how increased intelligence
leads to improved morality.
Panel 2
Neil Jacobstein
Innovative Applications of Early Stage AI
Summary from www.singinst.org
Early stage artificial intelligence has already produced a wide range of valuable but narrowly
focused knowledge systems applications in industry and government. Many of these
applications have performed complex tasks such as planning, monitoring, design, risk
assessment, diagnosis, training, process control, classification, and analysis. For example,
AAAI’s Innovative Applications of AI Conference has published hundreds of successful
applications of AI. The applications are in fields as diverse as biotechnology, space flight,
manufacturing, security, paleontology, construction, energy, music, military, intelligence,
banking, telecommunications, news media, management, law, emergency services,
agriculture, treaty verification, and many other areas. This talk will review the distribution of
these applications across tasks and domains, and discuss the patterns that connect these
applications: what worked, what didn’t, and what are the key trends. None of these systems
exhibited general intelligence, but each documented our ability to codify and distribute
human problem solving knowledge, and put it to work. The answer to the question about how
far are we from advanced AI depends on the operational definition of “advanced”. It is clear from
the knowledge systems produced thus far that even relatively straightforward applications can
be valuable. The larger endeavor to produce AI systems that learn and reason at human levels
and beyond is promising, and will require both enlightened research sponsorship and
appropriate safeguards.
Ben Goertzel
Nine Years To a Positive Singularity – If We Really, Really Try
Summary from www.singinst.org
Common wisdom holds that powerful artificial general intelligence is decades to centuries off.
Even techno-futurist Ray Kurzweil projects a date of 2029 for human-level AI via human brain
emulation. My contention, however, is that powerful and beneficial AGI could come much
sooner – if sufficient attention and resources are devoted to the right approaches. My favored
approach involves integrating probabilistic and evolutionary learning, artificial economics, and
other cutting-edge computer science techniques in a cognitive architecture informed by
cognitive science and systems theory; and then embedding this architecture in virtual agents
that interact with humans and each other in online virtual worlds. Among other advantages, I
argue that this sort of AGI architecture is intrinsically better suited for stably ethical behavior
than more closely human brain based architectures, due to the presence of a coherent and
logical goal system. Current prototype work will be discussed, aimed at actualizing this
approach via the release of intelligent agents controlled by the Novamente AI Engine in Second
Life and other virtual worlds. Of course, it is difficult to place any kind of reliable estimate on
the course of development of this kind of technology, given the R&D that remains to be done,
and the uncertainties regarding funding and other practical exigencies. But radical success
within less than a decade does not seem an outrageously unlikely possibility, in the view of
this AGI researcher and entrepreneur.
You should develop in virtual reality space rather than fiddling with sensors and actuators.
You can get a huge mass of people on Second Life to play with your AI and teach it stuff.
AGI at home
Ladder of increasing intelligence
animals
virtual pets such as parrots
virtual babies
virtual humans
virtual superhumans
vastly transhuman
superhuman god cluster
Paul Saffo
Machines of Loving Grace: Envisioning Advanced AI
Summary from www.singinst.org
A long-anticipated vision of advanced AI is on the verge of arriving late and in utterly
unexpected ways. As we approach this event, a quick look through the rear-view mirror at
earlier AI visions can do much to reduce the uncertainty around how things may unfold.
Moreover, these earlier visions are a powerful reminder that we are not hapless bystanders, but
active participants in what this future should look like. Just as William Gibson once served up
the the vision of cyberspace that shaped the 1990s Internet revolution, a poet writing here in
San Francisco almost exactly 40 years ago penned his vision for what a world of advanced AI
should be.
Panel 3
Day Two
Peter Norvig, Director of Search Quality, Google
The History and Future of Technological Change
9:00AM
Summary from www.singinst.org
The invention of new technology is limited only by the laws of science and by the degree of
ingenuity in the lab. But the proliferation of new technology into everyday life is a complex
social process involving entrepreneurs, venture capitalists, international corporations,
politicians, consumers, and dumb luck. It's hard to predict, especially the future, but by
examining past and present examples of technological change, we can begin to come to grips
with the possibility of abrupt change in the future.
- emphasizes findings that on political issues, educated lay people can be just as effective as
“expert” theorists
- shows that GDP has increased at a steady rate without large jumps due to technology
- life expectancy steadily increasing, no marked acceleration though
- field he made up: AGS: artificial general space exploration
- talks about his “former life” at NASA Ames
- they sent people to a crater in the Arctic to understand what it would like to be an astronaut
- using it to simulate Mars exploration
- another field that’s made up - AGMS - artificial general materials science
- no one is saying “we need one material that can have all these special properties”
- he talked to the Nobelist of RNA interference who said he was extremely excited about the
future of AI - not biotech.
- AGC: artificial general culture
- what makes humans distinct is not our individual intelligence but our collective intelligence
- he believes the Singularity will be a period, not an event
Session IV: What Risks May We Face?
J. Storrs Hall, author of Beyond AI, 10:00 - 10:35AM
Asimov’s Laws of Robotics – Revised
Summary from www.singinst.org
- he used to make tin can robots with bicycle reflectors for eyes, wrote down Asimov’s laws on a
piece of paper and put it into its head
- Asimov’s robots were not self-improving
- law #1: superintelligent AIs should understand memetic evolution
- we need to select evolutionarily stable strategies that are also beneficial
- start AIs out in an evolutionarily stable strategy attractor that took us millions of years to find
- ESC (evolutionary stable conscience)
- an AI with an ESC who “knows” that if it moves off the attractor it will just go back
- will have guidelines to extend the spirit and morality of our ethics into new situations
- law #2: robots should be open source
- we live in a world largely run by artificial information processing structures which have no
conscience: corporations and governments
- law #3: a robot shall be economically sentient
- in other words, at least a part of its utility function should be a consideration of the values
placed on things by others
- this turns out to be an evolutionary stable conscience
- the more transparent the government, the less likely it is to commit atrocities
- law #4: a robot shall be trustworthy, loyal, helpful, friendly, courteous, kind, obedient,
cheerful, thrifty
- summary: he thinks robots should be boy scouts
Peter Thiel, venture capitalist
Financial Markets and the Singularity
Summary from www.singinst.org
- investing in general: do something fundamentally true, and that no one else is doing
- how would markets be different if the Singularity were to happen?
- in a world where there’s a possibility of things going extraordinary good or bad
- the Singularity could either cause a tremendous boom or a world where there’s nothing else
to invest in
- this causes an unusual investment environment
- in a world where everything blows up, there’d be nothing left to buy or sell
- you can’t invest in the bad scenarios of the Singularity
- he thinks the volatility of markets is suggestive of accelerating change
- as we get more information, things should smooth out, but they don’t
- if you were to chart all the booms and busts in history of the modern world, you’d see the
recent amplitudes are the largest
- first hyper-boom took place in Japan in the 1980s
- Japanese stock market in 1989 was worth more than all other markets combined
- people said the entire future of the world would be based in Tokyo
- there was eventually a massive bust there
- another hyper boom: the dot com era in Silicon Valley
- post-mortem: march 2000 - people were completely crazy, he could say a lot of anecdotes that
support that, but…
- what if that “peak of insanity” was just a peak of sanity?
- what if they were right that the old ways of business would all stop working
- what if you had to bet on the one way that would be the way out?
- people thought it was the Internet, but it turned out some companies were totally useless
- bust in 02/03 was extraordinary
- in the last three years, there have been a series of new booms
- you have to choose - which area is going to rapidly accelerate?
- enormous boom in hedge funds, applying computer science to Wall Street
- making them allocate money perfectly, maybe Wall Street, finance will kickstart a Singularity
- Web 2.0 boom is happening, obviously
- macro thesis: even though most of these massive moves may be fake, it can’t the case that they’
re all fake
- as an investor, you have to try to figure out which ones are real
- people may be betting on the Singularity, or parts of it, without even knowing it
- much of Buffett portfolio in the last decade has been about catastrophic insurance products
- different scenarios: you just get premiums, a disaster happens and it benefits the company, a
disaster so big happens that millions die and the fabric of society suffers
- as an investor, it’s important to look at incipient booms
- in the next 20 years, booms and busts will get even more extreme
Charles L. Harper Jr., John Templeton Foundation, 10:45 - 11:30AM
Superintelligence, the Dilemma of Power, and the Transformation of Desire
Summary from www.singinst.org
- remarks he doesn’t know anything about AI
- first question: what does a slug know of Mozart
- we tend to think that superintelligence is just von Neumann, only smarter - continuity
- or there could be radical discontinuities, like phase changes in matter
- to a slug, all of Mozart are primitive perceptions of vibrations
- limitations are based on the slug’s ontology
- slug not well matched to understand Mozart, doesn’t have the ontology
- Gork is speglic (based on orphensic “ontology”)
- he made these words up, making the point that these words mean nothing to us but could
mean a lot to superintelligence
- are we like slugs with respect to some higher complexity level?
- could major discontinuities that we know nothing about? (and can know nothing about?)
- (slugs are not) but are humans at the top?
- we tend to think that we’re on the top
- question #2: how serious is the “dilemma of power”?
- shows image Fritz Haber and his nitrogen fixation apparatus
- this is the fact that science and tech create new forms of power rapidly, but cultures don’t
easily create parallel capacities of stewardship requires to utility newly created powers for
benevolent use and restrain them from being used to serve malevolent ends
- does the dilemma of power matter?
- science/tech production of new powers is rapid and relatively easy
- civilizational production of new capacities of appropraite stewardship of these new expanding
human powers is hard
- suggests a reasonable prospect for future megadisruptive megadisasters
- also implies a vital challenge for innovation to advance our stewardship capabilities
- Christmas Eve, 1938: the idea of fission
- the power was immediately taken over by governments
- another excellent Haber image
- reminds us that Haber invented Zyklon B
- we are creating “mixing problems” - like giving kindergartners loaded AK 47s
- could New York be nuked in the next few decades?
- does the dilemma of power matter a lot? he says probably yes
- what can be done on the stewardship side?
- question #3: how important is the “transformation of desire”
- shows images of young and old Michael Jackson
- demonstrates that the transformation of desire can be problematic
- fasting in all traditions is the direct effect to transform desire for hunger into something
“higher”
- Bhuddist “superintelligence” - Four Noble Truths
- Nirvana is the extinction of desire
- humanist version: Library of Celsus in Ephesus
- Sophia (wisdom), Arete (virtue/excellence), Ennoia (thought/intelligence/intention), Episteme
(knowledge)
- Christian humanism ads agapic love to the conversation
- in addition to IQ, what is it “intelligent” to advance?
- wisdom, empathy, compassion, agapic love, altruistic vision and purpose, courage, fortitude,
etc.
- how does this relate to computational science?
- Martin Nowak’s agenda - develop a formal mathematical understanding of human cooperation
- could the games industry, like SimCity, have some overlap with his deep theoretical work?
- connection between games industry and learning of virtues?
- Olaf Stapledon - Starmaker
- might it be possible to develop win-win synergy between evolutionary logic of virtues and
gaming?
- the success of modernity follows a productivity logic involving the division of labor
- there is some opposition to integration in universities because they are so used to these
divisions
- Bacon, Boyle, Newton, Hooke, etc all pursued integrated visions
- this meeting is fascinating, specifically because people care about
Lunch: Special Presentation by X PRIZE Foundation
- they are playing the standard X-Prize promotion video (this was also showed by Diamandis at
TV07)
- Genomics X-Prize: $10 million for 100 human genomes in 10 days
- Automotive X-Prize: $25 million for 100 mpg and proven market demand for their design
- “Revolution Through Competition”
- target categories that are “stuck”
- looking at educational software for Education X-Prize
- trying to develop educational software that can improve student achievement by as much as 2
sigma
Panel 5
Steve Jurvetson, Draper Fischer Jurvetson, 1:30 - 2:00PM
Dichotomy of Designed and Evolutionary Paths to AI Futures
Summary from www.singinst.org
- asked people whether they think AI will be designed or evolved
- 1/5 said designed, about 2/3 said evolved
- DFJ invests about $5 billion
- he talks very fast, very impressive
- nanotech futures: top-down vs. bottom-up
- progress in nanotech using bio-components
- he finds bottom-up more interesting and more near-term potential
- reminds us that life sciences have just as much interesting accelerating change and activity
as IT
- shows us an exponential graph of genome base pairs sequenced
- Craig Venter’s Socrerer II Expedition (Moore Foundation program)
- there’s a renaissance in genomics and it may relate to AI
- building robust complex systems is the similarity
- evolved information systems
- biology is existence proof of something that can create products more complex than its
antecedents
- emergent layerse of abstraction
- subsystem inscrutability
- no simple shortcuts across the iterations
- design: control, brittle, simple problems, subsystem clarity, artifact learning, modular reuse
and portability
- evolution: out of control, robust resilient, adaptive, complex, transcendent, inscrutability,
hierarchial subsumption
- implications: co-evolutionary islands, path dependence, uploads will definitely require bodies
as well
- mentions the company Genomatica, knocks out a bunch of genes in the metabolic pathway of
microbes to make custom-wanted chemicals
- bio-factories out of e-coli using artificial selection and selective gene repression
- “we will not engineer an AI, rather we will set up the right conditions under which an
intelligence can emerge.”
Christine L. Peterson, Foresight Institute, 1:45 - 2:30PM
Preparing for Bizarreness: Open Source Physical Security
Summary from www.singinst.org
- encourages us to support life extension, SENS
- reminds us we may need to stick around if AGI takes a while
- preparing for risks: we need both info and physical tech to be secure
- three stages of physical risk
- stage 1: explosives, chemical, nuclear
- stage 2: bio
- stage 3: nano
- each one gets harder
- very scary world ahead
- current approaches to risk mitigation are too top-down and short-term
- the top-down way: centralized, mandatory, monolithic, limited in participation, secretive
- the bottom-up way: decentralized, voluntary to extent possible, experimental, collaborative,
open, transparent
- general goals: make voluntary/privatize wherever possible, decentralize remaining gov’t tasks
to lowest levels possible
- how would open source-style security principles apply to these sample situations?
- immediate: airline seucrity
- mid-term: sensing for bioweapons
- long-term: sensing for advanced nanotech weapons
- sousveillance: surveillance from below
- role of sims and games: from Engines of Creation, “We can turn loose a horde of…”
- role of openness, reciprocal transparency
- a lot of what bad guys can do has to do with what they can do in secret before they’re found out
- long-term security goal: automated system able to protect without being threatening
- role of alpha geeks: just like Franklin and Jefferson, when necessary they dealt with politics
superbly well
- it’s like we’re living in a Heinlein novel
- “What — who, me?”
- call for action: many talented people exist here, we have to participate, and start early!
- EoC free at e-drexler.com
- Foresight Open Source Physical Security Project
James Hughes, Institute for Ethics and Emerging Technologies
Waiting for the Great Leap… Forward?, 2:30 - 3:15PM
Summary from www.singinst.org
- lefty transhumanist, wants to build a bridge between us fringe futurists and mainstream
- AGI is likely, “sentient, self-willed, greater-than-human machine minds are very likely in the
coming fifty years”
- AGI Probably Very Dangerous, so steps must be taken to ensure its safety
- he agrees with Singularitarians on those two points
- AGI will be radically alien; if you can’t communicate with a gecko with 60% genetic
similarity, how will we communicate with AI?
- attempt Friendly AI, but it is ultimately futile
- how to ensure that human-derived superintelligences are accountable?
- we could design a perfectly utilitarian computer
- wouldn’t necessarily want a perfect utilitarian AGI
- believes motivations can drift, be edited
- Millennialist Cognitive Biases in Singularitarianism?
- yes, apocalypse and rapture both possible
- but we shouldn’t assume either
- we have some ability to determine outcomes
- “positive assumptions that what comes out of this would be good”
- the AI that comes out of the box will just be like Jesus Christ
- have to say “I’m going to engage in these public policy actions”
- need to have serious engagement, not magical thinking
- emergent A-Life may evolve from primitive designed AI in infosphere ecosystem
- connecting S^ with cyber-security initiatives
- Storm Worm botnet acting in a defensive matter, launches DDoS against researchers
- most cybersecurity ignores AGI
- global tech regulation: techs of mass destruction require transnational regulation
- AGI Police Infrastructure: detection and counter-measures may require machine minds as
well
- with AGIs that have loyalty tied to human institutions somehow?
- human intelligence augmentation is integral to keep up with AI
- perhaps all AGI should be driven by mammal-origin brains
- CogAug and uploads as safe AGI
- toaster caricature
Eliezer Yudkowsky, co-founder of the Singularity Institute
Panel 6
Ray Kurzweil
Summary from www.singinst.org
Trackback URI
21 Responses to “Liveblogging Singularity Summit 2007”
Lincoln Cannon Says:
September 8th, 2007 at 11:25 am
Thanks, Michael.
Mormon Transhumanists: Lincoln Cannon Says:
September 8th, 2007 at 11:36 am
Singularity Summit 2007 Underway with Notes from Michael Anissimov…
The Singularity Summit 2007, AI and the future of humanity, is now underway at the Palace
of……
Anonymous Says:
September 8th, 2007 at 12:20 pm
Dienst am Leser: Singularity Summit II…
Bis jetzt sehe ich 2 Blogs die live vom 2. Singularity Summit aus San Francisco bloggen:
Michael Anissimov
Frontier Channel
Wem die Singularitätshypothese nicht fremd ist, dem dürfte auch die englische Sprache
geläufig sein. Die Restmenge könnte au…
David Orban Says:
September 8th, 2007 at 12:47 pm
Michael, I am sitting at the power strip behind the camera in the right side of the theater. I am
also liveblogging on http://www.davidorban.com. The lunch break is approaching, and it’d be
good to meet.
George Dvorsky Says:
September 8th, 2007 at 7:01 pm
Thanks for this, Michael.
Tom McCabe Says:
September 8th, 2007 at 7:07 pm
It would have been easier, I think, just to webcast the whole thing.
AnneC Says:
September 8th, 2007 at 11:50 pm
Cool, I’m glad I’ll be able to catch the talks I missed on media later on; I’ll be blogging what I
caught today over the course of the next week or so (with the usual verbose commentary).
(Logistical/scheduling stuff made it so I could only be there Saturday until 6-ish, but I got a ton
of material written down today). It was definitely worth going today; I think I have a much better
handle on the relevant subject matter.
Al Fin Says:
September 9th, 2007 at 6:12 am
Nice summaries, Michael.
Interesting how the different approaches to AI appear to diverge with different speakers–some
say “follow the child”, some say that AI will probably be completely unrelated to human
intelligence.
Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.
Jeffrey Herrlich Says:
September 9th, 2007 at 11:24 am
“Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.”
I think it could potentially be very unrelated at the behavioral level. There definitely will be
some design constraints that are common to human minds. For example, formative algorithms,
and constructed through-put (you can’t have a mind without those things). And I suspect that
there will also have to be a macroscopic architecture. Look at the behavioral differences
between different humans - now multiply that potential divergence by about a gazillion.
Jeffrey Herrlich Says:
September 9th, 2007 at 12:27 pm
Michael, could you or someone else at the Summit inform Tyler that the SIAI website has been
intermittently down and won’t load. I know that this is a important period for SIAI’s outreach -
so I’d be good to get the site working properly, ASAP. Maybe it’s just a result of high traffic or
maybe hacking, I don’t know.
Michael Anissimov Says:
September 9th, 2007 at 1:39 pm
Given that humans are the intelligence devising and evaluating the AI, I wonder how an AI
unrelated to human intelligence would even be possible.
By making up an abstract theory of intelligence and creating a system based on it. Given that
previous successes in AI use systems highly unrelated to human brains, it would be unusual if
future AIs would much more explicitly human-like.
Bob Mottram Says:
September 10th, 2007 at 12:38 am
A lot of this sounds like very hand-wavy stuff, low on detail.
Also, I don’t think that tieing AGIs to human institutions is necessarily a good idea. Human
institutions are the source of many of the problems we see in the world today. Giving these
institutions enormously greater powers is hardly likely to make things better.
If you imagine AGI to be an amplifying device, if naively applied to human affairs as they exist
now the result could be far from friendly.
Jeffrey Herrlich Says:
September 10th, 2007 at 7:43 am
“- attempt Friendly AI, but it is ultimately futile”
I like James, but why do so many very smart and knowledgeable people so readily dismiss the
feasibility of a Friendly AI (eg. Many active AGI researchers and Ray Kurzweil too, at least
previously)? Where there is *no motivation* to be rebellious against its goals, it will not actively
defy its goals. Lots of people anthropomorphize this really, really badly. The only challenge is to
frame the goals in a way that’s *automatically* stable through recursive self-improvement. The
AI doesn’t need to have any conflicting self-interests like we humans are hardwired with. This
can’t be stressed enough.
Bob Mottram Says:
September 10th, 2007 at 10:57 am
An AGI having a provably friendly goal system might be possible, but like many of the other
pundits I’m rather skeptical that this can be achieved in practice. If the AI operated within a
narrow domain and had only limited learning ability it may be possible to prove by rigerous
formal methods that such a system could never exhibit unfriendly behavior X. However, for a
system interacting with the full complexity of the real world and continuosly adapting to it I
think it’s going to be extremely challenging to characterise the behavior of such a system over
extended periods of time.
There is also the more fundamental problem that even very sophisticated formal methods may
not be able to prove some true statements (according to Godel). If one of these true statements
happens to relate to some unfriendly behavior then we’re in trouble.
Jeffrey Herrlich Says:
September 10th, 2007 at 12:21 pm
“An AGI having a provably friendly goal system might be possible, but like many of the other
pundits I’m rather skeptical that this can be achieved in practice. If the AI operated within a
narrow domain and had only limited learning ability it may be possible to prove by rigerous
formal methods that such a system could never exhibit unfriendly behavior X.”
It may or not be possible to demonstrate with 100% provability. But with enough effort we can
steer the likely outcome arbitrarily close to 100%, I beleive. 99.999…% would probably suffice
for most people. I’d flip the switch.
“However, for a system interacting with the full complexity of the real world and continuosly
adapting to it I think it’s going to be extremely challenging to characterise the behavior of such
a system over extended periods of time.”
I suspect that we’ll be fine as long as we can keep the Friendly goal-system stable under self-
improvement. Its self-improvement, I believe, will only lead to more accurate, and intelligent
perception of the meaning of its assigned goals. We need, in effect, a self-strengthening goal-
system, but I think that it’s very achievable.
“If one of these true statements happens to relate to some unfriendly behavior then we’re in
trouble.”
Ultimately, it’s about playing the odds. This would apply to all minds. We already have some
existence proofs that some minds equate to nice people. We just need to make sure with a
reasonable probability that we design a nice baby.
SR Says:
September 10th, 2007 at 2:04 pm
Hmm, I am even more appreciative of Josh Storr Hall’s talk in retrospect than I was “in real
time” (media speak, lol). I like that he talked about the importance of a robot’s consciousness of
its own programming as well as the reasons for the decisions made in the early processes of its
creation. This more interactive (if you will)approach seemed far more logical than treating a
being with a potential to far out-perform humanity as a mere child to be chastised.Result is that
my mild interest in his book has grown to an “eager” pitch.
Charles Harper’s talk was insightful as well. However, more retrospective thoughts tend toward
the fearful: although I admire a certain element of the Buddhist “extinction of desire” concept
and would certainly want an AGI to be sex/jealousy/hunger-free, I certainly don’t wish that for
myself. I don’t think this is necessarily what he meant, actually, but want to take this
opportunity to voice some common, mayhap even semi-luddite-ish fears. While I agree in a
large part with Ray Kurzweil when he talks about culture changing so much that taking meals
together may very well no longer be a way to cement group bonds, nor sex a way to cement
romantic ones,I don’t think this shift in cultural values is going to happen simultaneously with
the emergence of AGI, but will trod along a bit until uploading (which I imagine happening
later - who knows) - and may even then, if the uploadees desire it, continue to exist in a virtual
reality. In my mind these “desires,” while totally unnecessary in terms of something we’d want
an AGI to have or experience, will continue to be fairly valuable in - and to - even those
humans who develop it.
Nick Tarleton Says:
September 10th, 2007 at 8:42 pm
Any idea how soon videos will be online?
Digital Crusader Says:
September 10th, 2007 at 11:32 pm
Singularity Summit 2007: My Thoughts…
The Singularity Summit 2007, I was there both days, although I missed the first two talks on
Sunday Summary from www.singinst.org because of a prior commitment to black-belt outdoor
conditioning :-). Read on for some brief thoughts and notes that I took. Also,……
Michael Anissimov Says:
September 11th, 2007 at 6:55 am
I love how Michael Vassar looks like some hot shot investor with the cell phone and huge smile.
No idea when the videos will be online.
Sarah, I just had a few drinks with Josh Hall here in Tucson. He made up this joke:
What do you call it when Ray Kurzweil gets uploaded into the galactic supercomputer?
Cosmic Ray.
Chris Petersen Says:
September 12th, 2007 at 12:10 am
Yup, Singularitarianism isn’t indie enough anymore … I think I’ll form a splinter group named
‘BAYES LIVES! (and he’s hopping mad)’.
Michael Varian Daly Says:
September 15th, 2007 at 7:41 am
- question: are there robots with machine guns in the military?
- answer:no, but we need to think about that soon
This jumped out at me while I scrolled down. Actually, the correct answer is yes. They are being
used in Iraq to deal with IED’s. You guys need to watch “Weaponology” on The Military
Channel. This stuff is getting out there faster than you realize.
 | |  | |  | |  |