 | |  | |  | |  |
http://www.foresight.org/nanodot/?p=2962
Part III: Singularity from the bottom up
In the previous essay in this series, I argued top-down, from historical and economic
precedents, that the coming singularity might look approximately like the second half of the
computer/internet revolution. Today I’ll argue the same conclusion from the bottom up: by
looking at things from the point of view of the individual AI.
Note: this analysis is taken from a paper I gave at AGI-08.
A major concern in some transhumanist and singularitarian schools of thought is autogenous—
self-modifying and extending—AIs. It is worried that hyper-intelligent machines might
appear literally overnight, as a result of runaway self-improvement by a “seed AI'’.
How likely is runaway self-improvement?
As a baseline, let us consider the self-improving intelligence we understand best, our own.
Humans not only learn new facts and techniques, but improve our learning ability. The
invention of the scientific method, for example, accelerated the uptake of useful
knowledge tremendously. Improvements in knowledge communication and handling, ranging
from the invention of writing and the printing press to the internet and Google, amplify our
analytical and decision-making abilities, including, crucially, the rate at which we (as a
culture) learn.
Individual humans spend much of our lives arduously relearning the corpus of culturally
transmitted knowledge, and then add back a tiny fraction more. Thus on the personal scale
our intelligence does not look “recursively self-improving'’ — but in the large view it
definitely is.
Technological development usually follows an exponential improvement curve.
Examples abound from the power-to-weight ratio of engines, which has tracked an
exponential steadily for 300 years, to the celebrated Moore’s Law curve for
semiconductors, which has done so for 50. These improvement curves fit a simple
reinvestment model, where some constant of proportionality (an “interest rate'’)
determines the overall growth rate.
Any agent must make a decision on how much of its resources to reinvest, and how much to
use for other purposes (including food, clothing, shelter, defense, entertainment, etc.).
Human societies as a whole have invested relatively low percentages gross product, and
even of their surplus, in scientific research.
The proportion of scientists and engineers in the US population can be estimated at 1%, and
those in cognitive-science related fields as 1% of that. Thus we can estimate the current
rate of improvement of AI as being due to the efforts of 30,000 people (with a wide
margin for error, including the fact that there are many cognitive scientists outside the
US!), and estimate the rate of improvement in computer hardware and software generally,
as being due to the efforts of possibly 10 times as many.
It is not clear what a sustainable rate of reinvestment would be for an AI attempting to
improve itself. In the general economy, it would require the same factors of production —
capital, power, space, communication, and so forth — as any other enterprise, and so its
maximum reinvestment rate would be its profit margin. Let us assume for the moment a
rate of 10%, 1000 times the rate of investment by current human society in AI
improvement. (This is germane because the AI is faced with exactly the same choice as an
investor in the general economy: how to allocate its resources for best return.)
Note that from one perspective, an AI running in a lab on equipment it did not have to pay
for could devote 100% of its time to self-improvement; but such cases are limited by the
all-too-restricted resources of AI labs in the first place. Similarly, it seems unlikely that
AIs using stolen resources, e.g. botnets, could manage to devote more than 10% of their
resources to basic research.
Another point to note is that one model for fast self-improvement is the notion that a
hyperintelligence will improve its own hardware. This argument, too, falls to an economic
analysis. If the AI is not a hardware expert, it makes more sense for it to do whatever it
does best, perhaps software improvement, and trade for improved hardware. But this is no
different from any other form of reinvestment, and must come out of the self-
improvement budget. If the AI is a hardware expert, it can make money doing hardware
design for the market, and should do that exclusively, and buy software improvements, for
the overall most optimal upgrade path.
Thus we can assume a 10% reinvestment rate, but we do not know the productivity
coefficient. It is occasionally proposed that, as a creature of software, an AI would be
considerably more proficient at improving its own source code than humans would be.
However, while there is a steady improvement in software science and techniques, these
advances are quickly written into tools and made available to human programmers. In other
words, if automatic programming were really such low-hanging fruit for AI as is assumed, it
would be amenable to narrow-AI techniques and we would have programmer’s assistants
that improved human programmers’ performance drastically. What we see is steady
progress but no huge explosion.
In practice the most difficult part of programming is higher-level conceptual systems
design, not lower level instruction optimization (which is mostly automated now as per the
previous point anyway). Abstract conceptualization has proven to be the hardest part of
human competence to capture in AI. Although occasionally possible, it is quite difficult to
make major improvements in a program when the program itself is the precise problem
specification. Most real-world improvements involve a much more fluid concept of what the
program must do; the improved version does something different but just as good (or
better). So programming in the large requires the full panoply of cognitive capabilities, and
is thus not likely to be enormously out of scale compared to general competence. I think
that many of the more commonly seen scenarios for overnight hard takeoff are circular —
they seem to assume hyperhuman capabilities at the starting point of the self-improvement
process.
We can finesse the productivity, then, by simply letting it be one human equivalent, and
adjusting the timescale to let 0 be whatever point in time a learning, self-improving, human-
level AI is achieved. Then we estimate human productivity at intelligence improvement by
assuming that the human cognitive science community are improving their models at a rate
equivalent to Moore’s Law. As this is the sum effort of 30,000 people, each human’s
productivity coefficient is 0.00002.
This gives us a self-improvement rate for the efforts of a single AI that is essentially flat,
as one would expect: the analysis for a single human would be the same. A single human-level
AI would be much, much better off hiring itself out as an accountant, and buying new
hardware every year with its salary, than by trying to improve itself by its own efforts.
Recursive self-improvement for such an AI would then mean buying new hardware (or
software) every year, improving its prowess at accounting, for an increased growth rate
compounded of its own (tiny) growth and Moore’s Law. Only when it reached a size where it
could match the growth rate of Moore’s Law purely by its on efforts, would it make sense
for it to abandon trade and indulge in self-construction.
But that assumes Moore’s Law, and indeed all other economic parameters, remained
constant over the period. A much more realistic assumption is that, once human-level AI
exists at a price that is less than the net present value of a human of similar capabilities,
the cost of labor will proceed to decline according to Moore’s Law, and therefore the
number of human equivalent minds working in cognitive science and computer hardware will
increase at a Moore’s Law rate, both increasing the rate of progress and decreasing the
price from the current trendline.
In other words, the break-even point for an AI hoping to do all its own development instead
of specializing in a general market and trading for improvements, is a moving target. It will
track the same growth curves that would have allowed the AI to catch up with a fixed
break-even point. (In simple terms: you’re better off buying chips from Intel than trying to
build them yourself. You may improve your chip-building ability — but so will Intel; you’ll
always be better off buying.)
We can conclude that, given some very reasonable assumptions, it will always be more
optimal for an AI to trade; any one which attempts solitary self-improvement will steadily
fall farther and farther behind the technology level of the general marketplace. Note that
this conclusion is very robust to the parameter estimates above: it holds even if the AI’s
reinvestment rate is 100% and the number of researchers required to produce a Moore’s
Law technology improvement rate is 1% of the reasonable estimate.)
Let us now consider a fanciful example in which 30,000 cognitive science researchers,
having created an AI capable of doing their research individually, instantiate 30,000 copies
of it and resign in favor of them. The AIs will be hosted on commercial servers rented by
the salaries of the erstwhile researchers; price per MIPs of such a resource will be
assumed to fall, and thus resources available at a fixed income to rise, with Moore’s Law.
At the starting point, the scientific efforts of the machines would equal those of the human
scientists by assumption. But the effective size of the scientific community would increase
at Moore’s Law rates. On top of that, improvements would come from the fact that further
research in cognitive science would serve to optimize the machines’ own programming. Such
a rate of increase is much harder to quantify, but there have been a few studies that tend
to show a (very) rough parity for Moore’s Law and the rate of software improvement, so let
us use that here. This gives us a total improvement curve of double the Moore’s Law rate.
This is a growth rate that would increase effectiveness from the 30,000 human
equivalents at the start, to approximately 5 billion human equivalents a decade later.
I claim that this growth rate is an upper bound on possible self-improvement rates given
current realities. Note that the assumptions subsume many of the mechanisms that are
often taken in qualitative arguments for hard takeoff: self-improvement is taken account
of; very high effectiveness of software construction by AIs is assumed — 2 years into the
process, each human equivalent of processing power is assumed to be doing 11 times as much
programming as a single human could, for example. Nanotechnology is implied by Moore’s Law
itself not too many years from current date.
This upper bound, a growth rate of approximately 300% per year, is unlikely to be uniformly
achieved. Most technological growth paths are S-curves, exponential at first but levelling
out as diminishing returns effects set in. Maintaining an overall exponential typically
requires paradigm shifts, and those require search and experimentation, as well as breaking
down heretofore efficient social and intellectual structures. In any system, bottleneck
effects will predominate: Moore’s Law has different rates for CPUs, memory, disks,
communications, etc.
The slowest rate of increase will be a limiting factor — the same “Amdahl’s Law of
Singularity” I mentioned before. And finally, we do not really expect the entire cognitive
science field to resign, giving their salaries over to the maintenance of AIs.
--
Brian Atkins
Singularity Institute for Artificial Intelligence http://www.singinst.org/
Next
Part III: Singularity from the bottom up
In the previous essay in this series, I argued top-down, from historical and economic
precedents, that the coming singularity might look approximately like the second half of the
computer/internet revolution. Today I’ll argue the same conclusion from the bottom up: by
looking at things from the point of view of the individual AI.
Note: this analysis is taken from a paper I gave at AGI-08.
A major concern in some transhumanist and singularitarian schools of thought is autogenous—
self-modifying and extending—AIs. It is worried that hyper-intelligent machines might
appear literally overnight, as a result of runaway self-improvement by a “seed AI'’.
How likely is runaway self-improvement?
As a baseline, let us consider the self-improving intelligence we understand best, our own.
Humans not only learn new facts and techniques, but improve our learning ability. The
invention of the scientific method, for example, accelerated the uptake of useful
knowledge tremendously. Improvements in knowledge communication and handling, ranging
from the invention of writing and the printing press to the internet and Google, amplify our
analytical and decision-making abilities, including, crucially, the rate at which we (as a
culture) learn.
Individual humans spend much of our lives arduously relearning the corpus of culturally
transmitted knowledge, and then add back a tiny fraction more. Thus on the personal scale
our intelligence does not look “recursively self-improving'’ — but in the large view it
definitely is.
Technological development usually follows an exponential improvement curve.
Examples abound from the power-to-weight ratio of engines, which has tracked an
exponential steadily for 300 years, to the celebrated Moore’s Law curve for
semiconductors, which has done so for 50. These improvement curves fit a simple
reinvestment model, where some constant of proportionality (an “interest rate'’)
determines the overall growth rate.
Any agent must make a decision on how much of its resources to reinvest, and how much to
use for other purposes (including food, clothing, shelter, defense, entertainment, etc.).
Human societies as a whole have invested relatively low percentages gross product, and
even of their surplus, in scientific research.
The proportion of scientists and engineers in the US population can be estimated at 1%, and
those in cognitive-science related fields as 1% of that. Thus we can estimate the current
rate of improvement of AI as being due to the efforts of 30,000 people (with a wide
margin for error, including the fact that there are many cognitive scientists outside the
US!), and estimate the rate of improvement in computer hardware and software generally,
as being due to the efforts of possibly 10 times as many.
It is not clear what a sustainable rate of reinvestment would be for an AI attempting to
improve itself. In the general economy, it would require the same factors of production —
capital, power, space, communication, and so forth — as any other enterprise, and so its
maximum reinvestment rate would be its profit margin. Let us assume for the moment a
rate of 10%, 1000 times the rate of investment by current human society in AI
improvement. (This is germane because the AI is faced with exactly the same choice as an
investor in the general economy: how to allocate its resources for best return.)
Note that from one perspective, an AI running in a lab on equipment it did not have to pay
for could devote 100% of its time to self-improvement; but such cases are limited by the
all-too-restricted resources of AI labs in the first place. Similarly, it seems unlikely that
AIs using stolen resources, e.g. botnets, could manage to devote more than 10% of their
resources to basic research.
Another point to note is that one model for fast self-improvement is the notion that a
hyperintelligence will improve its own hardware. This argument, too, falls to an economic
analysis. If the AI is not a hardware expert, it makes more sense for it to do whatever it
does best, perhaps software improvement, and trade for improved hardware. But this is no
different from any other form of reinvestment, and must come out of the self-
improvement budget. If the AI is a hardware expert, it can make money doing hardware
design for the market, and should do that exclusively, and buy software improvements, for
the overall most optimal upgrade path.
Thus we can assume a 10% reinvestment rate, but we do not know the productivity
coefficient. It is occasionally proposed that, as a creature of software, an AI would be
considerably more proficient at improving its own source code than humans would be.
However, while there is a steady improvement in software science and techniques, these
advances are quickly written into tools and made available to human programmers. In other
words, if automatic programming were really such low-hanging fruit for AI as is assumed, it
would be amenable to narrow-AI techniques and we would have programmer’s assistants
that improved human programmers’ performance drastically. What we see is steady
progress but no huge explosion.
In practice the most difficult part of programming is higher-level conceptual systems
design, not lower level instruction optimization (which is mostly automated now as per the
previous point anyway). Abstract conceptualization has proven to be the hardest part of
human competence to capture in AI. Although occasionally possible, it is quite difficult to
make major improvements in a program when the program itself is the precise problem
specification. Most real-world improvements involve a much more fluid concept of what the
program must do; the improved version does something different but just as good (or
better). So programming in the large requires the full panoply of cognitive capabilities, and
is thus not likely to be enormously out of scale compared to general competence. I think
that many of the more commonly seen scenarios for overnight hard takeoff are circular —
they seem to assume hyperhuman capabilities at the starting point of the self-improvement
process.
We can finesse the productivity, then, by simply letting it be one human equivalent, and
adjusting the timescale to let 0 be whatever point in time a learning, self-improving, human-
level AI is achieved. Then we estimate human productivity at intelligence improvement by
assuming that the human cognitive science community are improving their models at a rate
equivalent to Moore’s Law. As this is the sum effort of 30,000 people, each human’s
productivity coefficient is 0.00002.
This gives us a self-improvement rate for the efforts of a single AI that is essentially flat,
as one would expect: the analysis for a single human would be the same. A single human-level
AI would be much, much better off hiring itself out as an accountant, and buying new
hardware every year with its salary, than by trying to improve itself by its own efforts.
Recursive self-improvement for such an AI would then mean buying new hardware (or
software) every year, improving its prowess at accounting, for an increased growth rate
compounded of its own (tiny) growth and Moore’s Law. Only when it reached a size where it
could match the growth rate of Moore’s Law purely by its on efforts, would it make sense
for it to abandon trade and indulge in self-construction.
But that assumes Moore’s Law, and indeed all other economic parameters, remained
constant over the period. A much more realistic assumption is that, once human-level AI
exists at a price that is less than the net present value of a human of similar capabilities,
the cost of labor will proceed to decline according to Moore’s Law, and therefore the
number of human equivalent minds working in cognitive science and computer hardware will
increase at a Moore’s Law rate, both increasing the rate of progress and decreasing the
price from the current trendline.
In other words, the break-even point for an AI hoping to do all its own development instead
of specializing in a general market and trading for improvements, is a moving target. It will
track the same growth curves that would have allowed the AI to catch up with a fixed
break-even point. (In simple terms: you’re better off buying chips from Intel than trying to
build them yourself. You may improve your chip-building ability — but so will Intel; you’ll
always be better off buying.)
We can conclude that, given some very reasonable assumptions, it will always be more
optimal for an AI to trade; any one which attempts solitary self-improvement will steadily
fall farther and farther behind the technology level of the general marketplace. Note that
this conclusion is very robust to the parameter estimates above: it holds even if the AI’s
reinvestment rate is 100% and the number of researchers required to produce a Moore’s
Law technology improvement rate is 1% of the reasonable estimate.)
Let us now consider a fanciful example in which 30,000 cognitive science researchers,
having created an AI capable of doing their research individually, instantiate 30,000 copies
of it and resign in favor of them. The AIs will be hosted on commercial servers rented by
the salaries of the erstwhile researchers; price per MIPs of such a resource will be
assumed to fall, and thus resources available at a fixed income to rise, with Moore’s Law.
At the starting point, the scientific efforts of the machines would equal those of the human
scientists by assumption. But the effective size of the scientific community would increase
at Moore’s Law rates. On top of that, improvements would come from the fact that further
research in cognitive science would serve to optimize the machines’ own programming. Such
a rate of increase is much harder to quantify, but there have been a few studies that tend
to show a (very) rough parity for Moore’s Law and the rate of software improvement, so let
us use that here. This gives us a total improvement curve of double the Moore’s Law rate.
This is a growth rate that would increase effectiveness from the 30,000 human
equivalents at the start, to approximately 5 billion human equivalents a decade later.
I claim that this growth rate is an upper bound on possible self-improvement rates given
current realities. Note that the assumptions subsume many of the mechanisms that are
often taken in qualitative arguments for hard takeoff: self-improvement is taken account
of; very high effectiveness of software construction by AIs is assumed — 2 years into the
process, each human equivalent of processing power is assumed to be doing 11 times as much
programming as a single human could, for example. Nanotechnology is implied by Moore’s Law
itself not too many years from current date.
This upper bound, a growth rate of approximately 300% per year, is unlikely to be uniformly
achieved. Most technological growth paths are S-curves, exponential at first but levelling
out as diminishing returns effects set in. Maintaining an overall exponential typically
requires paradigm shifts, and those require search and experimentation, as well as breaking
down heretofore efficient social and intellectual structures. In any system, bottleneck
effects will predominate: Moore’s Law has different rates for CPUs, memory, disks,
communications, etc.
The slowest rate of increase will be a limiting factor — the same “Amdahl’s Law of
Singularity” I mentioned before. And finally, we do not really expect the entire cognitive
science field to resign, giving their salaries over to the maintenance of AIs.
--
Brian Atkins
Singularity Institute for Artificial Intelligence http://www.singinst.org/
Next