29 Comments
User's avatar
Andy G's avatar

“If, on the other hand, it is sensitive to the fifth, sixth, twentieth, if any change however small in the initial conditions can cause a large change in the final state, it is hard to see how any AI however superintelligent could use the effect.”

It’s not just the sensitivity, but knowing what the sensitivity is for any given thing.

I have little problem believing a sufficiently advanced AI could come up with the answer *if* it had all the data. But in a human world, just as with the weather, why is there any reason to believe it could have access to all the necessary data?

Expand full comment
DavesNotHere's avatar

That is an important problem, but a separate one. If the model is too sensitive to small changes in inputs, it is hard to use it even if you have ideal data (which of course you won't).

Assume, unrealistically, that you can get all the data you want, at whatever resolution and accuracy. If you plug it into a linear model, then changing resolution will make changes in the resulting predictions that are proportional. They will be different, and the difference will increase over time, but at a constant rate. Similar inputs will generate similar results.

If the model is nonlinear, the difference is no longer proportional, and similar inputs can generate different enough outputs that the model can become useless.

Expand full comment
Philalethes's avatar

I tend to agree with the comment that perfect knowledge of the initial state is necessary to make perfect prediction. I do not see how AI could get the initial data.

Expand full comment
Frog H Emoth's avatar

Behold! The AI cannot have all of the data, because the AI itself generates heat, which changes the data. In order to predict its own heat generation, its model of the universe would have to include a model of itself. That model, in turn, in order to be a faithful model, would have to itself simulate the universe, including the AI. That model AI would, in order to be a faithful model, would itself have to have a model of the AI, etc, creating an infinite recursion.

Expand full comment
omar's avatar

Interesting point on infinite recursion. It reminds me of the story where mathematician John von Neumann was asked about the distance a bee travelling 200mph travels when it zigzags between two trains 200 miles apart, each speeding towards the other at 100mph until the trains collides. Von Neumann instantly answered that the bee traveled exactly 200 miles. Asked if he'd taken the shortcut (one hour at 200 mph), he smiled and replied, "No, I summed the infinite series of legs the bee traveled."

Though likely apocryphal, this story illustrates how infinite processes can yield precise solutions. Similarly, just as von Neumann's bee travels a finite distance despite infinite steps, an AI’s recursive self-modeling might produce a useable discrete model, though modelling the entire universe is likely impossible because of computational complexity and fundamental uncertainties.

Expand full comment
David Friedman's avatar

In the version of the Von Neumann story I am familiar with, he gives the answer, when the other man is surprised that he summed the series, he asks "Is there another way?"

Expand full comment
THulsey's avatar

David, here I shoehorn in my SOLUTION TO AI CONSCIOUSNESS. (Rebuke me if you like.)

The current deadlock assumes that we need to define consciousness, then apply it to AI. Most begin with a Cartesian view of consciousness as the "ghost in the machine" -- an alien indeterminately willful "something" grafted onto the chemically deterministic body. THIS APPROACH CANNOT SUCCEED.

Consciousness begins in moral judgments, all rooted in the sense of touch. A baby learns that a spoon of food tastes good; but that the same spoon of food, if touched while steaming, might be so hot as to burn the tongue; he learns that if he feels the sensation of cold, a noise from him can bring the warmth of a parent; he learns that making a noise in a certain way delights the face of those parents. All of these judgments from him are moral judgments, since they are evaluations whose correct response to some stimulus based in touch produces pleasure, the rudimentary signpost toward life. All of these millions of moral judgments form part of a single emotional sensus communis (SC). This is the emotional “felt sense” of consciousness. The conceptual SC is built from the emotional one. Notice that people learn something when they feel it's important to them personally. Even when doing something seemingly pointless, like solving a math puzzle, strengthens a mental tool that ultimately is felt to advance life. Simplified diagram:

LIFE -> TOUCH -> MORAL JUDGMENTS -> AGGREGATE TO EMOTIONAL SC -> AGGREGATE TO CONCEPTUAL SC. Consciousness IS this aggregated SC.

HERE IS YOUR SOLUTION FOR AI: Build a database not of "factoids" (since we have that already), but of these millions of aggregated moral judgments, and call it a CASUISTIC DATABASE, which is the ersatz emotional SC that the human has in reality. THIS is the ersatz patch that the AI robot consults to fake its "consciousness."

Further, the casuistic database of someone raised in the Japanese culture might have recorded a response to a moral event with deference; while the casuistic database of someone from a Western culture might have recorded a more aggressive response to that very same moral event. Thus for many cultures, many casuistic databases.

Expand full comment
omar's avatar

As a poker player, albeit a rather poor one, I have contemplated how tiny, random events, like a bright light reflecting off someone’s watch, drawing a dealer's attention for a moment while they are cutting a deck, can change the entire sequence of cards dealt.  This would change every hand dealt from that point forward, forever.  Yet, despite changing every player's future hands, some patterns seem pretty consistent:  skilled players usually win, and weaker players tend to lose.

You argue that extreme sensitivity to minor conditions could stop even a superintelligent AI from reliably predicting outcomes.  But this overlooks an important point: just because things are chaotic in the details doesn’t mean they're chaotic in the bigger picture.  Isaac Asimov discussed something similar with "psychohistory" in his Foundation series, where he described how individual human behavior might be unpredictable, but groups of people tend to behave in  more predictable ways.

A superintelligent AI, though it may not be able to predict small events with extreme precision, would probably excel at spotting larger patterns hidden within the chaos.  It seems not unlikely that it could foresee and even shape big-picture outcomes, like economic or political trends, in ways that humans simply can't.

The critical question is not whether a superintelligent AI can perfectly predict every event at a micro or even a macro level, but whether its predictive abilities are so superior to humans that humans become effectively powerless in comparison.

Expand full comment
David Friedman's avatar

I suspect that Asimov's psychohistory was modeled on Marx, who claimed to be doing that sort of thing — volume 1 of Capital felt to me very much like Stapledon's sf.

My version is economics. My first economics journal article was a theory of the size and shape of states. If it is right, it would provide one tool to predict the future. Theories of economic growth are another.

But there is still a difference between predicting and controlling, and I was speculating about how a sufficiently intelligent being could control the future. The point about insufficiently detailed knowledge of the present is probably right. The kind of ability you describe would let such a being influence the future but not, I think, by very much.

Expand full comment
omar's avatar

You talked about the difference between predicting and controlling, noting how limited knowledge of the present would probably limit the ability to control the future. While I agree that detailed, deterministic predictions and control are unlikely, I believe that even minor predictive advantages can result in a significant cumulative impact.

You cited economics, which, even though inherently imperfect, can, at a macro-level, predict, or at least provide insights about, the future. Investment markets (& poker) clearly demonstrate how such small predictive insights can compound. A slight 1% monthly advantage in the stock market would generate outsized gains over time. Similarly, a super intelligent AI with even just a small stochastic advantage in foresight in financial, political, or technological domains seems likely to be able to influence long-term outcomes, more than the phrase "not very much" might suggest.

Expand full comment
Paul Brinkley's avatar

"The world is a complicated places." ...Indeed!

Expand full comment
Max More's avatar

I have always found the Butterfly Effect to be highly questionable. It seems to me that some mechanism is needed to amplify a tiny effect into a large effect. I see no such mechanism for the flapping of wings to have a major effect on the weather (or on anything else). Instead, background conditions typically swamp the small effect, diluting and dissolving it. Of course, you can construct fictional examples and some may be possible although almost certainly the outcome is not predictable beforehand.

Expand full comment
DavesNotHere's avatar

The butterfly effect is not an empirical observation about the weather, but an observation about nonlinear math. Tiny changes in the input data end up creating large differences in the result. Your objection amounts to saying that it is implausible that weather could be modeled better by a nonlinear system than a linear one. But even if the nonlinear model is better, that isn’t a claim that butterflies actually cause hurricanes, but that the resolution and accuracy of the inputs can put a severe limitation on a model's usefulness.

Expand full comment
Max More's avatar

If the effect is specified as about nonlinear math, then fine. But that's not how I usually see it mentioned. In fact, the original two examples (a bird's wings and a butterfly's wing's) suggest something different. The statement "the resolution and accuracy of the inputs can put a severe limitation on a model's usefulness" seems far more sensible and defensible than the way the Butterfly Effect is usually put (in my experience, which may differ from yours).

Expand full comment
DavesNotHere's avatar

The connection is, the difference in inputs can be as small as the effect of a butterfly flapping or not flapping its wings. Maybe that is a difference in the 20th decimal point or something ridiculously small. If the model is iterated enough times, the results will differ considerably depending on the tiny initial difference. A linear model acts as we would expect: the results may not be good, but the difference between the results using one set of inputs and another with a tiny difference will always be fairly small. Not so with a nonlinear model.

Expand full comment
Paul Brinkley's avatar

Some background conditions will dampen an arbitrarily small change. Other conditions will amplify it. It may help to think of conditions as mathematical functions, which take one number or vector or matrix or other structure as input, and output an identical structure, which may be fed back in as input on the next iteration, and so on.

An example: the function f(theta) = (theta+2) mod 360 is not chaotic. If you start with theta=1 and set it running (3, 5, 7, ...) and start a different series with theta=1.01 (3.01, 5.01, 7.01, ...), the difference between the two at any iteration will be the same as when it started.

By contrast, the function f(theta) = (theta*2) mod 360 is chaotic. Any change, however small, will eventually get up to 360 (and then drop down but rise again). Same for (theta*1.5) mod 360, or even (theta*1.00000001) mod 360.

Notice that the "mod 360" limits the possible output to the range 0-360 (for any input in that range); nevertheless, any difference between two series will eventually grow to the size of the range itself for the latter examples.

Systems with functions like (theta+2) mod 360 or (theta*0.5) mod 360 are often called homeostatic - changes shrink, rather than grow.

Some systems are based on functions with multiple factors, some that amplify change, some that dampen it. These factors may themselves kick in conditionally - the amplifying factor dominates until some threshold is reached, upon which the dampener dominates. Weather properties appear to be such systems. For instance, the temperature of one location near a furnace may vary from second to second, but mostly increasing; until the fuel fed into the furnace reaches a peak rate of supply, or the air further out conducts heat faster than the furnace can produce it, at which the temperature now stabilizes or drops.

Expand full comment
Frog H Emoth's avatar

Behold! Think of the butterfly's wing impact as an analog to compound interest. It is not that the single change 'cascades' into large effects, it is that the cumulative effect of billions of tiny changes over time cascades into large effects

Expand full comment
Max More's avatar

That seems like an entirely different phenomenon. Other than "small leading to big" I don't see any relationship between the butterfly example and compound interest.

Expand full comment
Doctor Hammer's avatar

I think it helps to understand to not think of it as a butterfly causing hurricanes in some direct sense, but in terms of things with sharp edge cases between A and B being affected by the sum of small things.

So for instance, rain has a fairly large impact on voter turn out (when they had to leave their house, at least), and so in elections where there is a narrow margin rain might make the difference in who votes, and thus change the election, which has large knock on effects on policy and economic factors years later. Did the rain cause the local zoning law changes? Well, in part, yes.

Or consider the phrase the straw that broke the camel's back. The effect of each straw is very close to zero, but it is importantly not zero, such that it aggregates over time.

Or the plot from Superman 3 and Office Space, of collecting all the tiny fractions of pennies from financial accounts over time to amass a fortune.

That's the intuition here, that tiny differences we would normally round down to zero in our heads actually do sum up and matter, because across a very large system there are enough tiny differences to add up. That then suggests that our ability to measure and know initial conditions is a huge limit on our ability to predict most complex systems where feedback loops come into play.

Expand full comment
David Friedman's avatar

I believe the tiny fractions is real, someone who programmed a bank computer to transfer rounding errors to his account.

Expand full comment
Max More's avatar

Yes, but I don't see how the Lex Luthor scheme has anything to do with nonlinear effects.

Expand full comment
Chartertopia's avatar

I heard that story long long ago, as if it were real, complete with how long it took to even realize it was happening. But no references, no names, not even gonna try finding it. About the most I can say is I think it was pre-internet.

Expand full comment
David Friedman's avatar

I am reasonably sure that I saw it as a real story, with relevant details, back when I was covering computer crime in my "Legal Issues of the 21st Century" class, which I started in the 20th century.

Expand full comment
Doctor Hammer's avatar

I would totally believe that. Really, I would be kind of disappointed in humanity if no one did. :)

Expand full comment
Andy G's avatar

"Instead, background conditions typically swamp the small effect..."

Typically, no doubt.

But "typically" is not *always*...

Expand full comment
Max More's avatar

Clearly. But the rare exceptions require explanation. For instance, you have to identify positive feedback loops that actually work in the real world.

Expand full comment
DinoNerd's avatar

Yes it is. At least to a human mind that can't conceive of not-quite-married, intersex, or other exceptions to binary models they've mistaken for Truth.

Sarcasm aside, I fear we all fall into traps like mistaking typically for always, even when we know they are lying in wait, and try to consciously avoid them. The best we can do is stay out of more of them than we would acting totally unconsciously.

Expand full comment
David Friedman's avatar

Binary models can be a lot easier to work with than continuous model. Consider criminal trials. It's much easier to frame the outcome as innocent/guilty than as probability of guilt. Nothing relevant to who should vote or be treated as an adult in other ways changes discontinuously on someone's birthday.

Sometimes we use continuous models, but that requires mapping to a continuous conclusion. If you tell me you will meet me at five I don't stop waiting for you one second after, I just let my probability that you will show up and my opinion of your reliability decline continuously over the next hour. But sometimes we need discontinuous conclusions.

Expand full comment
DinoNerd's avatar

Yes, it's convenient to use binary models. It's unfortunate when that convenience comes at the expense of accuracy, or justice, but often the one harmed is not the person who finds the model convenient.

I had the misfortune to be born with a body that proclaimed to all binary thinkers a number of things that were not true about my capabilities and interests, and recent American politics reinforces both my anger about this situation, and my fear that the situation is worsening, maybe back to what it was when I was born, or even beyond that.

But it would be convenient if everyone like me was e.g. assigned work congenial to the majority of people like me. Well, convenient to everyone but those who'd be better suited if allowed to follow their own talents and preferences. Surely more convenient for society and its rulers. You can't expect nuance from a leader that has so many more important decisions to make.

Expand full comment