The Problem with Existential Risk Dialogue in Regards to AGI

First, I'd like to preface this post by stating my general skepticism of the possibility of "AGI"; however loosely defined we take that concept to be. I wrote a post on my other blog "Why Large Language Models are Not and Will Not Become Artificial General Intelligence" where I talked about some of the resources I've been reading leading to my suspicions of the possibility of AGI. For this post, I will assume that AGI is possible, and will likely emerge in the near future. However, I am not sure how much my prior beliefs will impact my assessment of AI X-Risk. My tentative conclusion is that we simply do not know whether AGI, in its near term instantiation, will be equivalent to other risks such as nuclear proliferation or global pandemics, without assuming some very specific trajectories, capabilities of AGI, and how it will interact within a complex socio-economic system.

Here is the main contention; if Strong AI (AGI) were to emerge, human societies will approach the top right section of the Scope-Severity grid introduced by Nick Bostrom in his paper "Existential Risk Prevention as Global Priority". (See link for a ton of papers on Existential-Risk)

Unleashing unconstrained AGI will cause society to approach the Cosmic-Hellish scope. At face value I find this kind of hilarious. This is not to say that we shouldn't be concerned with the unforeseen risks associated with open-sourcing a new powerful technology. I just mean to say that these risks are in principle unknowable and much of the hysteria is coming from unwarranted expectations of what an AGI can do. Again, I will reiterate, I think there will be risks, I am just not quite sure AGI by itself pushes us towards that area of the chart. 

In essence, the problem we are attempting to solve is something that has alluded people for centuries: "If new technology is introduced into a complex system, what will happen?" This is something financial people are particularly interested in because new technology implies new opportunity for alpha. Predicting the impact of these industry disrupting technologies has been extraordinarily difficult, and is something I take to be impossible in principle. This implies knowing something about human ingenuity, which to me simply seems impossible to anticipate. We do not have models that can account for such complexity. Stuart Kauffman wrote a paper "No entailing laws, but enablement in the evolution of the biosphere" which discusses concepts I find to be very relevant in the discussion of X-risk. A quote from the abstract:

Biological evolution is a complex blend of ever changing structural stability, variability and emergence of new phenotypes, niches, ecosystems. We wish to argue that the evolution of life marks the end of a physics world view of law entailed dynamics. Our considerations depend upon discussing the variability of the very "contexts of life": the interactions between organisms, biological niches and ecosystems. These are ever changing, intrinsically indeterminate and even unprestatable: we do not know ahead of time the "niches" which constitute the boundary conditions on selection. More generally, by the mathematical unprestatability of the "phase space" (space of possibilities), no laws of motion can be formulated for evolution. We call this radical emergence, from life to life.

I should note that Kauffman is a visionary when it comes to understanding complexity. He is a theoretical biologist (among other things) at Santa Fe Institute. Kauffman is trying to understand how useful mathematics is for studying the evolution of the biosphere. Consider mathematical models in physics (where we get our physical laws), he claims we cannot write down equations like this for systems describing biological phenomenon. In physics, the entire phase space can always be written down, all possible positions of a particle are defined by the boundary conditions. Without these, you cannot write down a system of differential equations to trace out the dynamics given some initial conditions. Kauffman claims that functions are part of the biological phase space. This is radically different than simple systems describing lower level behavior. Evolution provides for ever more novel functionality; and the number of possible functions is indefinite (not infinite). You cannot pre-state the all of the possible functionality emerging in biological phase space, making these systems far less predictable than the models in physics. He introduces the "Screwdriver Argument", posing the question "How many possible uses are there of a screw driver?" When you consider this thought experiment, you realize that the number of possible functions is not infinite, but not pre-statable. A screwdriver might have many possible functions in one environment, but far less in another environment. If we consider something like biological exaptation, we can see that repurposing is something fundamental in biological domains. There is no algorithm that can list all possible uses of a screwdriver; in other words it is an undecidable problem. New functionality is not entailed by anything; there are no law-like entailing features of biological systems. In other words, these are pre-adaptations that cannot be known by the researcher ahead of time. His conclusion is that since we cannot state biological phase space, we cannot write down the relevant variables, so we cannot write down differential equations describing the laws of motion for the evolving biosphere. 

How is this relevant to X-Risk? Well, I consider socio-economic systems to be just as creative as biological systems, perhaps even more. We literally cannot state the possible functions of an AGI, so we have little basis for asserting what outcomes are more likely than others. If we take "risk" to mean "the possibility of something bad happening", more specifically "the potential for negative downside measured by loss in some relevant variable", I don't see how we can construct a space of possibilities and assign probabilities to calculate expected loss in any meaningful way. Let's take the ISO definition of Risk to see if a bit of clarity can help us consider whether constructing a risk model is feasible in this situation. 

ISO Guide 73:2009 defines risk as, effect of uncertainty on objectives

Note 1: An effect is a deviation from the expected – positive or negative.

Note 2: Objectives can have different aspects (such as financial, health and safety, and environmental goals) and can apply at different levels (such as strategic, organization-wide, project, product and process).

Note 3: Risk is often characterized by reference to potential events and consequences or a combination of these.

Note 4: Risk is often expressed in terms of a combination of the consequences of an event (including changes in circumstances) and the associated likelihood of occurrence.

Note 5: Uncertainty is the state, even partial, of deficiency of information related to, understanding or knowledge of, an event, its consequence, or likelihood.

Note 2 seems relevant in this discussion because something can only be risky with respect to some pre-specified desired outcome. Note 5 is interesting as well because it describes risk as being related to our state of knowledge; something I alluded to above is that we tend to be in complete ignorance in situations like these. There are other proposed definitions listed on the wikipedia page, I'll list a few and comment on the ones I think are relevant. 

"Measurable uncertainty". This definition comes from Knight's "Risk, Uncertainty and Profit" (1921).[9] It allows "risk" to be used equally for positive and negative outcomes. In insurance, risk involves situations with unknown outcomes but known probability distributions.[10]

"Volatility of return". Equivalence between risk and variance of return was first identified in Markovitz's "Portfolio Selection" (1952).[11] In finance, volatility of return is often equated to risk.[12]

"Statistically expected loss". The expected value of loss was used to define risk by Wald (1939) in what is now known as decision theory.[13] The probability of an event multiplied by its magnitude was proposed as a definition of risk for the planning of the Delta Works in 1953, a flood protection program in the Netherlands.[14] It was adopted by the US Nuclear Regulatory Commission (1975),[15] and remains widely used.[7]

"Consequences and associated uncertainty". This was proposed by Kaplan & Garrick (1981).[16] This definition is preferred in Bayesian analysis, which sees risk as the combination of events and uncertainties about them.[17]

"Uncertain events affecting objectives". This definition was adopted by the Association for Project Management (1997).[18][19] With slight rewording it became the definition in ISO Guide 73.[3]

"Asset, threat and vulnerability". This definition comes from the Threat Analysis Group (2010) in the context of computer security.[21]

There are a few reoccurring themes: Measurability of uncertainty and deviation from a target value. A set of probabilities over a set of outcomes that are unpreferable. Risk minimization then becomes shifting that distribution away from that set of outcomes towards something preferred; or controlling a set of outcomes. I have no problems with these definitions. My main concern is whether these definitions and models are suitable for assessing whether AGI poses existential risk. Remember above when I was mentioning biological phase space; we have a very similar issue here. How feasible is it to map out the sources of risk and assign useful probabilities to a set of outcomes? I have no doubts that there will be some transformative effect, but to immediately jump to the x-risk quadrant absent any measurable metrics  and well-defined outcomes seems hasty. Contrast AGI with something known to be a serious x-risk; water pollution. We actually know what the immediate consequences will be given our understanding of human health. We can somewhat anticipate the subsequent economic disruptions that will follow if a water source becomes overly polluted. We could do some sort of geographical analysis and identify points of stress on the surrounding areas. Within the X-Risk AGI discussions, I don't really see any grounded approach. Rather, I see constant reference to "AGI creating an even smarter AGI that will suddenly decide to wipe out the human race"; the justification coming purely from a-priori thought experiments. I've not seen anything measurable. The only reasonable fears I've seen are in regards to cybersecurity and bioterrorism. I'll talk more about that later. My main point so far is to show that in proper risk analysis you can actually estimate empirical probabilities against a set of outcomes, but that we fundamentally constrained in doing this in the context of AGI. Apart from the few scenarios I mentioned above, I've not seen any comprehensive attempts to identify sources of risk, events, their causes and their potential consequences. This is likely due to the fact that the effects of AGI will literally be unpredictable. Since we've not even come as far as risk identification, we have obviously not done much proper risk analysis. To my knowledge, we have not:

1. Determined the sources, causes and drivers of risk
2. Investigated the effectiveness of existing or new controls
3. Been able to analyze possible consequences and their likelihood
4. Understood the interactions and dependencies between risks
5. Determined any measures of risk with respect to AGI
6. Verified or validated any results, since there are none
7. Conducted uncertainty and sensitivity analysis

To fully appreciate the problem, lets use the formal risk description introduced by Terje Aven in Quantitative Risk Assessment:

Risk is described as a tuple

where:

${\displaystyle \text{A}}$ is an event that might occur

${\displaystyle \text{C}}$ is the consequences of the event

${\displaystyle \text{U}}$ is an assessment of uncertainties

${\displaystyle \text{P}}$ is a knowledge-based probability of the event

${\displaystyle \text{K}}$ is the background knowledge that U and P are based on

Remember that X-Risk is defined as risk that's approaching that Cosmic-Hellish region on the Scope-Severity grid and that risk management is primarily concerned with controlling a set of outcomes or manipulating what you have control over to avoid outcomes you find unfavorable. This implies understanding the set of procedures, actions, decisions, and controls that lead to particular outcomes. We can immediately infer that K will be an incredibly small. I claim we have very little background knowledge in this particular case and that there are no reasonable analogues to compare against. Furthermore, our track record of understanding (or knowing) what will happen if some exogenous shock effects our system, is very bad (gauged by prior performance of economists and other social scientists). Since K is minimal, P will be based on an insufficient amount of knowledge necessary to estimate quantify our uncertainty. For any event in A, the corresponding P will be unstable. Furthermore, given the nature of the system under discussion, U and A are without a doubt, not going to be exhausted. We simply do not know all of the events contained in A, and so we cannot reasonably assign probabilities to the members in this set. We don't know this because the state space of possibilities is unknown; analogous to Kauffman's reasoning, there is an indefinite number of possible scenarios and interactions enabled by an AGI because we are in the space of functionality. All of this implies that C is pretty much unknown; we are in the realm of speculation at this point. It's probably fair to assume that the consequences will be non-linear. Like most complex systems, there is likely to be a convexity bias: 

What this tells us is that our consequences f(X) are a function of random variables, but the responses are non-linear. In a financial context, a gain is defined as the positive difference between the expected value and the actual, and a loss is defined as the negative difference. These tend to be non-linear because of unknown interactions. So in the context of AGI risk, it's very possible that there are convexities in terms of consequences. I think we can more or less expect this to happen regardless of whatever intervention someone is trying to propose. 

My main point so far is that we are operating in a regime characterized by Knightian Uncertainty, so its hard to say where AGI will place us on the Scope-Severity chart with any degree of confidence, let alone asserting confidently that AGI introduces us to x-risk. Frank Knight distinguished risk from uncertainty. On this view, we are lacking quantifiable knowledge about possible occurrences, accounting for essential unpredictability:

"Uncertainty must be taken in a sense radically distinct from the familiar notion of Risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far-reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all."

In reference to the model of risk earlier, there is a separate category of immeasurable risk that is a function of ignorance, rather than statistical noise. Instead of R being a function of U, there is a separate class U determined by our lack of knowledge. In so many words, there is a certain degree of unknowability when a system is sufficiently complex. Knightian risk is computable, Knightian Uncertainty is not computable. Nassim Taleb introduced a concept I find relevant in these scenarios: the "Ludic Fallacy". In The Black Swan he juxtaposes casino games from games in the real world. Gambling scenarios are a type of domesticated uncertainty; you know the rules, you can calculate the odds, but the type of uncertainty encountered here is typically non-transferrable outside into real situations. Casino probabilities are always Gaussian and computable, the rules of the games are also not going to abruptly change unexpectedly. Taleb states that in real life, you don't know the odds, they need to be discovered and the sources of uncertainty are not defined. The idea behind the Ludic Fallacy is that there is a tendency to transfer laboratory contrived models onto the real world. This is similar to what I've been saying before in regards to biological phase space being indefinite. The fallacy is a variation of confusing the map for the territory and not recognizing model uncertainty; although this is not quite representative of the concept because "Epistemic uncertainty, which usually is due

to insufficient knowledge about the model, can be reduced by collecting more data

or refining the learning models." Unlike epistemic uncertainty, Knightian uncertainty cannot be resolved through more information or better models because it is about the inherent unpredictability or ambiguity of the situation. The paradigmatic example of Knightian Uncertainty is a technological innovation; it is simply unprecedented. The Ludic Fallacy will arise when someone thinks they can account for this type of uncertainty simply by refining a model after collecting more data. We could simply bootstrap our distribution to get tighter parameter estimates, under conditions of epistemic uncertainty. But Knightian uncertainty cannot be quantified or measured because the underlying probability distributions are unknowable. Epistemic uncertainty is related to measurement error, Knightian is related to our fundamental ignorance. 

Okay so bringing this back to X-Risk; I've not seen any risk analysis and if I did I would find it highly suspect because we are operating under Knightian Uncertainty. We simply do not know what will happen. Trying to predict a Black Swan event arising from the introduction of this new technology is futile. Even if we could manage to attain decent forecasts, socio-economic systems are composed of adaptive agents who will respond in unanticipated ways to some proposed intervention, undermining our original forecast (see the Lucas Critique). Feedback dynamics are notoriously difficult to capture and represent with mathematical models representing complex systems. If I grant that AGI means superior intelligence and processing power relative to humans, I don't see how the agent couldn't identify optimal adaptive strategies to our ability to control it. But as I alluded to earlier, runaway AI is not the risk I think we should be concerned with. My idea is simply that socio-economic systems are not control systems, they are complex adaptive systems that require fundamentally different methods for understanding their dynamics. Economists have relied heavily on methods such as the Kalman Filter in macro-econometrics, at the expense of understanding concepts like path dependency and niche construction which are crucial for understanding something complex like a modern economy. 

So far I've been arguing that we literally have no reason to suspect we are approaching the X-Risk quadrant for two reasons: No one (to my knowledge) has demonstrated a principled approach for identifying the actual risks and even if they did, the model will be fundamentally flawed and uninformative. So what are the reasons people give to justify such a position? Existential dread and apocalypticism are truly fascinating properties of human psychology that have been with us for millennia. People have been predicting catastrophic events that will end the world throughout the entire historical record. Simultaneously, humans are incredible bad at identifying very subtle pernicious risks that pose existential threats; the ones that don't pose immediate threats because our perception of time prevents us from understanding gradualism. People tend to anthropomorphize, which I think is incredibly relevant in this situation characterized by unaligned AI. It's far easier to assume this "thing" has agency, will become our overlord, and one day decide to wipe us out rather than understanding that "Alignment" is simply yet another manifestation of an alignment problem institutions and individuals have been dealing with for centuries: collective decision making, collective behavior, and collective stupidity. It's much simpler to characterize AGI as something that will cause immediate devastation rather than systemic instabilities, structural uncertainties, and risk amplification. So at what point do these fears and anxieties enter the discussion? I propose they are always lurking in the background, systematically obscuring our ability to generate unbiased descriptions of the true parameters governing our uncertainty. If we think of a traditional risk analysis where concepts like hazards, exposures, and reliability are paramount, we see a flow like this:

I found a nice flow chart online that shows different steps in the risk assessment process.

We could summarize the above process with an additional pre-condition that seems to be present in any risk assessment:

(Within the context of Human psychology influenced by heuristics, biases, and dread)...

1. Define scope
2. Vulnerability identification
3. Threat identification 
4. Control analysis
5. Probability determination 
6. Impact analysis 
7. Residual risk determination 
8. Control recommendation 

Unsurprisingly, "cognitive bias" has been used to justify the exact opposite of what I've been suggesting throughout this post. Some people argue that existential risks are ignored due to biases such as Scope Neglect or Availability Heuristic, suggesting x-risk is ignored because we simply cannot comprehend the scope of the problem and have no immediate examples of existential destruction to compel us to action. Unfortunately, this undermines it's very own position. In the absence of a principled method for determining whether something is an existential risk, I can easily invoke the cognitive bias proposition to undercut the idea that some novelty poses an existential risk, by appealing to an alternative subset of biases that would explain alarmism. The cherry-picking can go both ways; unfortunately people have become lazy with their arguments, selecting cognitive biases that undermine their opposition, while failing to check against the rest of the possible biases that might undermine their own position. A cognitive bias is a deviation from a rational ideal; are these people suggesting they have rationally estimated the true probabilities? Advocates of the position might pose the argument:

• P1. X Poses an Existential Threat
• P2. People do not see what I see is an existential threat because of cognitive biases A,B,C
• Assumption: Cognitive biases A,B,C are actively instantiated in the detractor
• Warrant: Prima Facie, we should assume X is an Existential Threat, absent any defeaters
• C. The assumption of Existential Threat holds

On its face we should be able to see that a symmetric argument can be constructed yielding the negation of this chain of reasoning. This is called a symmetry breaker in philosophical argumentation:

• P1. Person Y claims X poses an Existential Threat
• P2. Person Y is overhyping, amplifying, or being alarmist due to alternative cognitive biases D,E,F
• Assumption: Cognitive biases D,E,F are actively instantiated in the detractor
• Warrant: Prima Facie, we should not accept such a claim in the absence of a principled analysis, or if there are any overriding defeaters
• C. There is no existential threat of X 

Essentially I am arguing that we should give more credence to the second argument structure because the first argument structure gives too much leeway. Under structure number one, anyone can propose something is an existential risk without any principled reason. This effectively shifts the burden of proof on to anyone having contentions with it, while appealing to baseless assumptions reliant on a shared dread that might not reflect the true conditions. The second structure imposes a constraint preventing whimsical assertions, demanding a burden of production; it is the preferred null hypothesis. Therefore, I will be reliant on the second structure. Given the nature of AGI X-Risk hysteria, I think there is ample evidence to suggest that x-risk is a function of an alternative set of cognitive biases that distort proper risk assessment, overriding any suggestion that AGI X-Risk is a genuine concern but is overlooked by some contrary set of biases. Using Bayes Theorem, I would suggest:

P(H=NOT X-RISK|E,B1,K)*P(H=NOT X-RISK) > P(H=X-RISK|E,B2,K)*P(H=X-RISK)

Where 

• K = Common Stock of Knowledge
• B1 = Set of cognitive biases in favor of the negation
• B2 = Set of cognitive biases in affirming the proposition
• E = Relevant evidence or data

Invoking Bayes Factor to yield posterior odds:

Let...

• D1 = (E,B1,K)
• D2 = (E,B2,K)
• H1 = NOT X-RISK
• H2 = X-RISK
• BF = P(D1|H1)/P(D2|H2) -> Bayes Factor
• PR =  P(H1)/P(H2) -> Prior Odds
• PO = P(H1|D1)/P(H2|D2) - >Posterior Odds

P(H1|D1)/P(H2|D2) =  P(H1)/P(H2)* P(D1|H1)/P(D2|H2) 

or

PO = PR * BF

This framework is useful because I think it would help us understand what is driving some of the hysteria. Under this framework the question becomes more tractable. A value of PO > 1 means that H1is more strongly supported by the data under consideration than H2. According to Jeffreys:

Bayes factor	Evidence category
> 100	Extreme evidence for H1
30 - 100	Very strong evidence for H1
10 - 30	Strong evidence for H1
3 - 10	Moderate evidence for H1
1 - 3	Anecdotal evidence for H1
1	No evidence
1/3 - 1	Anecdotal evidence for H0
1/10 - 1/3	Moderate evidence for H0
1/30 - 1/10	Strong evidence for H0
1/100 - 1/30	Very strong evidence for H0
< 1/100	Extreme evidence for H0

So I am suggesting PO > 1. Others would suggest PO< 1.  Consider the bayes factor and prior odds. The Bayes Factor might be indicating that P(D1|H1)/P(D2|H2) < 1, meaning that the data is more probable under H2. The Prior Odds might be indicating that P(H1)/P(H2) < 1, suggesting that the prior probability of H2 was larger prior to observing any data. If the advocate of X-Risk wants to suggest that the posterior odds are less than one, this would mean that one of these components (or both) would be less than one. Lets consider the scenario that Bayes Factor is less than one. This would suggest that there is significant evidence in favor of the idea that X-Risk is more probable given everything we know. Where is this evidence I might ask? I've yet to see any concrete evidence indicating "Cosmic Hellish" consequences. I will grant that AGI poses risks, but the hypothesis is X-RISK; so there must be evidence beyond trivial risks to convince a rational person of the doomsday. What about the prior odds? If you are trying to convince me that the posterior odds are are less than one under this framework, this would mean that the prior odds are our source of bias. This is what I suggest is the problem: our prior beliefs contain irrational fears and anxieties causing the posterior odds to seem favorable to H2. The prior odds are the main source contributing to the idea that H2 is more likely than H1. 

I'd first just like to clarify that, as mentioned before, I obviously believe there are existential risks to humanity. This is not controversial. My contention is that there is no basis to suggest AGI is an existential risk and much of the fear is coming from our priors which have been skewed by media, cinema, and hysteria. I should also note that I am merely suggesting that these are possible explanatory factors. I do not wish to give a comprehensive argument that they are indeed impacting the priors. Unfortunately, in the field of "Futures Studies", there simply is no good methodology for distinguishing between a "risk with significant consequences" and an "existential risk". My reading of this literature suggests that much of it comes down to arbitrary thought experiments with very little thought into the side effects of intervening on our system. Contrast AGI risk with other known concrete existential threats and we find that people are simply committing a category error. If we take a look at the Wikipedia page for Existential risk from artificial general intelligence, it seems a lot of the hysteria derives from arguments about superintelligence presented by Nick Bostrom and the idea of a feasible superintelligence argued for by David Chalmers. The problem is that AGI is being conflated with the arbitrary notion of a superintelligence. Intelligence itself is not well defined or understood, let alone a "super" intelligence. The arguments also rely upon highly contestable theories of mind. I've touched on this in my post on the limitations of AGI. We should not utilize such a poorly understood concept to inform policy. With silly philosophical arguments like those presented by Bostrom and Chalmers, we are directing discussion away from the actual implications and risks presented by AGI. 

So where should the discussion be directed? I have a few thoughts, influenced by Carla Cremer (linked below) and Igor Krawczuk. The risks posed by AGI will be yet another human problem, nothing more, nothing to do with a superintelligence. AGI serves as a risk amplifier; it introduces additional instabilities into an already unstable system, interacting in ways impossible to know a-priori. Our approach should not be to "program the best ethical system" into the AGI. That is a recipe for disaster. We typically don't even know what the "good" decision is ahead of time, and tend to rationalize ex-post-facto with the aid of Hindsight Bias. Some moral problems are fundamentally unsolvable in a mathematical sense. Consider the Trolley Problem, should we pre-program some algorithmic ethic to determine how to handle analogous scenarios? AGI will certainly cause a regime shift. There will be a transition period with unknown duration. The outcome is fundamentally uncertain. Our strategy should be adaptive. Our efforts should revolve around ensuring our systems have sufficient agility to handle unforeseen shocks. Here are a few considerations:

1. AGI risk is primarily a consequence of deploying tools that are not quite ready to be applied because their environment or context is too complicated or complex. If we prematurely deploy AGI as a subsystem within a larger system, introducing uncertainty, the tool cannot capture all of the complexity. This leads to poorer performance. In other words, our problem will be that we have too much confidence in these systems.
2. Instead of conceiving runaway intelligence overlord who takes control over everyone (a very abstract and implausible story) where we all die due to one single event , think of the problem in terms of entangled systemic interacting problems, that are hard to disentangle, creating particular scenarios (that we don’t know ahead of time) where something blows up (this depends on my deflationary view of AGI as being merely a collection of statistics and algorithms). Shift the conversation away from superintelligence as much as possible. 
3. Interrogate your assumptions. Many AI doomers base their claims on a set of assumptions, resting on a set of assumptions, that ultimately seem like spooky science fiction. Focus your attention on democratic, participative solutions. 
4. The risks come from system design and collective social choice problems from economics and political science. How to aggregate preferences democratically. It’s not about writing an algorithm for ethical behavior or figuring out the one master algorithm. It is very much a political problem, organizational in nature. Questions of risk imply a power dynamic that should involve all stakeholders in a society, not just a few people in silicon valley. 
5. A corollary to number two, we ought to identify points of vulnerability that lead to cascading failures. If we begin replacing well-tested control systems with probabilistic systems like an AGI, there might be faults that could propagate through the system, leading to a crash. This will be very crucial for vital infrastructure. 
6. A corollary to number 5, if we are to roll out systems with inherent uncertainty, we should invest more in our understanding of concepts like Percolation Theory, Interdependent networks, diffusion, and many other concepts from network theory and complexity science so we can better anticipate domino effects and vicious cycles that are difficult to escape. We need to understand how uncertain events can lead to second and third order events in a principled and reliable way. We need better sources of information to help us identify structural instabilities that lead to progressive collapse. 

Transformative innovations absolutely require adaptive mechanisms to withstand such shocks. This means creating massive research programs to identify solutions in this new environment. Brittle institutions will not be able to withstand shocks. We cannot know all of the shocks ahead of time. We need to adopt a strategy that generates the tools necessary to remain adaptive as possible in this new regime. 

Fragile systems will almost certainly fail. This is almost a truism. But before I finish off, I would just like to consider some historical examples where highly disruptive innovations resulted in chaos. Consider the printing press. This was arguably one of the most transformative technologies we have ever created. There were many positive gains such as literacy, but institutions were absolutely not ready for such a technology. Revolts and wars suddenly ensued. The protestant revolution begun. Civilization fundamentally changed. Some of our most very fundamental concepts came into question, such as the nature of authority, the nature of knowledge, and the very concept of Truth itself. Even under all of this chaos, this was not an X-Risk. It was nevertheless dramatically destabilizing. Political institutions were not ready to handle an educated population. Ancient Greek texts began recirculating, creating a pluralism Europe hadn't seen for centuries. We can definitely expect something like this to happen, and can see some of it in its early stages. All of our propaganda models will become obsolete. We will need to better understand how information diffuses through society.  I don't want to list everything. There are plenty of books written on this massive transformation in history. It's safe to say that rigid institutions in Europe, incapable of forward thinking and agility, did not survive. Another incredible example of technological innovations impacting society is with Tsar Nicholas II. Obviously the overthrow of the Tsar is more complex than a single technological innovation. Nevertheless, it is illustrative, because it shows how rigid institutions collapse in face of rapid technological and social innovations.  Citizens began demanding representation and contact with western Liberal democracies made them rethink their own political arrangement; all of this amplified by the adoption of industrial technologies increasing the standard of living. The severity of the shocks facing Russian civilization in the late 1800's and early 1900's lead to a systemic collapse, the rest is history. When you read the history it is fascinating. The monarch seemed almost ignorant to the massive disruptions and growing resentment. This is a textbook example of not be adaptable. Alas, this was not an existential thread. 

Technological disruptions happen unexpectedly. Some result in consequences more sever than others. We cannot predict ahead of time what will happen, but we can employ strategies to minimize the impact of exogenous shocks such that the system transitions smoothly to a new regime; avoiding a collapse. Focusing on existential risk and superintelligence obfuscates the dialogue, deteriorating it severely by focusing on fictitious constructs rather than emphasizing actionable policies to cope with an incoming shock.  

Extra Resources:

• #99 - CARLA CREMER & IGOR KRAWCZUK - X-Risk, Governance, Effective Altruism
• SUPERINTELLIGENCE (DAVID CHALMERS)
• Democratising Risk: In Search of a Methodology to Study Existential Risk