Discusses why I think this database may be valuable
Discusses some pros and cons of using or making such estimates
Key links relevant to existential risk estimates
Here’s a spreadsheet listing some key existential-risk-related things people have estimated, without estimates in it.
The link makes a copy of the spreadsheet so that you can add your own estimates to it.
I mention this first so that you have the option of providing somewhat independent estimates, before looking at (more?) estimates from others.
Some discussion of good techniques for forecasting, which may or may not apply to such long-range and extreme-outcome forecasts, can be found here, here, here, here, and here.
Here’s a database of all estimates of existential risks, or similarly extreme outcomes (e.g., reduction in the expected value of the long-term future), which I’m aware of. I intend to add to it over time, and hope readers suggest additions as well.
Here's an EAGx talk I gave that's basically a better structured version of this post (as I've now thought about this topic more), though without the links. So you may wish to watch that, and then just skim this post for links.
The appendix of this article by Beard et al. is where I got many of the estimates from, and it provides more detail on the context and methodologies of those estimates than I do in the database.
Beard et al. also critically discuss the various methodologies by which existential-risk-relevant estimates have been or could be derived. And Baum [EA · GW] provides additional excellent commentary.
In this post [EA · GW], I discuss some pros and cons of using or stating explicit probabilities in general.
Why this database may be valuable
I’d bet that the majority of people reading this sentence have, at some point, seen one or more estimates of extinction risk by the year 2100, from one particular source. These estimates may in fact have played a role in major decisions of yours; I believe they played a role in my own career transition. That source is an informal survey of global catastrophic risk [EA(p) · GW(p)] researchers, from 2008.
The disadvantage [of that survey] is that the estimates were likely highly subjective and unreliable, especially as the survey did not account for response bias, and the respondents were not calibrated beforehand.
Additionally, in any case, it was just one informal survey, and is now 12 years old. So why is it so frequently referenced? And why has it plausibly (in my view) influenced so many people?
I originally thought the answer to each of these questions was essentially that we have so little else to go on, and the topic is so important. It seemed to me there had just been so few attempts to actually estimate existential risks or similarly extreme outcomes (e.g., extinction risk, reduction in the expected value of the long-term future).
I’d argue that this causes two problems:
We have less information to inform decisions such as whether to prioritise longtermism over other cause areas, whether to prioritise existential risk reduction over other longtermist strategies, and especially which existential risks to be most concerned about.
Indeed, it seems to me probably not ideal how many strategic decisions people concerned about existential risks (myself included) have made so far without having first collected and critiqued a wide array of such estimates.
This second issue seems all the more concerning given the many reasons we have for skepticism about estimates on these matters, such as:
The lack of directly relevant data (e.g., prior human extinction events) to inform the estimates
Estimates often being little more than quick guesses
It often being hard to interpret what is actually being estimated
Thus, we may be anchoring on a small handful of estimates which could in fact warrant little trust.
But what else are we to do? One option would be to each form our own, independent estimates. This may be ideal, which is why I opened by linking to a spreadsheet you can duplicate to do that in.
But it seems plausible that “expert” estimates of these long-range, extreme outcomes would correlate better with what the future actually has in store for us than the estimates most of us could come up with, at least without investing a great deal of time into it. (Though I’m certainly not sure of that, and I don’t know of any directly relevant evidence. Plus, Tetlock’s work may provide some reason to doubt the idea that “experts” can be assumed to make better forecasts in general.)
And in any case, most of us have probably already been exposed to a handful of relevant estimates, making it harder to generate “independent” estimates of our own.
So I decided to collect all quantitative estimates I could find of existential risks or other similarly “extreme outcomes”, as this might:
Be useful for decision-making, because I do think these estimates are likely better than nothing or than just phrases like “plausible”, plausibly by a very large margin.
Reduce a relatively “blind” reliance on any particular source of estimates, and perhaps even make it easier for people to produce effectively somewhat independent estimates. My hope is that providing a variety of different estimates will throw anchors in many directions (so to speak) in a way that somewhat “cancels out”, and that it’ll also highlight the occasionally major discrepancies between different estimates. I do not want you to take these estimates as gospel.
Above, I said “I originally thought the answer to each of these questions was essentially that we have so little else to go on, and the topic is so important.” I still think that’s not far off the truth. But since starting this database, I discovered Beard et al.’s appendix, which has a variety of other relevant estimates. And I wondered whether that meant this database wouldn’t be useful.
But then I realised that many or most of the relevant estimates from that appendix weren’t mentioned in Ord’s book. And it’s still the case that I’ve seen the 2008 survey estimates many times, and never seen most of the estimates in Beard et al.’s appendix (see this post [EA · GW] for an indication that I’m not alone in that). And Beard et al. don’t mention Ord’s estimates (perhaps because it was released only a couple months before Ord’s book was), nor a variety of other estimates I found (perhaps because they’re from “informal sources” like 80,000 Hours articles).
So I still think collecting all relevant estimates in one place may be useful. And another benefit is that this database can be updated over time, including via suggestions from readers, so it can hopefully become comprehensive and stay up to date.
For the benefit of other readers: These estimates are about the "Timeline until human-level AGI". The post Pablo linked to describes the methodology behind the estimates.
I don't think this really falls directly under the scope of the spreadsheet (in that this isn't directly an estimate of the chance of a catastrophe), so I haven't added it to the spreadsheet itself, but it's definitely relevant when considering x-risk/GCR estimates.
Do you know if there's a collection of AI timelines forecasts somewhere? I'd imagine there would be?
Yes, sorry, I should have made that clearer—I posted here because I couldn't think of a more appropriate thread, and I didn't want to create a separate post (though maybe I could have used the "shortform" feature).
AI Impacts has a bunch of posts on this topic. This one discusses what appears to be the largest dataset of such predictions in existence. This one lists several analyses of time to human-level AI. And this one provides an overview of all their writings on AI timelines.
I think sharing that link in a comment here made sense; it's definitely relevant in relation to this topic. I was just flagging that I didn't think it made sense to add it to the spreadsheet itself.
But I think a similar spreadsheet to collect AI timelines estimates seems valuable (though I haven't thought much about information/attention hazards [LW · GW] considerations in relation to that). So thanks for linking to that AI Impacts post.
I've now sent AI Impacts a message asking whether they'd considered making it easy for people to comment on that dataset with links to additional predictions. So perhaps they'll do so, or perhaps they'll decide not to for good reasons.
I think this is an excellent initiative, thank you, Michael! (Disclaimer: Michael and I work together on Convergence.)
An assortment of thoughts:
More and more studious estimates of x-risks seem clearly very high value to me due to how much the likelihood of risks and events affect priorities and how the quality of the estimates affect our communication about these matters.
More estimates should generally should increase our common knowledge of the risks, and individually, if people think about how to make these estimates, they will reach a deeper understanding of the questions.
Breaking down the causes of one's estimates is generally valuable. It allows one to improve one's estimates, understanding of causation, and to discuss them in more detail.
More estimates can be bad if low quality estimates swamp out better quality ones somehow.
Estimates building on new (compared to earlier estimates) sources of information are especially interesting. Independent data sources increase our overall knowledge.
I strongly agree about the value of breaking down the causes of one's estimates, and about estimates building on new sources of info being particularly interesting. And I tentatively agree with your other points.
Two things I'd add:
Beard et al. have an interesting passage relevant to the idea of "Breaking down the causes of one's estimates":
Another approach is what Tonn and Stiefel refer to as ‘Evidential Reasoning’. This involves specifying the effect every piece of evidence has on one’s beliefs about the survival of humanity. Importantly, these probabilities should only reflect the change that this evidence makes, not one's initial prior beliefs, allowing others to assess them independently. As such, they will only be ‘imprecise probabilities’ that describe a small portion of the overall probability space, where the contribution each piece of evidence makes to one’s belief and its complement need not sum to 1. For instance, one might reason that evidence about the adaptability of humans to environmental changes suggests a 30 % probability that we will survive the next 1000 years, but only a 10 % probability that we will not.4 Combination functions can then be used to aggregate these imprecise probabilities to return the overall probability of extinction within this period. This method not only helps assessors determine the probability of extinction, but also provides others with information about the sources of evidence that contributed to this decision and the opportunity to determine how additional information might affect this.
I think an intro post on how to do estimates of this type better could be valuable. I also think it would likely benefit by drawing on the insights in (among other things) the sources I linked to in this sentence: "Some discussion of good techniques for forecasting, which may or may not apply to such long-range and extreme-outcome forecasts, can be found here, here, here, here, and here." And Beard et al. is also relevant, though much of what it covers might be hard for individual forecasters to implement with low effort.
Thank you for putting this spreadsheet database together! This seemed like a non-trivial amount of work, and it's pretty useful to have it all in one place. Similar to other comments, seeing this spreadsheet really made me want more consistent questions and forecasting formats such that all these people can make comparable predictions, and also to see the breakdowns of how people are thinking about these forecasts (I'm very excited about people structuring their forecasts more into specific assumptions and evidence).
I thought the 2008 GCR questions were really interesting, and plotted the median estimates here. I was surprised by / interested in:
How many more deaths were expected from wars than other disaster scenarios
For superintelligent AI, most of the probability mass was < 1M deaths, but there was a high probability (5%) on extinction
A natural pandemic was seen as more likely to cause > 1M deaths than an engineered pandemic (although less likely to cause > 1B deaths)
(I can't figure out how to make the image bigger, but you can click the link here to see the full snapshot)
Also, your comment suggests you already saw my comment quoting Beard et al. on breaking down the thought process behind forecasts. You might also be interested in Baum's great paper replying to and building on Beard et al.'s paper. I also commented on Baum's paper here [EA · GW], and Beard et al. replied to it here.
Yes, I share that desire for more comparability in predictions and more breakdowns of what's informing one's predictions. Though I'd also highlight that the predictions are often not even very clear in what they're about, let alone very comparable or clear in what's informing them. So clarity in what's being predicted might be the issue I'd target first. (Or one could target multiple issues at once.)
And your comments on and graph of the 2008 GCR conference results are interesting. Are the units on the y axis percentage points? E.g., is that indicating something like a 2.5% chance superintelligent AI kills 1 or fewer people? I feel like I wouldn't want to extrapolate that from the predictions the GCR researchers made (which didn't include predictions for 1 or fewer deaths); I'd guess they'd put more probability mass on 0.
The probability of an interval is the area under the graph! Currently, it's set to 0% that any of the disaster scenarios kill < 1 people. I agree this is probably incorrect, but I didn't want to make any other assumptions about points they didn't specify. Here's a version that explicitly states that.
The probability of an interval is the area under the graph!
It's not obvious to me how to interpret this without specifying the units on the y axis (percentage points?), and when the x axis is logarithmic and in units of numbers of deaths. E.g., for the probability of superintelligent AI killing between 1 and 10 people, should I multiply ~2.5 (height along x axis) by ~10 (length along y axis) and get 25%? But then I'll often be multiplying the height along the x axis by more than 100 and getting insane probabilities?
So at the moment I can make sense of which events are seen as more likely than other ones, but not the absolute likelihood they're assigned.
I may be making some basic mistake. Also feel free to point me to a pre-written guide to interpreting Elicit graphs.
“Personally, I now think we humans will be wiped out this century,” Frank Tipler told me. He may be the most pessimistic of Bayesian doomsayers, followed by Willard Wells (who gives the same time frame for the end of civilization and the beginning of the postapocalypse).
Thanks for mentioning this; it indeed wasn't in the database, and I've now added it.
Though I felt a bit hesitant about doing so, and haven't filled in the "What is their estimate?" column; I left this just as a quote.
We could interpret Tiper as saying "I now think there's a greater than 50% chance humans will be wiped out this century". But when people just say things like "I think X will happen" or "I think X won't happen", I don't feel super confident that they really meant "greater than 50%" or "less than 50%". For example, it seems plausible they just mean something like "I think we should pay [or less] more attention to the chance of X".
Also, when people don't give quantitative estimates, I feel a little less confident that they've thought for more than a few seconds about (a) precisely what they're predicting (e.g., does Tipler really mean extinction, or just the end of civilization as we know it?), and (b) how likely they really think the thing is. I don't feel super confident people have thought about (a) and (b) when they give quantitative predictions either, and indeed it's unclear what many predictions in the database are really about, and many of the predictors explicitly say things like "but this is quite a made up number and I'd probably say something different if you asked me again tomorrow". But I feel like people might tend to take their statements even less seriously when those statements remain fully qualitative.
Thanks for this database! I'm currently working on a project for the Foresight Centre (a think-tank at the Estonian parliament) about existential risks and the EU’s role in reducing them. I cover risks form AI, engineered pandemics, and climate change. For each risk, I discuss possible scenarios, probabilities, and the EUs role. I've found a couple of sources from your database on some of these risks that I hadn't seen before.
That sounds like a really valuable project! Glad to hear this database has been helpful in that.
I cover risks form AI, engineered pandemics, and climate change.
Do you mean that those are the only existential risks the Foresight Centre's/Estonian Parliament's work will cover (rather than just that those are the ones you're covering personally, or the ones that are being covered at the moment)? If so, I'd be interested to hear a bit about why? (Only if you're at liberty to discuss that publicly, of course!)
I ask because, while I do think that AI and engineered pandemics are probably the two biggest existential risks, I also think that other risks are noteworthy and probably roughly on par with climate change. I have in mind in particular "unforeseen" risks/technologies, nuclear weapons, nanotechnology, stable and global authoritarianism, and natural pandemics. I'd guess that it'd be hard to get buy-in for discussing nanotechnology, authoritarianism, and maybe unforeseen risks/technologies in the sort of report you're writing, but not too hard to get buy-in for discussing nuclear weapons and natural pandemics.
Existential risks are not something they have worked on before, so my project is a new addition to their portfolio. I didn't mention this but I intend to have a section for other risks depending on space. The reason climate change gets prioritized in the project is that arguably the EU has more of a role to play in climate change initiatives compared to, say, nuclear risks.
I imagine having climate change feature prominently could also help get buy-in for the whole project, since lots of people already care about climate change and it could be highlighted to them that the same basic reasoning arguably suggests they should care about other existential risks as well.
FWIW, I'd guess that the EU could do quite a bit on nuclear risks. But I haven't thought about that question specifically very much yet, and I'd agree that they can tackle a larger fraction of the issue of climate change.
From a very quick look, I think that most of the things in that spreadsheet are either already in this database, only somewhat relevant (e.g., general AI capabilities questions), or hardly relevant at all. But I wouldn't be surprised if some fiddling around with that tool could narrow the results down to be more useful (I haven't tried the tool yet), or if it becomes more useful in future.
Thanks for the feedback! And for your helpful suggestions in the spreadsheet itself.
Could you expand on what you mean by "one can argue that the non-conditional estimates are not directly decision-relevant"?
It does seem that, holding constant the accuracy of different estimates, the most decision-relevant estimates would be things like pairings of "What's the total x-risk this century, conditional on the EA community doing nothing about it" with "What's the total x-risk this century, conditional on a decent fraction of EAs dedicating their careers to x-risk reduction".
But it still seems like I'd want to make different decisions if I had reason to believe the risks from AI are 100 times larger than those from engineered pandemics, compared to if I believed the inverse. Obviously, that's not the whole story - it ignores at least tractability, neglectedness, and comparative advantage. But it does seem like a major part of the story.
But it still seems like I'd want to make different decisions if I had reason to believe the risks from AI are 100 times larger than those from engineered pandemics, compared to if I believed the inverse.
Agreed, but I would argue that in this example, acquiring that belief is consequential because it makes you update towards the estimate: "conditioned on us not making more efforts to mitigate existential risks from AI, the probability of an AI related existential catastrophe is ...".
I might have been nitpicking here, but just to give a sense of why I think this issue might be relevant:
Suppose Alice, who is involved in EA, tells a random person Bob: "there's a 5% chance we'll all die due to X". What is Bob more likely to do next: (1) convince himself that Alice is a crackpot and that her estimate is nonsense; or (2) accept Alice's estimate, i.e. accept that there's a 5% chance that he and everyone he cares about will die due to X, no matter what he and the rest of humanity will do. (Because the 5% estimate already took into account everything that Bob and the rest of humanity will do about X.)
Now suppose instead that Alice tells Bob "If humanity won't take X seriously, there's a 10% chance we'll all die due to X." I suspect that in this scenario Bob is more likely to seriously think about X.
I think I see your reasoning. But I might frame things differently.
In your first scenario, if Alice sees Bob seems to be saying "Oh, well then I've just got to accept that there's a 5% chance", she can clarify "No, hold up! That 5% was my unconditional estimate. That's what I think is going to happen. But I don't know how much effort people are going to put in. And you and I can help decide what the truth is on that - we can contribute to changing how much effort is put in. My estimate conditional on humanity taking this somewhat more seriously than I currently expect is a 1% chance of catastrophe, and conditional on us taking it as seriously as I think we should, the chance is <0.1%. And we can help make that happen."
See also a relevant passage from The Precipice, which I quote here [EA(p) · GW(p)].
Also, I think I see the difference between unconditional and conditional on us doing more than currently expected as more significant than the difference between unconditional and conditional on us doing less than currently expected. For me, the unconditional estimate is relevant mainly because it updates me towards beliefs like "There are probably low-hanging fruit for reducing this risk", or "conditioned on us making more efforts to mitigate existential risks from AI than currently expected, the probability of an AI related existential catastrophe is [notably lower number]."
Sort-of relevant to the above: "If humanity won't take X seriously, there's a 10% chance we'll all die due to X" is consistent with "Even if humanity does take X really really seriously, there still a 10% chance we'll all die due to X." So in either case, you plausibly might need to clarify what you mean and what the alternative scenarios are, depending on how people react.
Another thought: If you do make your own existential risk estimates (whether or not you use the template I link to for that), feel free to "comment" a link to them in the main database. And if you do so, perhaps consider noting somewhere how you came up with those estimates, which prior estimates you believe you'd seen already, whether you had those in mind when you made your estimates, etc.
It seems to me that it could be valuable to pool together new estimates from the "general EA public" and track how "independent" they seem likely to be (though I'm unsure about how valuable that'd be).
The relevant questions I'm aware of are this and these. So I guess I'd encourage people to answer those as well. Those questions are also linked to in the database already, so answers there would contribute to the pool here. And I believe people could comment there to explain their reasoning, what estimates they'd previously seen, etc.
There are more questions I'd be interested in estimates for which aren't on Metaculus at the moment (e.g., 100% rather than >95%, dystopia rather than other risks). But I imagine they could easily be put up by someone. (I'm not very familiar with the platform yet.)
If I wrote this post again, I would probably include brief discussion somewhere of hedge drift [LW · GW], and perhaps memetic downside risk [LW · GW]. I think one downside of this database is that it could induce those things.
Quoting from the first link, hedge drift refers to cases where:
The originator of [a] claim adds a number of caveats and hedges to their claim [e.g., “somewhat”, qualifications of where the claim doesn’t hold], which makes it more defensible, but less striking and sometimes also less interesting. When others refer to the same claim, the caveats and hedges gradually disappear, however.
Relatedly and somewhat unfortunately, this database draws more attention to the numbers people gave than to the precise phrasing of their statements and (especially) the context surrounding those statements.
There's currently an active thread on LessWrong [LW · GW] for making forecasts of existential risk across the time period from now to 2120. Activity on that thread is already looking interesting, and I'm hoping more people add their forecasts, their reasoning, and questions/comments to probe and improve other people's reasoning.
I plan to later somehow add to this database those forecasts, an aggregate of them, and/or links.
Since I'm now linking to this post decently often, I'll mention a couple more sources that are somewhat relevant to the pros and cons of explicit, quantitative estimates, and best practices for making them. (But note that these sources aren't about existential risk estimates in particular.)
Your estimates are presented as numerical values similar to probabilities. Is it actually probabilities and if yes, are they frequentist probabilities or Bayesian? And more generally: How we can define the "probability of end of the world"?
I believe that all the numbers I've shown were probabilities. I'm pretty sure they were always presented in the original source as percentages, decimals, or "1 in [number]".
What was the alternative you had in mind? I've seen some related estimates presented like "This can be expected to happen every x years", or "There's an x year return rate". Is that what you were thinking of? Several such estimates are given in Beard et al.'s appendix. But I don't think any are in the database. That wasn't primarily because they not quite probabilities (or not quite the right type), but rather because they were typically of things like the chance of an asteroid impact of a certain size, rather than direct estimates of the chance of existential catastrophe. (It's possible the asteroid impact wouldn't cause such a catastrophe.)
As for whether the probabilities are frequentist or Bayesian, I think many sources weren't explicit about that. But I generally assumed they were meant as Bayesian, though they might have been based on frequentist probabilities. E.g., Toby Ord's estimates of natural risks seems to be based on the frequency in the past, but then they're explicitly about what'll happen in the next 100 years, and they're modified based on knowledge of whether our other knowledge suggests this period is more or less likely than average to have e.g. an asteroid impact. But to be certain how a given number was meant to be interpreted, one might have to check the original source (which I provide links or references to in the database).
In the main sheet, the estimates are all unconditional (unless I made mistakes). They're just people's estimates of the probabilities that things will actually occur. There's a separate sheet for conditional estimates.
So presumably people's estimates of the chances these catastrophes occur would be lower conditional on people put in unexpectedly much effort to solve the problems.
Also, here's a relevant quote from The Precipice, which helps contextualise Ord's estimates. He writes that his estimates already:
incorporate the possibility that we get our act together and start taking these risks very seriously. Future risks are often estimated with an assumption of ‘business as usual’: that our levels of concern and resources devoted to addressing the risks stay where they are today. If I had assumed business as usual, my risk estimates would have been substantially higher. But I think they would have been misleading, overstating the chance that we actually suffer an existential catastrophe. So instead, I’ve made allowances for the fact that we will likely respond to the escalating risks, with substantial efforts to reduce them.
The numbers therefore represent my actual best guesses of the chance the threats materialise, taking our responses into account. If we outperform my expectations, we could bring the remaining risk down below these estimates. Perhaps one could say that we were heading towards Russian roulette with two bullets in the gun, but that I think we will remove one of these before it’s time to pull the trigger. And there might just be time to remove the last one too, if we really try.