I think he quit his PhD actually. So you could ask him why, and what factors people should consider when choosing to do a PhD or deciding to change while on it.
<Before that he did a PhD in the Philosophy of Machine Learning at Cambridge, on the topic of "to what extent is the development of artificial intelligence analogous to the biological and cultural evolution of human intelligence?">
I'm very pro framing this as an externality. Doesn't just help with left-leaning people, it can also be helpful for talking to other audiences, such as those immersed in economics or antitrust/competition law.
I actually think the EA world has been pretty good epistemically on winter: appropriately humble and exploratory, mostly funding research to work out how big a problem it is, not basing big claims on (possibly) unsettled science. The argument for serious action on reducing nuclear risk doesn't rely on claims about nuclear winter - though nuclear winter would really underline its importance. The Rethink Priorities report you critique talks at length about the debate over winter, which is great. See also 80,000 Hours profile, which is similarly cautious/hedged.
The EA world has been the major recent funder of research on nuclear winter: OpenPhil in 2017, 2020, perhaps Longview, and soon FLI. The research has advanced considerably since 2016. Indeed, most of the research ever published on nuclear winter has been published in the last few years, using the latest climate modelling. The most recent papers are getting published in Nature. I would disagree that theres a "reliance on papers that have a number of obvious flaws".
So as I see it the main phenomenon is that there's just much more being posted on the forum. I think there's two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there's lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc 'Fundamentals' courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I'm not so sure that the quality of the best posts has gone down, and I'm not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is "signal" - producing really high-quality content - and its certainly got harder to find that. But another purpose is more instrumental - its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research - where the benefit is more that they see whether its a fit for them or not, but they wouldn't actually do the work if it wasn't structured towards writing something public.
So the main thing that those of us who are looking for "signal" need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
This is really great work! Very clearly structured and written, persuasively argued and (fairly) well supported by the evidence.
I’m currently doing my PhD/DPhil on the history of arms control agreements, and 1972 is one of my four case-studies. So obviously Ithink its really important and interesting, and that more people should know about it – and I have a lot of views on the subject! So I’ve got a few thoughts on methodology, further literature and possible extensions which I’ll share below. But they’re all to adding to what is excellent work.
Its a bit unclear to me what your claim is for the link between these Track II discussions and the ultimate outcome of the two 1972 agreements. Its not that they were sufficient (needed SALT negotiations, and even then needed Kissinger/Dobrynin backchannel). Is it that the discussions were necessary for the outcome? Or just that they contributed in a positive way? I would be interested in your view.
The limitations section is good. But I think you could have been even clearer on the limits and strengths of a ‘single N’ approach. The limits are how much this can be generalised to the entire ‘universe of cases’. However, single N also has strengths - its most useful for developing and exploring mechanisms. So I think you could frame your contribution as exploring and deepening an analysis of the mechanisms. For example, something like "Two main mechanisms are proposed in the literature, this case study provides strong evidence for mechanism 1 (conveying new conceptions/ideas) and demonstrates how it works".
On another point, I'd be concerned that if you chose this case because it was one of the most successful Track II cases you'd be ‘selecting on the dependent variable’ (apologies for the political science jargon – something like “cherrypicked for having a particular outcome”) . Can you justify your motivation and case-selection differently, for example as one of (the?) biggest and most sustained Track 2 dialogues? e.g. you say: “when the first Pugwash conference happened in 1957, there were either no, or almost no, other opportunities for Soviet and American scientists to have conversations about security policy and nuclear issues”
Adler + Schelling are great on the US side of the story. I assume you would be familiar with them, but I don’t see them cited. If you haven’t read them, you’re in for a treat – they’re great, and largely agree with you.
If you want to go down a tangent, you might want to engage with new line of argument that many US nuclear policymakers never accepted the parity of MAD, but continued seeking advantage (Green and Long 2017; Green 2020; Lieber and Press 2006, 2020; Long and Green 2015).
As a sidenote, I’m curious why so much of the research on the two nuclear 1972 agreements focusses on ABM. ABM is the more intellectually interesting and counterintuitive. But its not clear to me it was *more important* then the limits on offensive weapons though.
Next steps/possible extensions
My impression is your main audiences are funders (and to a lesser extent general researchers and activists) within GCR. However if you wanted to adapt it, this very plausibly could be a paper. Its already a paper length, ~8,000 words. If you wanted to go down that route, there's a few things I'd do:
I’d cut most of the personal best guesses (“it seems likely to me” etc).
I think the notes are really great and interesting! If you incorporated some of them in the text of the piece you could deepen some of your claims in section 4, slim down the other sections.
Have a paragraph or two placing this piece within wider IR theoretical debates on constructivism, epistemic communities, going against systemic theories to open the 'black box' of the unitary state, etc.
If you wanted to continue this research, you could contrast this case with a similar conference and see what the difference in outcomes was; or try and draw up a list of the whole universe of cases (all major Track II dialogues).
Hmm I strongly read it as focussed on magnitude 7. Eg In the paper they focus on magnitude 7 eruptions, and the 1/6 this century probability: "The last magnitude-7 event was in Tambora, Indonesia, in 1815." / "Given the estimated recurrence rate for a magnitude-7 event, this equates to more than US$1 billion per year." This would be corroborated by their thread, Forum post, and previouswork, which emphasise 7 & 1/6.
Sorry to be annoying/pedantic about this. I'm being pernickety as I view a key thrust of their research as distinguishing 7 from 8. We can't just group magnitude 7 (1/6 chance) along with magnitude 8 and write them off as a teeny 1/14,000 chance. We need to distinguish 7 from 8, consider their severity/probability seperately, and prioritise them differently.
"Michael Cassidy and Lara Mani warn about the risk from huge volcanic eruptions. Humanity devotes significant resources to managing risk from asteroids, and yet very little into risk from supervolcanic eruptions, despite these being substantially more likely. The absolute numbers are nonetheless low; super-eruptions are expected roughly once every 14,000 years. Interventions proposed by the authors include better monitoring of eruptions, investments in preparedness, and research into geoengineering to mitigate the climatic impacts of large eruptions or (most speculatively) into ways of intervening on volcanoes directly to prevent eruptions."
However, their Nature paper is about magnitude 7 eruptions, which may have a probability this century of 1/6, not supervolcanic eruptions (magnitude 8), which as you point out have a much lower probability.
I think its a fascinating paper that in a prominent, rigorous and novel way applies importance/neglectedness/tractability to a comparison of two hazards:
"Over the next century, large-scale volcanic eruptions are hundreds of times more likely to occur than are asteroid and comet impacts, put together. The climatic impact of these events is comparable, yet the response is vastly different. ‘Planetary defence’ receives hundreds of millions of dollars in funding each year, and has several global agencies devoted to it. [...] By contrast, there is no coordinated action, nor large-scale investment, to mitigate the global effects of large-magnitude eruptions. This needs to change."
Yep totally fair point, my examples were about pieces. However, note that the quote you pulled out referred to 'good work in the segments' (though this is quite a squirmy lawyerly point for me to make). Also, interestingly 2019-era Will was a bit more skeptical of xrisk - or at least wrote a piece exploring that view.
I'm a bit wary of naming specific people whose views I know personally but haven't expressed them publicly, so I'll just give some orgs who mostly work in those two segments, if you don't mind:
'Long-term + EA': the APPG for Future Generations does a lot of work here, and I'd add Tyler John's work (here & here), plausibly Beckstead's thesis.
'Xrisk + EA': my impression is that some of the more normy groups OpenPhil have funded are here, working with the EA community on xrisk topics, but not necessarily buying longtermism.
I agree that Effective Altruism and the existential risk prevention movement are not the same thing. Let me use this as an opportunity to trot out my Venn diagrams again. The point is that these communities and ideas overlap but don't necessarily imply each other - you don't have to agree to all of them because you agree with one of them, and there are good people doing good work in all the segments.
You're quite right that if this post were arguing that there is an overall pattern, it would quite clearly be inadequate. It doesn't define the universe of cases or make clear how representative these cases are of that universe, the two main studies could be criticised for selecting on the dependent variable, and its based primarily on quotes from two books.
However, I didn't set out to answer something like the research question "which is more common in 20th century history, mistakenly sprinting or mistakenly failing to sprint?" - though I think that's a very interesting question, and would like someone to look into it!
My intention for this blog post was for it to be fairly clear and memorable, aimed at a general audience - especially perhaps a machine learning researcher who doesn't know much about history. The main takeaway I wanted wasn't for people to think "this is the most common/likely outcome" but rather to add a historic example to their repertoire that they can refer to - "this was an outcome". It was supposed to be a cautionary tale, a prompt to people to think not "all sprints are wrong" but rather "wait am I in an Ellsberg situation?" - and if so to have some general, sensible recommendations and questions to ask.
My aim was to express a worry ("be careful about mistaken sprints") and illustrate that with two clear, memorable stories. There's a reasonable scenario in the next few decades that we're in a situation where we feel we need to back a sprint, prompted by concern about another group/country's sprint. If we do, and I'm not around to say "hey lets be careful about this and check we're actually in a race" then I hope these two case studies may stick in someone's mind and lead them to say "OK but lets just check, don't want to make the same mistake as Szilard and Ellsberg..."
If these experts regularly have a large impact on these decisions, that's an argument for transparency about them. This is a factor that could of course be outweighed by other considerations (ability to give frank advice, confidentiality, etc). Perhaps might be worth asking them how they'd feel about being named (with no pressure attached, obviously).
Also, can one volunteer as an expert? I would - and I imagine others (just on this post, perhaps Ian and Sam?) would too.
"The implication of these historical outcomes is that in order to reliably affect decision-making, you must yourself be the decision-maker. Prestige, access to decision-makers, relevant expertise, and cogent reasoning are not sufficient; even with all these you are liable to be ignored. By understanding the complex workings of decision-making at the highest levels, you can improve your chances of influencing outcomes in the way you desire, but even if you understand how the game is played, you are ultimately subject to the judgment of those who wield power, and this judgment can be frustratingly capricious. Without even such an understanding, you stand little or no chance whatsoever. "
Nevertheless, I think this claim is overconfident and unfounded. We can't just generalise from one case to the entire universe of cases! A more accurate assessment needs to reckon with the success of the nuclear and biological weapons arms control epistemic community in the early 1970s (such as Kissinger and Meselson) - as well as the many other examples of scientific advisers being influential.
It's really important that there is public, good-faith, well-reasoned critique of this important chapter in a central book in the field. You raise some excellent points that I'd love to see Ord (and/or others) respond to. Congratulations on your work, and thank you!
There has been sustained activism from the AI community to emphasise that AI should be developed and deployed in a safe and beneficial manner. This has involved Open Letters, AI principles, the establishment of new centres, and influencing governments.
The Puerto Rico Conference in January 2015 was a landmark event to promote the beneficial and safe development of AI. It led to an Open Letter signed by over 8,000 people calling for the safe and beneficial development of AI, and a research agenda to that end . The Asilomar Conference in January 2017 led to the Asilomar AI Principles, signed by several thousand AI researchers . Over a dozen sets of principles from a range of groups followed .
The AI community has established several research groups to understand and shape the societal impact of AI. AI conferences have also expanded their work to consider the impact of AI. New groups include:
OpenAI (December 2015)
Centre for Human-Compatible AI (August 2016)
Leverhulme Centre for the Future of Intelligence (October 2016)3
DeepMind Ethics and Society (October 2017)
UK Government’s Centre for Data Ethics and Innovation (November 2017)"
Great post! Mass extinctions and historical societal collapses are important data sources - I would also suggest ecological regime shifts. My main takeaway is actually about multicausality: several ‘external’ shocks typically occur in a similar period. ‘Internal’ factors matter too - very similar shocks can affect societies very differently depending on their internal structure and leadership. When complex adaptive systems shift equilibria, several causes are normally at play.
Myself, Luke Kemp and Anders Sandberg (and many others!) have three seperate chapters touching on these topics in a forthcoming book on 'Historical Systemic Collapse' edited by Princeton's Miguel Centeno et al . Hopefully coming out this year.
Thanks for this. I'm more counselling "be careful about secrecy" rather than "don't be secret". Especially be careful about secret sprints, being told you're in a race but can't see the secret information why, and careful about "you have to take part in this secret project".
On the capability side, the shift in AI/ML publication and release norms towards staged release (not releasing full model immediately but carefully checking for misuse potential first), structured access (through APIs) and so on has been positive, I think.
On the risks/analysis side, MIRI have their own “nondisclosed-by-default” policy on publication. CSER and other academic research groups tend towards more of a "disclosed-by-default” policy.
Recommender systems are a great example of a broader concern. Another is lethal autonomous weapons, where a big focus is "meaningful human control". Automation bias is an issue even up to the nuclear level - the concern is that people will more blindly trust ML systems, and won't disbelieve them as people did in several Cold War close calls (eg Petrov not believing his computer warning of an attack). See Autonomy and machine learning at the interface of nuclear weapons, computers and people.
"And it was just a lot better for Boeing and Lockheed and Northrop Grumman and General Dynamics to go that way than not to have them, then they wouldn’t be selling the weapons. And by the way what I’ve learned just recently by books like … A guys named Kofsky wrote a book called Harry Truman And The War Scare of 1947.
Reveals that at the end of the war, Ford and GM who had made most of our bombers went back to making cars very profitably. But Boeing and Lockheed didn’t make products for the commercial market, only for commercial air except there wasn’t a big enough market to keep them from bankruptcy. They had suddenly lost their vast orders for military planes in mid 1945. The only way they could avoid bankruptcy was to sell a lot of planes to the government, military planes. But against who? Not Germany we were occupying Germany, not Japan we were occupying Japan. Who was our enemy that you needed a lot of planes against. Well Russia had been our ally during the war, but Russia had enough targets to justify, so they had to be an enemy and they had to be the enemy, and we went off from there.
I would say that having read that book and a few others I could say, I now see since my book was written nine months ago, that the Cold War was a marketing campaign for selling war planes to the government and to our allies. It was a marketing campaign for annual subsidies to the aerospace industry, and the electronics industry. And also the basis for a protection racket for Europe, that kept us as a major European power. Strictly speaking we’re not a European power. But we are in effect because we provide their protection against Russia the super enemy with nuclear weapons, and for that purpose it’s better for the Russians to have ICBM, and missiles, and H-bombs, as an enemy we can prepare against. It’s the preparations that are profitable. All wars have been very profitable for the arms manufacturers, nuclear war will not be, but preparation for it is very profitable, and therefore we have to be prepared."
Hi, yes good question, and one that has been much discussed - here's three papers on the topic. I'm personally of the view that there shouldn't really be much conflict/contradictions - we're all pushing for the safe, beneficial and responsible development and deployment of AI, and there's lots of common ground.
Thanks Rohin. Yes I should perhaps have spelled this out more. I was thinking about two things - focussed on those two stages of advocacy and participation.
1. Don't just get swept up in race rhetoric and join the advocacy: "oh there's nothing we can do to prevent this, we may as well just join and be loud advocates so we have some chance to shape it". Well no, whether a sprint occurs is not just in the hands of politicians and the military, but also to a large extent in the hands of scientists. Scientists have proven crucial to advocacy for, and participation in, sprints. Don't give up your power too easily.
2. You don't have to stay if it turns out you're not actually in a race and you don't have any influence on the sprint program. There were several times in 1945 when it seems to me that scientists gave up their power too easily - over when and how the bomb was used, and what information was given to the US public. Its striking that Rotblat was the only one to resign - and he was leant on to keep his real reasons secret.
One can also see this later in 1949 and the decision to go for the thermonuclear bomb. Oppenheimer, Conant, Fermi and Bethe all strongly opposed that second 'sprint' ("It is neccessarily an evil thing considerd in any light."). They were overruled, and yet continued to actively participate in the program. The only person to leave the program (Ellsberg thinks, p.291-296) was Ellsberg's own father, a factory designer - who also kept it secret.
Exit or the threat of exit can be a powerful way to shape outcomes - I discuss this further in Activism by the AI Community. Don't give up your power too easily.
Thanks Pablo for those thoughts and the link - very interesting to read in his own words.
I completely agree that stopping a 'sprint' project is very hard - probably harder than not beginning one. The US didn't slow down on ICBMs in 1960-2 either.
We can see some of the mechanisms by which this occurs around biological weapons programs. Nixon unilaterally ended the US one; Brezhnev increased the size of the secret Soviet one. So in the USSR there was a big political/military/industrial complex with a stake in the growth of the program and substantial lobbying power, and it shaped Soviet perceptions of 'sunk costs', precedent, doctrine, strategic need for a weapons technology, identities and norms; while in the US the oppossite occured.
I don't think its a hole at all, I think its quite reasonable to focus on major states. The private sector approach is a different one with a whole different set of actors/interventions/literature - completely makes sense that its outside the scope of this report. I was just doing classic whatabouterism, wondering about your take on a related but seperate approach.
Btw I completely agree with you about cluster munitions.
Great report! Looking forward to digging into it more.
It definitely makes sense to focus on (major) states. However a different intervention I don't think I saw in the piece is about targeting the private sector - those actually developing the tech. E.g. Reprogramming war by Pax for Peace, a Dutch NGO. They describe the project as follows:
"This is part of the PAX project aimed at dissuading the private sector from contributing to the development of lethal autonomous weapons. These weapons pose a serious threat to international peace and security, and would violate fundamental legal and ethical principles. PAX aims to engage with the private sector to help prevent lethal autonomous weapons from becoming a reality. In a series of four reports we look into which actors could potentially be involved in the development of these weapons. Each report looks at a different group of actors, namely states, the tech sector, universities & research institutes, and arms producers. This project is aimed at creating awareness in the private sector about the concerns related to lethal autonomous weapons, and at working with private sector actors to develop guidelines and regulations to ensure their work does not contribute to the development of these weapons."
It follows fairly successful investor campaigns on e.g. cluster munitions. This project could form the basis for shareholder activism or divestment by investors, and/or wider activism by the AI community by students, researchers, employees, etc - building on eg FLI's "we won't work on LAWS" pledge.
I'd be interested in your views on that kind of approach.
Thanks for these questions! I tried to answer your first in my reply to Christian.
On your second, "delaying development" makes it sound like the natural outcome/null hypothesis is a sprint - but its remarkable how the more 'natural' outcome was to not sprint, and how much effort it took to make the US sprint.
To get initial interest at the beginning of the war required lots of advocacy from top scientists, like Einstein. Even then, the USA didn't really do anything from 1939 until 1941, when an Australian scientist went to the USA, persuaded US scientists and promised that Britain would share all its research and resources. Britain was later cut out by the Americans, and didn't have a serious independent program for the rest of the war. Germany considered it in the early war, but decided against in 1942. During the war, neither the USSR nor Japan had serious programs (and France was collaborating with Germany). All four major states (UK, Germany, USSR, Japan) realised it would cost a huge amount in terms of money, people and scarce resources like iron, and probably not come in time to affect the course of the war.
The counterfactual is just "The US acts like the other major powers of the time and decides not to launch a sprint program that costs 0.4% of GDP during a total war, and that probably won't affect who wins the war".
Thanks for the kind words Christian - I'm looking forward to reading that report, it sounds fascinating.
I agree with your first point - I say "They were arguably right, ex ante, to advocate for and participate in a project to deter the Nazi use of nuclear weapons." Actions in 1939-42 or around 1957-1959 are defensible. However, I think this highlights 1) accurate information in 1942-3 (and 1957) would have been useful and 2) when they found out the accurate information (in 1944 and 1961) , its very interesting that it didn't stop the arms buildup.
The question of whether over, under or calibrated confidence is more common is an interesting one that I'd like someone to research. It perhaps could be usefully narrowed to WWII & postwar USA. I offered some short examples, but this could easily be a paper. There are some theoretical reasons to expect overconfidence, I'd think: such as paranoia and risk-aversion, or political economy incentives for the military-industrial complex to overemphasise risk (to get funding). But yes, an interesting open empirical question.
On the second point, what about climate change in India-Pakistan? e.g. an event worse than the current terrible heatwave - heat stress and agriculture/economic shock leads to migration, instability, rise in tension and accidental use of nuclear weapons. The recent modelling papers indicate that would lead to 'nuclear autumn' and probably be a global catastrophe.
Note that "humanity is doomed" is not the same as 'direct extinction', as there are many other ways for us to waste our potential.
I think its an interesting argument, but I'm unsure that we can get to a rigorous, defensible distinction between 'direct' and 'indirect' risks. I'm also unsure how this framework fits with the "risk/risk factor" framework, or the 'hazard/vulnerability/exposure' framework that's common across disaster risk reduction, business + govt planning, etc. I'd be interested in hearing more in favour of this view, and in favour of the 2 claims I picked out above.
We've talked about this before, but in general I've got such uncertainty about the state of our knowledge and the future of the world that I incline towards grouping together nuclear, bio and climate as being in roughly the same scale/importance 'tier' and then spending most of our focus seeing if any particular research strand or intervention is neglected and solvable (e.g. your work flagging something underexplored like cement).
On your food production point, as I understand it the issue is more shocks than averages. Food system shocks that can lead to "economic shocks, socio-political instability as well as starvation, migration and conflict" (from the 'causal loop diagram' paper). However, I'm not a food systems expert, I'd suggest the best people to discuss this with more are our Catherine Richards and Asaf Tzachor, authors of e.g. Future Foods For Risk-Resilient Diets.
Seems a bit odd to me to just post the 'direct extinction' bit, as essentially no serious researcher argues that there is a significant chance that climate change could 'directly' (and we can debate what that means) cause extinction. However, maybe this view is more widespread amongst the general public (and therefore worth responding to)?
On 'indirect risk', I'd be interested in hearing more on these two claims:
"it's less important to reduce upstream issues that could be making them worse vs trying to fix them directly" (footnote 25); and
"our guess is that [climate change's 'indirect'] contribution to other existential risks is at most an order of magnitude higher — so something like 1 in 1,000" - which "still seems more than 10 times less likely to cause extinction than nuclear war or pandemics."
If people are interested in reading more about climate change as a contributor to GCR, here are two CSER papers from last year (and we have a big one coming out soon)
Thanks for this Jeffrey and Lennart! Very interesting, and I broadly agree. Good area for people to gain skills/expertise, and private companies should beef up their infosec to make it harder for them to be hacked and stop some adversaries.
However, I think its worth being humble/realistic. IMO a small/medium tech company (even Big Tech themselves) are not going to be able to stop a motivated state-linked actor from the P5. Would you broadly agree?