Posts
Comments
I guess I would think that if one wants to argue for democracy as an intrinsic good, that would get you global democracy (and global control of EA funds), and it's practical and instrumental considerations (which, anyway, are all the considerations in my view) that bite against it.
Yes, that seems right.
It seems like the critics would claim that EA is, if not coercing or subjugating, at least substantially influencing something like the world population in a way that meets the criteria for democratisation. This seems to be the claim in arguments about billionaire philanthropy, for example. I'm not defending or vouching for that claim, but I think whether we are in a sufficiently different situation may be contentious.
Well I think MIC relies on some sort of discontinuity this century, and when we start getting into the range of precedented growth rates, the discontinuity looks less likely.
But we might not be disagreeing much here. It seems like a plausibly important update, but I'm not sure how large.
This is a valuable point, but I do think that giving real weight to a world where we have neither extinction nor 30% growth would still be an update to important views about superhuman AI. It seems like evidence against the Most Important Century thesis, for example.
It might be challenging to borrow (though I'm not sure), but there seem to be plenty of sophisticated entities that should be selling off their bonds and aren't. The top-level comment does cut into the gains from shorting (as the OP concedes), but I think it's right that there are borrowing-esque things to do.
I'm trying to make sure I understand: Is this (a more colorful version) of the same point as the OP makes at the end of "Bet on real rates rising"?
The other risk that could motivate not making this bet is the risk that the market – for some unspecified reason – never has a chance to correct, because (1) transformative AI ends up unaligned and (2) humanity’s conversion into paperclips occurs overnight. This would prevent the market from ever “waking up”.
However, to be clear, expecting this specific scenario requires both:
- Buying into specific stories about how takeoff will occur: specifically, Yudkowskian foom-type scenarios with fast takeoff.
- Having a lot of skepticism about the optimization forces pushing financial markets towards informational efficiency.
You should be sure that your beliefs are actually congruent with these requirements, if you want to refuse to bet that real rates will rise. Additionally, we will see that the second suggestion in this section (“impatient philanthropy”) is not affected by the possibility of foom scenarios.
It doesn't seem all that relevant to me whether traders have a probability like that in their heads. Whether they have a low probability or are not thinking about it, they're approximately leaving money on the table in a short-timelines world, which should be surprising. People have a large incentive to hunt for important probabilities they're ignoring.
Of course, there are examples (cf. behavioral economics) of systemic biases in markets. But even within behavioral economics, it's fairly commonly known that it's hard to find ongoing, large-scale biases in financial markets.
Do you have a sense of whether the case is any stronger for specifically using cortical and pallial neurons? That's the approach Romain Espinosa takes in this paper, which is among the best work in economics on animal welfare.
My husband and I are planning to donate to Wild Animal Initiative and Animal Charity Evaluators; we've also supported a number of political candidates this year (not tax deductible) who share our values.
We've been donating to WAI for a while, as we think they have a thoughtful, skilled team tackling a problem with a sweeping scale and scant attention.
We also support ACE's work to evaluate and support effective ways to help animals. I'm on the board there, and we're excited about ACE's new approach to evaluations and trajectory for the coming years.
Yes, and thank you for the detailed private proposal you sent the research team. I didn't see it but heard about it, and it seems like it was a huge help and just a massive amount of volunteer labor. I know they really appreciated it.
I'm an ACE board member, so full disclosure on that, though what I say here is in my personal capacity.
I'm very glad about a number of improvements to the eval process that are not obvious from this post. In particular, there are now numeric cost-effectiveness ratings that I found clarifying, overall explanations for each recommendation, and clearer delineation of the roles the "programs" and "cost-effectiveness" sections play in the reviews. I expect these changes to make recommendations more scope sensitive. This leaves me grateful for and confident in the new review framework.
As I noted on the nuclear post, I believe this is based on a (loosely speaking) person-affecting view (mentioned in Joel and Ben's back-and-forth below). That seems likely to me to bias the cost-effectiveness downward.
Like Fin, I'm very surprised by how well this performs given takes in other places (e.g. The Precipice) on how asteroid prevention compares to other x-risk work.
Worth flagging that I believe this is based on a (loosely speaking) person-affecting view (mentioned in Joel and Ben's back-and-forth below). That seems to me to bias the cost-effectiveness of anything that poses a sizable extinction risk dramatically downward.
At the same time, I find both the empirical work and the inside-view thinking here very impressive for a week's work, and it seems like even those without a person-affecting view can learn a lot from this.
Thanks for writing this. I think about these sorts of things a lot. Given the title, do you know of examples of movements that did not start academic disciplines and appear to have suffered as a result?
The Global Priorities Institute and clusters of work around that do work in economics, including welfare economics. I'd also be curious to hear what you think they should do differently.
I'm toying with a project to gather reference classes for AGI-induced extinction and AGI takeover. If someone would like to collaborate, please get in touch.
(I'm aware of and giving thought to reference class tennis concerns but still think something like this is neglected.)
I don't think it's right that the broad project of alignment would look the same with and without considering religion. I'm curious what your reasoning is here and if I'm mistaken.
One way of reading this comment is that it's a semantic disagreement about what alignment means. The OP seems to be talking about the problem of getting an AI to do the right thing, writ large, which may encompass a broader set of topics than alignment research as you define it.
Two other ways of reading it are that (a) solving the problem the OP is addressing (getting an AI to do the right thing, writ large) does not depend on values, or (b) solving the alignment problem will necessarily solve the value problem. I don't entirely see how you can justify (a) without a claim like (b), though I'm curious if there's a way.
You might justify (b) via the argument that solving alignment involves coming up a way to extrapolate values. Perhaps it is irrelevant which particular person you start with, because the extrapolation process will end up at the same point. To me this seems quite dubious. We have no such method and observe deep disagreement in the world. Which methods we use to resolve disagreement and determine whose values we include seem to involve a question of values. And from my lay sense, the methods of alignment that are currently most-discussed involve aligning it with specific preferences.
One thing that's sad and perhaps not obvious to people is that, as I understand it, Nathan Robinson was initially sympathetic to EA (and this played a role in at-times vocal advocacy for animals). I don't know that there's much to be done about this. I think the course of events was perhaps inevitable, but that's relevant context for other Forum readers who see this.
And worth noting that Ben Franklin was involved in the constitution, so at least some of his longtermist time seems to have been well spent.
I don't have a strong view on the original setup, but I can clarify what the argument is. For the first point, that we maximize . The idea is that we want to maximize the likelihood that the organism chooses the action that leads to enjoyment (the one being selected for). That probability is a function of how much better it is to choose that action than the alternative. So if you get E from choosing that action and lose S from choosing the alternative, the benefit from choosing that action is E - (-S) = E + S. However, you only pay to produce the experience of the action you actually take. This last reason is why the costs are weighted by probability, while the benefits, which are only about the anticipation of the experience you would get conditional on your action, are not.
It occurs to me that a fuller model might endogenize n, i.e. be something like max P(E(C_E) + S(C_S)) s.t. P(.) C_E + (1 - P(.)) C_S = M. (Replacing n with 1 - P here so it's a rate, not a level. Also, perhaps this reduces to the same thing based on the envelope theorem.)
And on the last point, that point is relevant for the interpretation of the model (e.g. choosing the value of n), but it is not an assumption of the model.
Like others, I really appreciate these thoughts, and it resonates with me quite a lot. At this point, I think the biggest potential failure mode for EA is too much drift in this direction. I think the "EA needs megaprojects" thing has generated a view that the more we spend, the better, which we need to temper. Given all the resources, there's a good chance EA is around for a while and quite large and powerful. We need to make sure we put these tools to good use and retain the right values.
EA spending is often perceived as wasteful and self-serving
It's interesting here how far this is from the original version of EA and its criticisms; e.g. that EA was an unrealistic standard that involved sacrificing one's identity and sense of companionship for an ascetic universalism.
I think the old perception is likely still more common, but it's probably a matter of time (which means there's likely still time to change it). And I think you described the tensions brilliantly.
Yes, that's an accurate characterization of my suggestion. Re: digital sentience, intuitively something in the 80-90% range?
Yes, all those first points make sense. I did want to just point to where I see the most likely cruxes.
Re: neuron count, the idea would be to use various transformations of neuron counts, or of a particular type of neuron. I think it's a judgment call whether to leave it to the readers to judge; I would prefer giving what one thinks is the most plausible benchmark way of counting and then giving the tools to adjust from there, but your approach is sensible too.
Thanks for writing this post. I have similar concerns and am glad to see this composed. I particularly like the note about the initial design of space colonies. A couple things:
- My sense is that the dominance of digital minds (which you mention as a possible issue) is actually the main reason many longtermists think factory farming is likely to be small relative to the size of the future. You're right to note that this means future human welfare is also relatively unimportant, and my sense is that most would admit that. Humanity is instrumentally important, however, since it will create those digital minds. I do think it's an issue that a lot of discussion of the future treats it as the future "of humanity" when that's not really what it's about. I suspect that part of this is just a matter of avoiding overly weird messaging.
- It would be good to explore how your argument changes when you weight animals in different ways, e.g. by neuron count, since that [does appear to change things](https://forum.effectivealtruism.org/posts/NfkEqssr7qDazTquW/the-expected-value-of-extinction-risk-reduction-is-positive). I think we should probably take a variety of approaches and place some weight on each, although there's a sort of Pascalian problem with considering the possibility that each animal mind has equal weight in that it feels somewhat plausible but also leads to wild and seemingly wrong conclusions (e.g. that it's all about insect larvae). But in general, this seems like a central issue worth adjusting for.
Research institute focused on civilizational lock-in
Values and Reflective Processes, Economic Growth, Space Governance, Effective Altruism
One source of long-term risks and potential levers to positively shape the future is the possibility that certain values or social structures get locked in, such as via global totalitarianism, self-replicating colonies, or widespread dominance of a single set of values. Though organizations exist dedicated to work on risks of human extinction, we would like to see an academic or independent institute focused on other events that could have an impact on the order of millions of years or more. Are such events plausible, and which ones should be of most interest and concern? Such an institute might be similar in structure to FHI, GPI, or CSER, drawing on the social sciences, history, philosophy, and mathematics.
Consulting on best practices around info hazards
Epistemic Institutions, Effective Altruism, Research That Can Help Us Improve
Information about ways to influence the long-term future can in some cases give rise to information hazards, where true information can cause harm. Typical examples concern research into existential risks, such as around potential powerful weapons or algorithms prone to misuse. Other risks exist, however, and may also be especially important for longtermists. For example, better understanding of ways social structures and values can get locked in may help powerful actors achieve deeply misguided objectives.
We would like to support an organization that can develop a set of best practices and consult with important institutions, companies, and longtermist organizations on how best to manage information hazards. We would like to see work to help organizations think about the tradeoffs in sharing information. How common are info hazards? Are there ways to eliminate or minimize downsides? Is it typically the case that the downsides to information sharing are much smaller than upsides or vice versa?
Advocacy for digital minds
Artificial Intelligence, Values and Reflective Processes, Effective Altruism
Digital sentience is likely to be widespread in the most important future scenarios. It may be possible to shape the development and deployment of artificially sentient beings in various ways, e.g. through corporate outreach and lobbying. For example, constitutions can be drafted or revised to grant personhood on the basis of sentience; corporate charters can include responsibilities to sentient subroutines; and laws regarding safe artificial intelligence can be tailored to consider the interests of a sentient system. We would like to see an organization dedicated to identifying and pursuing opportunities to protect the interests of digital minds. There could be one or multiple organizations. We expect foundational research to be crucial here; a successful effort would hinge on thorough research into potential policies and the best ways of identifying digital suffering.
Lobbying architects of the future
Values and Reflective Processes, Effective Altruism
Advocacy often focuses on changing politics, but the most important decisions about the future of civilization may be made in domains that receive relatively less attention. Examples include the reward functions of generally intelligent algorithms that eventually get scaled up, the design of the first space colonies, and the structure of virtual reality. We would like to see one or more organizations focused on getting the right values considered by influential decision-makers at institutions like NASA and Google. We would be excited about targeted outreach to promote consideration of aligned artificial intelligence, existential risks, the interests of future generations, and nonhuman (both animal and digital) minds. The nature of this work could take various forms, but some potential strategies are prestigious conferences in important industries, retreats including a small number of highly-influential professionals, or shareholder activism.
Lobbying architects of the future
Moral Advocacy Aimed at Those Designing Critical Institutions and Processes
Advocacy often focuses on changing politics, but the most important decisions about the future of civilization may be made in domains that receive relatively less attention. Examples include the reward functions of generally intelligent algorithms that eventually get scaled up, the design of the first space colonies, and the structure of virtual reality. We would like to see one or more organizations focused on getting the right values considered by influential decision-makers at institutions like NASA and Google. We would be excited about targeted outreach to promote consideration of aligned artificial intelligence, existential risks, the interests of future generations, and nonhuman (both animal and digital) minds. The nature of this work could take various forms, but some potential strategies are prestigious conferences in important industries, retreats including a small number of highly-influential professionals, or shareholder activism.
Yeah, I think this would be good context—the CO gov's husband is a die-hard animal rights activist and seems to have influence: https://en.wikipedia.org/wiki/Marlon_Reis
He declared a "MeatOut" day recently to support plant-based eating and has signed various animal welfare initiatives into law, such as a cage-free law.
So it seems that someone very EA-minded could get this position if they apply.
I'm really excited to see this and look into it. I'm working on some long-term persistence issues, and this is largely in line with my intuitive feel for the literature. I haven't looked at the Church-WEIRDness one, though, and now I'm eager to read that one.
Let me note that on top of all your concrete accomplishments, you're just a very sweet and caring person, which has got to help a lot in building this vibrant community. I'm happy to know you!
There is nothing special about longtermism compared to any other big desideratum in this regard.
I'm not sure this is the case. E.g. Steven Pinker in Better Angels makes the case that utopian movements systematically tend to commit atrocities because this all-important end goal justifies anyting in the medium term. I haven't rigorously examined this argument and think it would be valuable for someone to do so, but much of longtermism in the EA community, especially of the strong variety, is based on something like utopia.
One reason why you might intuitively think there would be a relationship is that shorter-term impacts are typically somewhat more bounded; e.g. if thousands of American schoolchildren are getting suboptimal lunches, this obviously doesn't justify torturing hundreds of thousands of people. With the strong longtermist claims it's much less clear that there's any sort of upper bound, so to draw a firm line against atrocities you end up looking to somewhat more convoluted reasoning (e.g. some notion of deontological restraint that isn't completely absolute but yet can withstand astronomical consequences, or a sketchy and loose notion that atrocities have an instrumental downside).
I think the persistence studies stuff is the best bet. One thing to note there is that the literature is sort of a set of existence proofs. It shows that there are various things that have long-term impacts, but it might not give you a strong sense of the average long-term impact of poverty alleviation.
This is really impressive work. I've been looking for something like this to cite for economics work on animal welfare, and this seems well-suited for that.
I just wanted to give major kudos for evaluating a prediction you made and very publicly sharing the results even though they were not fully in line with your prediction.
Thanks. I'm aware of this sort of argument, though I think most of what's out there relies on anecdotes, and it's unclear exactly what the effect is (since there is likely some level of confounding here).
I guess there are still two things holding me up here. (1) It's not clear that the media is changing preferences or just offering [mis/dis]information. (2) I'm not sure it's a small leap. News channels' effects on preferences likely involve prolonged exposure, not a one-time sitting. For an algorithm to expose someone in a prolonged way, it has to either repeatedly recommend videos or recommend one video that leads to their watching many, many videos. The latter strikes me as unlikely; again, behavior is malleable but not that malleable. In the former case, I would think the direct effect on the reward function of all of those individual videos recommended and clicked on has to be way larger than the effect on the person's behavior after seeing the videos. If my reasoning were wrong, I would find that quite scary, because it would be evidence of substantially greater vulnerability to current algorithms than I previously thought.
Right. I mean, I privilege this simpler explanation you mention. He seems to have reason to think it's not the right explanation, but I can't figure out why.
BTW, I am interested in studying this question if anyone is interested in partnering up. I'm not entirely sure how to study it, as (given the post) I suspect the result may be a null, which is only interesting if we have access to one of the algorithms he is talking about and data on the scale such an algorithm would typically have.
My general approach would be an online experiment where I expose one group of people to a recommender system and don't expose another. Then place both groups in the same environment and observe whether the first group is now more predictable. (This does not account for the issue of information, though.)
It seems that dystopian novels are overrepresented relative to their share of the classics. I'm curious for others' thoughts why that is. I could imagine a case that they're more action-relevant than, e.g., Pride and Prejudice, but I also wonder if they might shape our expectations of the future in bad ways. (I say this as someone currently rereading 1984, which I adore...)
Links should be fixed! Thanks for pointing this out.
Thanks for pointing this out. It should work now.
Sorry—you're right that this doesn't work. To clarify, I was thinking that the method of picking the color should be fixed ex-ante (e.g. "I pick red as the color with 50% probability"), but that doesn't do the trick because you need to pool the colors for ambiguity to arise.
The issue is that the problem the paper identifies does not come up in your example. If I'm offered the two bets simultaneously, then an ambiguity averse decision maker, like an EU decision maker, will take both bets. If I'm offered the bets sequentially without knowing I'll be offered both when I'm offered the first one, then neither an ambiguity-averse nor a risk-averse EU decision-maker will take them. The reason is that the first one offers the EU decision-maker a 50% chance of winning, so given risk-aversion its value is less than 50% of $1. So your example doesn't distinguish a risk-averse EU decision-maker from an ambiguity-averse one.
So I think unfortunately we need to go with the more complicated examples in the paper. They are obviously very theoretical. I think it could be a valuable project for someone to translate these into more practical settings to show how these problems can come up in a real-world sense.
Thanks! Helpful follow-ups.
On the first point, I think your intuition does capture the information aversion here, but I still think information aversion is an accurate description. Offered a bet that pays $X if I pick a color and then see if a random ball matches that color, you'll pay more than for a bet that pays $X if a random ball is red. The only difference between these situations is that you have more information in the latter: you know the color to match is red. That makes you less willing to pay. And there's no obvious reason why this information aversion would be something like a useful heuristic.
I don't quite get the second point. Commitment doesn't seem very relevant here since it's really just a difference in what you would pay for each situation. If one comes first, I don't see any reason why it would make sense to commit, so I don't think that strengthens the case for ambiguity aversion in any way. But I think I might be confused here.
Yeah, that's the part I'm referring to. I take his comment that expectations are not random variables to be criticizing taking expectations over expected utility with respect to uncertain probabilities.
I think the critical review of ambiguity aversion I linked to us sufficiently general that any alternatives to taking expectations with respect to uncertain probabilities will have seriously undesirable features.
Thanks for writing this. I think it's very valuable to be having this discussion. Longtermism is a novel, strange, and highly demanding idea, so it merits a great deal of scrutiny. That said, I agree with the thesis and don't currently find your objections against longtermism persuasive (although in one case I think they suggest a specific set of approaches to longtermism).
I'll start with the expected value argument, specifically the note that probabilities here are uncertain and therefore random valuables, whereas in traditional EU they're constant. To me a charitable version of Greaves and MacAskill's argument is that, taking the expectation over the probabilities times the outcomes, you have a large future in expectation. (What you need for the randomness of probabilities to sink longtermism is for the probabilities to correlate inversely and strongly with the size of the future.) I don't think they'd claim the probabilities are certain.
Maybe the claim you want to make, then, is that we should treat random probabilities differently from certain probabilities, i.e. you should not "take expectations" over probabilities in the way I've described. The problem with this is that (a) alternatives to taking expectations over probabilities have been explored in the literature, and they have a lot of undesirable features; and (b) alternatives to taking expectations over probabilities do not necessarily reject longtermism. I'll discuss (b), since it involves providing an example for (a).
(b) In economics at least, Gilboa and Schmeidler (1989) propose what's probably the best-known alternative to EU when the probabilities are uncertain, which involves maximizing expected utility for the prior according to which utility is the lowest, sort of a meta-level risk aversion. They prove that this is the optimal decision rule according to some remarkably weak assumptions. If you take this approach, it's far from clear you'll reject longtermism: more likely, you end up with a sort of longtermism focused on averting long-term suffering, i.e. focused on maximizing expected value according to the most pessimistic probabilities. There's a bunch of other approaches, but they tend to have similar flavors. So alternatives on EU may agree on longtermism and just disagree on the flavor of it.
(a) Moving away from EU leads to a lot of problems. As I'm sure you know given your technical background, EU derives from a really nice set of axioms (The Savage Axioms). Things go awry when you leave it. Al-Najjar and Weinstein (2009) offer a persuasive discussion of this (H/T Phil Trammell). For example, non-EU models imply information aversion. Now, a certain sort of information aversion might make sense in the context of longtermism. In line with your Popper quote, it might make sense to avoid information about the feasibility of highly-specific future scenarios. But that's not really the sort of information non-EU models imply aversion to. Instead, they imply aversion to info that would shift you toward the option that currently has a lot of ambiguity about it because you dislike it based on its current ambiguity.
So I don't think we can leave behind EU for another approach to evaluating outcomes. The problems, to me, seem to lie elsewhere. I think there are problems with the way we're arriving at probabilities (inventing subjective ones that invite biases and failing to adequately stick to base rates, for example). I also think there might be a point to be made about having priors on unlikely conclusions so that, for example, the conclusion of strong longtermism is so strange that we should be disinclined to buy into it based on the uncertainty about probabilities feeding into the claim. But the approach itself seems right to me. I honestly spent some time looking for alternative approaches because of these last two concerns I mentioned and came away thinking that EU is the best we've got.
I'd note, finally, that I take the utopianism point well and wold like to see more discussion of this. Utopian movements have a sordid history, and Popper is spot-on. Longtermism doesn't have to be utopian, though. Avoiding really bad outcomes, or striving for a middling outcome, is not utopian. This seems to me to dovetail with my proposal in the last paragraph to improve our probability estimates. Sticking carefully to base rates and things we have some idea about seems to be a good way to avoid utopianism and its pitfalls. So I'd suggest a form of longtermism that is humble about what we know and strives to get the least-bad empirical data possible, but I still think longtermism comes out on top.
This this this! As a PhD student in economics, I'm always pushing for the same thing in academia. People usually think saying nice job is useless, because it doesn't help people improve. It's important for people to know what they're doing right, though. It's also important for people to get positive reinforcement to keep going down a path, so if you want someone to keep persevering (which I hope we generally do), it's good to give them a boost when they do a good job.
Thanks for writing this. I've had similar questions myself.
I think the incentives issue here is a big one. One way I've wondered about addressing it is to find a bunch of people who forecast really well and whose judgments are not substantially affected by forecasting incentives. Then have them forecast risks. Might that work, and has anyone tried it?
I'm excited to see this! One thing I'd mention on the historian path and its competitiveness is you could probably do a lot of this sort of work as an economic historian with a PhD in economics. Economic historians study everything from gender roles to religion and do ambitious if controversial quantitative analyses of long-term trends. While economists broadly may give little consideration to historical context, the field of economic history prides itself on actually caring about history for its own sake as well, so you can spend time doing traditional historian things, like working with archival documents (see the Preface to the Oxford Encyclopedia of Economic History for a discussion of the field's norms).
The good thing here is it probably allows for greater outside options and potentially less competitiveness than a history PhD given the upsides of an economics PhD. You could also probably do similar work in political science.
>> Our impression is that although many of these topics have received attention from historians (examples: 1, 2, 3, 4, 5), some are comparatively neglected within the subject, especially from a more quantitative or impact-focused perspective.
I'd also note that some of these cites I don't think are history—2 and 4 are written by anthropologists. (I think in the former case he's sometimes classified as a biologist, psychologist, or economist too.)
I really do hope we have EAs studying history and fully support it, and I just wanted to give some closely related options!
Great post, and thanks for writing it. One note: if polarization is defined as "more extreme views on each issue" (e.g. more people wanting extremely high or extremely low taxes), then it does not seem to be happening according to some research. The sort of polarization happening in the U.S. is more characterized as ideological sorting. That is, views on any particular issue (abortion, affirmative action, gun control) don't have more mass on the extremes than before, but the views in each political party are less mixed.
This is nonetheless important, and I don't think it radically changes much of what you said. Affect toward the opposite party is still much more negative than before. But it might suggest we should be more concerned about the conflict between the parties itself (e.g. abusing constitutional norms, cancellation) and less concerned about their policies per se.