## Posts

## Comments

**tobycrisford**on Casting the Decisive Vote · 2023-03-29T11:12:02.607Z · EA · GW

I really like this argument. I think there's another way of framing it that occurred to me when reading it, that I also found insightful (though it may already be obvious):

- Suppose the value of your candidate winning is X, and their probability of winning if you don't do anything is p.
- If you could buy all the votes, you would pay X(1-p) to do so (value of your candidate winning minus a correction because they could have won anyway). This works out at X(1-p)/N per vote on average.
- If p>1/2, then buying votes probably has diminishing returns (certainly this is implied by the unimodal assumption).
- Therefore, if p>1/2, the amount you would pay for a single vote must be bounded below by X(1-p)/N.
- If p<1/2, I think you can just suppose that you are in a zero-sum game with the opposition party(ies), and take their perspective instead to get the same bound reflected about p=1/2.

The lower bound this gives seems less strict (1/2 * *X/N in the case that p=1/2, instead of X/N), but it helps me understand intuitively *why* the answer has to come out this way, and why the value of contributing to voting is directly analogous to the value of contributing to, say, Parfit's water tank for the injured soldiers, even though there are no probabilities involved there.

If as a group you do something with value O(1), then the value of individual contributions should usually be O(1/N), since value (even in expectation) is additive.

**tobycrisford**on Cooperative or Competitive Altruism, and Antisocial Counterfactuals · 2023-03-27T10:16:46.997Z · EA · GW

Point taken, although I think this is analogous to saying: *Counterfactual analysis will not leave us predictably worse off if we get the probabilities of others deciding to contribute right.*

**tobycrisford**on Cooperative or Competitive Altruism, and Antisocial Counterfactuals · 2023-03-27T10:13:15.032Z · EA · GW

Thank you for this correction, I think you're right! I had misunderstood how to apply Shapley values here, and I appreciate you taking the time to work through this in detail.

If I understand correctly now, the right way to apply Shapley values to this problem (with X=8, Y=2) is not to work with N (the number of players who end up contributing, which is unknown), but instead to work with N', the number of 'live' players who *could* contribute (known with certainty here, not something you can select), and then:

- N'=3, the number of 'live' players who are deciding whether to contribute.
- With N'=3, the Shapley value of the coordination is 1/3 for each player (expected value of 1 split between 3 people), which is
*positive.* - A positive Shapley value means that all players decide to contribute (if basing their decisions off Shapley values as advocated in this post), and you then end up with N=3.

Have I understood the Shapley value approach correctly? If so, I think my final conclusion still stands (even if for the wrong reasons) that a Shapley value analysis will lead to sub-optimal N (number of players deciding to participate). Since the optimal N here is 2 (or 1, which has same value).

As for whether the framing of the problem makes sense, with N as something we can select, the point I was making was that in a lot of real-world situations, N might well be something we can select. If a group of people have the same goals, they can coordinate to choose N, and then you're not really in a game-theory situation at all. (This wasn't a central point to my original comment but was the point I was defending in the comment you're responding to)

Even if you don't all have exactly the same goals, or if there's a lot of actors, it seems like you'll often be able to benefit by communicating and coordinating, and then you'll be able to improve over the approach of everyone deciding independently according to a Shapley value estimate: e.g. Givewell recommending a funding allocation split between their top charities.

**tobycrisford**on Cooperative or Competitive Altruism, and Antisocial Counterfactuals · 2023-03-20T21:55:46.779Z · EA · GW

*Edit: Vasco Grilo has pointed out a mistake in the final paragraph of this comment (see thread below), as I had misunderstood how to apply Shapley values, although I think the conclusion is not affected.*

If the value of success is X, and the cost of each group pursuing the intervention is Y, then ideally we would want to pick N (the number of groups that will pursue the intervention) from the possible values 0,1,2 or 3, so as to maximize:

(1-(1/2)^N) X - N Y

i.e., to maximize expected value.

If all 3 groups have the same goals, they'll all agree what N is. If N is not 0 or 3, then the best thing for them to do is to get together and decide which of them will pursue the intervention, and which of them won't, in order to get the optimum N. They can base their decision of how to allocate the groups on secondary factors (or by chance if everything else really is equal). If they all have the same goals then there's no game theory here. They'll all be happy with this, and they'll all be maximizing their own individual counterfactual expected value by taking part in this coordination.

This is what I mean by coordination. The fact that their individual approaches are different is irrelevant to them benefiting from this form of coordination.

'Maximize Shapley value' will perform worse than this strategy. For example, suppose X is 8, Y is 2. The optimum value of N for expected value is then 2 (2 groups pursue intervention, 1 doesn't). But using Shapley values, I think you find that whatever N is, the Shapley value of your contribution is always >2. So whatever every other group is doing, each group should decide to take part, and we then end up at N=3, which is sub-optimal.

**tobycrisford**on Cooperative or Competitive Altruism, and Antisocial Counterfactuals · 2023-03-20T20:28:25.774Z · EA · GW

To arrive at the 12.5% value, you were assuming that you knew with certainty that the other two teams *will* try to create the vaccine without you (and that they each have a 50% chance of succeeding). And I still think that under that assumption, 12.5% is the correct figure.

If I understand your reasoning correctly for why you think this is incoherent, it's because:

If the 3 teams independently arrive at the 12.5% figure, and each use that to decide whether to proceed, then you might end up in a situation where none of them fund it, despite it being clearly worth it overall.

But in making this argument, you've changed the problem. The other 2 teams are now no longer funding the vaccine with certainty, they are also making decisions based on counterfactual cost-benefit. So 12.5% is no longer the right number.

To work out what the new right number is, you have to decide how likely you think it is that the other 2 teams will try to make a vaccine, and that might be tricky. Whatever arguments you think of, you might have to factor in whether the other 2 teams will be thinking similarly. But if you really do all have the same goals, and there's only 3 of you, there's a fairly easy solution here, which is to just talk to each other! As a group you can collectively figure out what set of actions distributed among the 3 of you will maximize the global good, and then just do those. Shapley values don't have to come into it.

It gets more complicated if there's too many actors involved to all get together and figure things out like this, or if you don't all have exactly the same goals, and maybe there is scope for concepts like Shapley values be useful in those cases. And you might well be right that EA is now often in situations like these.

Maybe we don't disagree much in that case. I just wanted to push back a bit against the way you presented Shapley values here (e.g. as the "indisputably correct way to think about counterfactual value in scenarios with cooperation"). Shapley values are not always the right way to approach these problems. For example, the two thought experiments at the beginning of Parfit's paper I linked above are specific cases where Shapley values would leave you predictably worse off (and all decision theories will have some cases where they leave you predictably worse off).

**tobycrisford**on Cooperative or Competitive Altruism, and Antisocial Counterfactuals · 2023-03-20T18:32:45.536Z · EA · GW

*Edited after more careful reading of the post*

As you say in the post, I think all these things can be true:

1) The expected counterfactual value is all that matters (i.e. we can ignore Shapley values).

2) The 3 vaccine programs had zero counterfactual value in hindsight.

3) It was still the correct decision to work on each of them *at the time*, with the information that was available then.

At the time, none of the 3 programs knew that any of the others would succeed, so the expected value of each programme was very high. It's not clear to me why the '12.5%' figure in your probabilistic analysis is getting anything wrong.

If one vaccine program actually had known with certainty that the other two would already succeed, and if that really rendered their own counterfactual value 0 (probably not literally true in practice but granted for the sake of argument) then it seems very plausible to me that they probably should have focused on other things (despite what a Shapley value analysis might have told them).

It gets more complicated if you imagine that each of the three knew with certainty that the other two *could* succeed if they tried, but might not necessarily actually do it, because they will be reasoning similarly. Then it becomes a sort of game of chicken between the three of them as to which will actually do the work, and I think this is the kind of anti-cooperative nature of counterfactual value that you're alluding to. This is a potential problem with focusing only on counterfactuals, but focusing only on Shapley values has problems too, because it gives the wrong answer in cases where the decisions of others are already set in stone.

Toby Ord left a really good comment on the linked post on Shapley values that I think it's worth people reading, and I would echo his recommendation to read Parfit's Five Mistakes in Moral Mathematics for a really good discussion of these problems.

**tobycrisford**on Some Reflections on Philosophy Tube's “The Rich Have Their Own Ethics: Effective Altruism & the Crypto Crash” · 2023-02-27T18:15:57.860Z · EA · GW

I should admit at this point that I didn't actually watch the Philosophy Tube video, so can't comment on how this argument was portrayed there! And your response to that specific portrayal of it might be spot on.

I also agree with you that most existential risk work probably doesn't need to rely on the possibility of 'Bostromian' futures (I like that term!) to justify itself. You only need extinction to be very bad (which I think it is), you don't need it to be very very very bad.

But I think there must be some prioritisation decisions where it becomes relevant whether you are a weak longtermist (existental risk would be very bad and is currently neglected) or a strong longtermist (reducing existential risk by a tiny amount has astronomical expected value).

This is also a common line of attack that EA is getting more and more, and I think the reply "well yeah but you don't have to be on board with these sci-fi sounding concepts to support work on existential risk" is a reply that people are understandably more suspicious of if they think the person making it *is* on board with these more sci-fi like arguments. It's like when a vegan tries to make the case that a particular form of farming is unnecessarily cruel, even if you're ok with eating meat otherwise. It's very natural to be suspicious of their true motivations. (I say this as a vegan who takes part in welfare campaigns).

**tobycrisford**on Some Reflections on Philosophy Tube's “The Rich Have Their Own Ethics: Effective Altruism & the Crypto Crash” · 2023-02-25T22:34:49.837Z · EA · GW

On your response to the Pascal's mugging objection, I've seen your argument made before about Pascal's mugging and strong longtermism (that existential risk is actually very high so we're not in a Pascal mugging situation at all) but I think that reply misses the point a bit.

When people worry about the strong longtermist argument taking the form of a Pascal mugging, the small probability they are thinking about is *not* the probability of extinction, *it is the probability that the future is enormous*.

The controversial question here is: *how bad would extinction be?*

The strong-longtermist answer to this question is: *there is a very small chance that the future contains an astronomical amount of value, so extinction would therefore be astronomically bad in terms of expected value*.

Under the strong longtermist point of view, existential risk then sort of automatically dominates all other considerations, because even a tiny shift in existential risk carries enormous expected value.

It is *this* argument that is said to resemble a Pascal mugging, in which we are threatened/tempted with a small probability of enormous harm/reward. And I think this is a very valid objection to strong longtermism. The 'small probability' involved here is not the probability of extinction, but the small probability of us colonizing the galaxy and filling it with 10^(something big) digital minds.

Pointing out that existential risk is quite high does not undermine this objection to strong longtermism. If anything it makes it stronger, because it reduces the chance that the future is going to be as big as it needs to be for the strong longtermist argument to go through.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-17T22:13:06.507Z · EA · GW

When you write:

"I decide what the probability of the Mugger's threat is, though. The mugger is not god, I will assume. So I can choose a probability of truth p < 1/(number of people threatened by the mugger) because no matter how many people that the mugger threatens, the mugger doesn't have the means to do it, and the probability p declines with the increasing number of people that the mugger threatens, or so I believe. In that case, aren't people better off if I give that money to charity after all?"

This is exactly the 'dogmatic' response to the mugger that I am trying to defend in this post! We are in complete agreement, I believe!

For possible problems with this view, see other comments that have been left, especially by MichaelStJules.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-17T22:04:35.568Z · EA · GW

I still don't think the position I'm trying to defend is circular. I'll have a go at explaining why.

I'll start with aswering your question: in practice the way I would come up with probabilities to assess a charitable intervention is the same as the way you probably would. I'd look at the available evidence and update my priors in a way that at least tries to approximate the principle expressed in Bayes' theorem. Savage's axioms imply that my decision-describing-numbers between 0 and 1 have to obey the usual laws of probability theory, and that includes Bayes' theorem. If there is any difference between our positions, it will only be in how we should pick our *priors*.* * You pick those before you look at any evidence at all. How should you do that?

Savage's axioms don't tell you how to pick your priors. But actually I don't know of any other principle that does either. If you're trying to quantify 'degrees of belief' in an abstract sense, I think you're sort of doomed (this is the problem of induction). My question for you is, how do *you* do that?

But we do have to make decisions. I want my decisions to be constrained by certain rational sounding axioms (like the sure thing principle), but I don't think I want to place many more constraints on myself than that. Even those fairly weak constraints turn out to imply that there are *some* numbers, which you can call subjective probabilities, that I need to start out with as priors over states of the world, and which I will then update in the usual Bayesian way. But there is very little constraint in how I pick those numbers. They have to obey the laws of probability theory, but that's quite a weak constraint. It *doesn't *by itself imply that I have to assign non-zero probability to things which are concievable (e.g. if you pick a real number at random from the uniform distribution between 0 and 1, every possible outcome has probability 0).

So this is the way I'm thinking about the whole problem of forming beliefs and making decisions. I'm asking the question:

" I want to make decisions in a way that is consistent with certain rational seeming properties, what does that mean I *must* do, and what, if anything, is left unconstrained?"

I think I *must* make decisions in a Bayesian-expected-utility-maximising sort of way, but I don't think that I have to assign a non-zero probability to every concievable event. In fact, if I make one of my desired properties be that I'm not susceptible to infinity threatening Pascal muggers, then I *shouldn't* assign non-zero probability to situations that would allow me to influence infinite utility.

I don't think there is anything circular here.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-17T14:31:42.061Z · EA · GW

I can see it might make sense to set yourself a threshold of how much risk you are willing to take to help others. And if that threshold is so low that you wouldn't even give all the cash currently in your wallet to help any number of others in need, then you could refuse the Pascal mugger.

But you haven't really avoided the problem, just re-phrased it slightly. Whatever the amount of money you *would* be willing to risk for others, then on expected utility terms, it seems better to give it to the mugger, than to an excellent charity, such as the Against Malaria Foundation. In this framing of the problem, the mugger is now effectively robbing the AMF, rather than you, but the problem is still there.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-17T14:24:30.170Z · EA · GW

I am comfortable using subjective probabilities to guide decisions, in the sense that I am happy with trying to assign to every possible event a real number between 0 and 1, which will describe how I will act when faced with gambles (I will maximize expected utility, if those numbers are interpreted as probabilities).

But the *meaning* of these numbers is that they describe my decision-making behaviour, not that they quantify a degree of belief. I am rejecting the use of subjective probabilities in *that* context, if it is removed from the context of decisions. I am rejecting the whole concept of a 'degree of belief', or of event A being 'more likely' than event B. Or at least, I am saying there is no meaning in those statements that goes *beyond* the meaning: 'I will choose to receive a prize if event A happens, rather than if event B happens, if forced to choose'.

And if that's all that probabilities mean, then it doesn't seem necessarily wrong to assign probability zero to something that is concievable. I am simply describing how I will make decisions. In the Pascal mugger context: I would choose the chance of a prize in the event that I flip 1000 heads in a row on a fair coin, over the chance of a prize in the event that the mugger is correct.

That's still a potentially counter-intuitive conclusion to end up at, but it's a bullet I'm comfortable biting. And I feel much happier doing this than I do if you define subjective probabilities in terms of degrees of belief. I believe this language obscures what subjective probabilities fundamentally are, and this, previously, made me needlessly worried that by assigning extreme probabilities, I was making some kind of grave epistemological error. In fact I'm just describing my decisions.

Savage's axioms don't rule out unbounded expected utility, I don't think. This is from Savage's book, 'Foundations of Statistics', Chapter 5, 'Utility', 'The extension of utility to more general acts':

"If the utility of consequences is unbounded, say from above, then, even in the presence of P1-7, acts (though not gambles) of infinite utility can easily be constructed. My personal feeling is that, theological questions aside, there are no acts of infinite or minus infinite utility, and that one might reasonable so postulate, which would amount to assuming utility to be bounded."

The distinction between 'acts' and 'gambles' is I think just that gambles are acts with a finite number of possible consequences (which obviously stops you constructing infinite expected value), but the postulates themselves don't rule out infinite utility acts.

I'm obviously disagreeing with the Savage's final remark in this post. I'm saying that you could also shift the 'no acts of infinite or minus infinite utility' constraint away from the utility function, and onto the probabilities themselves.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-16T17:34:01.924Z · EA · GW

I guess I am calling into question the use of subjective probabilities to quantify beliefs.

I think subjective probabilities make sense in the context of decisions, to describe your decision-making behaviour (see e.g. Savage's derivation of probabilities from certain properties of decision-making he thinks we should abide by). But if you take the decisions out of picture, and try to talk about 'beliefs' in abstract, and try to get me to assign a real number between 0 and 1 to them, I think I am entitled to ask "why would I want to do something like that?" Especially if it's going to lead me into strange conclusions like "you should give your wallet to a Pascal mugger".

I think rejecting the whole business of assigning subjective probabilities to things is a very good way to reply to the Pascal's mugger problem in general. Its big weakness is that there are several strong arguments you can make that tell you you *should* quanity subjective uncertainty with mathematical probabilities, of which I think Savage's is the strongest. But the actual fundamental interpretation of subjective probabilities in Savage's argument is that they describe how you will act in the context of decisions, not that they quantify some more abstract notion like a degree of belief (he deliberately avoids talking in those terms). For example, P(A)>P(B) fundamentally means that you will choose to receive a prize in the event that A occurs, rather than in the event that B occurs, if forced to choose between the two.

If that's what subjective probabilities fundamentally mean, then it doesn't seem necessarily absurd to assign zero probability to something that is concievable. It at least doesn't violate any of Savage's axioms. It seems to violate our intuition of how quantification of credences should behave, but I think I can reply to that by resorting to the "why would I want to do something like that?" argument. Quantifying credences is not actually what I'm doing. I'm trying to make decisions.

Thanks for the links! I'm still not sure I quite understand point 5. Is the idea that instead of giving my wallet to the mugger, I should donate it to somewhere else that I think has a more plausible sounding way of achieving infinite utility? I suppose that might be true, but doesn't really seem to solve the Pascal mugger problem to me, just reframe it a bit.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-13T18:00:07.283Z · EA · GW

Lets assume for the moment that the probabilities involved are known with certainty. If I understand your original 'way out' correctly, then it would apply just as well in this case. You would embrace being irrational and still refuse to give the mugger your wallet. But I think here, the recommendations of expected utility theory in a Pascal's mugger situation are doing well 'on their own terms'. This is because expected utility theory doesn't tell you to maximize the probability of increasing your utility, it tells you to maximize your utility *in expectation*, and that's exactly what handing over your wallet to the mugger does. And if enough people repeated it enough times, some of them would eventually find themselves in a rare situation where the mugger's promises were real.

In reality, the probabilities involved are not known. That's an added complication which gives you a different way out of having to hand over your wallet, and that's the way out I'm advocating we take in this post.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-13T17:49:30.410Z · EA · GW

I think that's a very persuasive way to make the case against assigning 0 probability to infinities. I think I've got maybe three things I'd say in response which could address the problem you've raised:

- I don't think we necessarily have to asign 0 probability to the universe being infinite (or even to an infinite afterlife), but only to our capacity to
*influence*an infinite amount of utility with any given decision we're faced with, which is different in significant ways, and more acceptable sounding (to me). - Infinity is a tricky concept to grapple with. Even if I woke up in something which appeared to be biblical heaven or hell, is that really convincing evidence that it is going to last
*literally forever*. I'm not sure. Maybe. I'd certainly update to believe that there are very powerful forces at work that I don't understand, and that I'm going to be there a long time (if I've been there millenia already), but maybe it's not so irrational to continue to avoid facing up to infinite value. - The second point probably sounded like a stretch, but I think a fundamental part of my take on this is a particular view on what subjective probabilities fundamentally mean: they fundamentally describe how I'm going to make decisions. They obey the axioms of probability theory, because certain axioms of rationality say they should, but their fundamental meaning is that they describe my decision making behaviour. The interpretation in terms of the more abstract notion of a 'credence' is an optional add-on. Assigning 0 probability to the infinite is then simply me describing how I'm going to treat it for the purposes of decisions/gambles, rather than me saying that I am certain it can't be true.

I'd be interested in hearing you elaborate more on your final sentence (or if you've got any links you could point me to which elaborate on it).

(Also, I don't think heat death depends on an expanding universe. It's an endgame for existence that would require a change to the 2nd law of thermodynamics in order to escape, in my understanding.)

**tobycrisford**on The base level of achievement in life is simply to minimize harm. Nothing more. · 2023-02-13T17:34:23.036Z · EA · GW

You've pointed to a lot of potential complications, which I agree with, but I think they all also apply in cases where someone has *done* harm, not just in cases where they have not helped.

I just don't think the act/ommission distinction is very relevant here, and I thought the main claim of your post was that it was (but could have got the wrong end of the stick here!)

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-12T14:39:05.749Z · EA · GW

If we know the probabilities with certainty somehow (because God tells us, or whatever) then dogmatism doesn't help us avoid reckless conclusions. But it's an explanation for how we can avoid most reckless conclusions in practice (it's why I used the word 'loophole', rather than 'flaw'). So if someone comes up and utters the Pascal's mugger line to you on the street in the real world, or maybe if someone makes an argument for very strong longtermism, you could reject it on dogmatic grounds.

On your point about diminishing returns to utility preventing recklessness, I think that's a very good point if you're making decisions for yourself. But what about when you're doing ethics? So deciding which charities to give to, for example? If some action affecting N individuals has utility X, then some action affecting 2N individuals should have utility 2X. And if you accept that, then suddenly your utility function is unbounded, and you are now open to all these reckless and fanatical thought experiments.

You don't even need a particular view on population ethics for this. |The Pascal mugger could tell you that the people they are threatening to torture/reward already exist in some alternate reality.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-12T14:30:17.618Z · EA · GW

I think this comes down to the question of what subjective probabilities actually are. If something is concievable, do we have to give it a probability greater than 0? This post is basically asking, why should we?

The main reason I'm comfortable adapting my priors to be dogmatic is that I think there is probably not a purely epistemological 'correct' prior anyway (essentially because of the problem of induction), and the best we can do is pick priors that might help us to make practical decisions.

I'm not sure subjective probabilities can necessarily be given much meaning outside of the context of decision theory anyway. The best defence I know for the use of subjective probabilities to quantify uncertainty is due to Savage, and in that defence decisions are central. Subjective probabilities fundamentally describe decision making behaviour (P(A) > P(B) means someone will choose to receive a prize if A occurs, rather than if B occurs, if forced to choose between the two).

And when I say that some infinite utility scenario has probability 0, I am not saying it is inconcievable, but merely describing the approach I am going to take to making decisions about it: I'm not going to be manipulated by a Pascal's wager type argument.

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-12T14:22:00.740Z · EA · GW

Thanks for your comment, these are good points!

First, I think there is an important difference between Pascal's mugger, and Kavka's poison/Newcomb's paradox. The latter two are examples of ways in which a theory of rationality might be *indirectly* *self-defeating. *That means: if we try to achive the aims given to us by the theory, they can sometimes be worse achieved than if we had followed a different theory instead. This means there is a sense in which the theory is failing *on its own terms*. It's troubling when theories of rationality or ethics have this property, but actually *any* theory will have this property in some concievable circumstances, because of Parfit's satan thought experiment (if you're not familiar, do a ctrl+F for satan here: https://www.stafforini.com/docs/Parfit%20-%20Reasons%20and%20persons.pdf doesn't seem to have a specific wikipedia article that i can find).

Pascal's mugger seems like a different category of problem. The naive expected utility maximizing course of action (without dogmatism) seems absurd, but *not* because it is self-defeating. The theory is actually doing well on its own terms. It is just that those terms seem absurd. I think the Pascal mugger scenario should therefore present more of a problem for the expected utility theory, than the Kavka's poison/Newcomb's paradox thought experiments do.

On your second point, I don't have a good reply. I know there's probably gaping holes in the defence of Occam's razor I gave in the post, and that's a good example of why.

I'm very interested though, do you know a better justification for Occam's razor than usability?

**tobycrisford**on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-12T14:03:54.249Z · EA · GW

The two specific examples that come to mind where I've seen dogmatism discussed and rejected (or at least not enthusiastically endorsed) are these:

- https://80000hours.org/podcast/episodes/alan-hajek-probability-expected-value/ 80,000 hours podcast with Alan Hajek.
- https://rucore.libraries.rutgers.edu/rutgers-lib/40469/PDF/1/play/ Nick Beckstead's thesis on longtermism

The first is not actually a paper, and to be fair I think Hajek ends up being pretty sympathetic to the view that in practice, maybe we do just have to be dogmatic. But my impression was it was a sort of reluctant thing, and I came away with the impression that dogmatism is supposed to have major problems.

In the second, I believe dogmatism is treated quite dismissively, although it's been a while since I've read it and I may have misunderstood it even then!

So I may be summarising these resources incorrectly, but iirc they are both really good, and would recommend checking them out if you haven't already!

**tobycrisford**on The base level of achievement in life is simply to minimize harm. Nothing more. · 2023-02-11T13:08:33.001Z · EA · GW

I agree with some of what you say here. For example, from a mental health perspective, teaching yourself to be content 'regardless of your achievements' sounds like a good thing.

But I think adopting 'minimize harm' as the only principle we can use to make judgements of people, is far too simplistic a principle to work in practice.

For example, if I find out that someone watched a child fall into a shallow pond, and didn't go to help them (when the pond is shallow enough that that would have posed no risk to them), then I will judge them for that. I am not convinced by your post that it is wrong for me to make this judgement.

On the other hand, if someone does go to help the child, and in doing so commits some other relatively minor harm (maybe they steal some sweets from a second child and use them to help treat the first child, who is now hypothermic), then I would certainly not judge them at all for that, even though they have failed according to your principle.

I'm of course just regurgitating the usual arguments for utilitarianism. And you could easily raise objections (why do I judge the person who leaves the child to drown more harshly than I judge someone who doesn't donate all their spare income to the AMF, for example?) I don't have the answers to these objections. My point is just that this topic is complicated, and that the 'minimize harm' principle is too simplistic.

**tobycrisford**on An in-progress experiment to test how Laplace’s rule of succession performs in practice. · 2023-02-01T12:22:09.512Z · EA · GW

Sure, will do!

**tobycrisford**on An in-progress experiment to test how Laplace’s rule of succession performs in practice. · 2023-01-31T21:07:07.919Z · EA · GW

Apologies, I misunderstood a fundamental aspect of what you're doing! For some reason in my head you'd picked a set of conjectures which had just been posited this year, and were seeing how Laplace's rule of succession would perform when using it to extrapolate forward with no historical input.

I don't know where I got this wrong impression from, because you state very clearly what you're doing in the first sentence of your post. I should have read it more carefully before making the bold claims in my last comment. I actually even had a go at stating the terms of the bet I suggested before quickly realising what I'd missed and retracting. But if you want to hold me to it you can (I might be interpreting the forum wrong but I think you can still see the deleted comment?)

I'm not embarrassed by my original concern about the dimensions, but your original reply addressed them nicely and I can see it likely doesn't make a huge difference here whether you take a year or a month, at least as long as the conjecture was posited a good number of years ago (in the limit that "trial period"/"time since posited" goes to zero, you presumably recover the timeless result you referenced).

New EA forum suggestion: you should be able to disagree with your own comments.

**tobycrisford**on An in-progress experiment to test how Laplace’s rule of succession performs in practice. · 2023-01-31T20:51:01.186Z · EA · GW

That doesn't mean that all your dimensionful quantities haven't been divided by 12.

Do you want to take my bet then? You said: "If we calculate the 90% and the 98% confidence intervals, these are respectively (6 to 16) and (4 to 18) problems solved in the next three years."

I predict if you repeat the whole analysis using a month as the trial period, you will get 90 and 98% confidence intervals of (6 to 16) and (4 to 18) problems solved in the next three *months* (conditional on your stated solution to the original problem being correct).

I'd be prepared to bet £100 on this (winner picks charity for the loser to donate to).

**tobycrisford**on An in-progress experiment to test how Laplace’s rule of succession performs in practice. · 2023-01-31T14:05:38.177Z · EA · GW

*Edit: This comment is wrong and I'm now very embarrassed by it. It was based on a misunderstanding of what the NunoSempere is doing that would have been resolved by a more careful read of the first sentence of the forum post!*

Thank you for the link to the timeless version, that is nice!

But I don't agree with your argument that this issue is moot in practice. I think you should repeat your R analysis with months instead of years, and see how your predicted percentiles change. I predict they will all be precisely 12 times smaller (willing to bet a small amount on this).

This follows from dimensional analysis. How does the R script know what a year is? Only because you picked a year as your trial. If you repeat your analysis using a month as a trial attempt, your predicted mean proof time will then be X months instead of X years (i.e. 12 times smaller).

The same goes for any other dimensionful quantity you've computed, like the percentiles.

You could try to apply the linked timeless version instead, although I think you'd find you run into insurmountable regularization problems around t=0, for exactly the same reasons. You can't get something dimensionful out of something dimensionless. The analysis doesn't know what a second is. The timeless version works when applied retrospectively, but it won't work predicting forward from scratch like you're trying to do here, unless you use some kind of prior to set a time-scale.

**tobycrisford**on An in-progress experiment to test how Laplace’s rule of succession performs in practice. · 2023-01-31T08:17:55.850Z · EA · GW

I'm confused about the methodology here. Laplace's law of succession seems dimensionless. How do you get something with units of 'years' out of it? Couldn't you just as easily have looked at the probability of the conjecture being proven on a given day, or month, or martian year, and come up with a different distribution?

I'm also confused about what this experiment will tell us about the utility of Laplace's law outside of the realm of mathematical conjectures. If you used the same logic to estimate human life expectancy, for example, it would clearly be very wrong. If Laplace's rule has a hope of being useful, it seems it would only be after taking some kind of average performance over a variety of different domains. I don't think its usefulness in one particular domain should tell us very much.

**tobycrisford**on Should strong longtermists really want to minimize existential risk? · 2022-12-05T18:23:36.131Z · EA · GW

Thanks for the comment! I have quite a few thoughts on that:

First, the intention of this post was to criticize strong longtermism by showing that it has some seemingly ridiculous implications. So in that sense, I completely agree that the sentence you picked out has some weird edge cases. That's exactly the claim I wanted to make! I also want to claim that you can't reject these weird edge cases without also rejecting the core logic of strong longtermism that tells us to give enormous priority to longterm considerations.

The second thing to say though is that I wanted to exclude infinite value cases from the discussion, and I think both of your examples probably come under that. The reason for this is not that infinite value cases are not also problematic for strong longtermism (they really are!) but strong longtermists have already adapted their point of view in light of this. In Nick Beckstead's thesis, he says that in infinite value cases, the usual expected utility maximization framework should not apply. That's fair enough. If I want to criticize strong longtermists, I should criticize what they actually believe, not a strawman, so I stuck to examples containing very large (but finite) value in this post.

The third and final thought I have is a specific comment on your quantum multiverse case. If we'd make any possible decision in any branch, does that really mean that none of our decisions have any relevance? This seems like a fundamentally different type of argument to the Pascal's wager-type arguments that this post relates to, in that I think this objection would apply to *any* decision framework, not just EV maximization. If you're going to make all the decisions anyway, why does any decision matter? But you still might make the right decision on more branches than you make the wrong decision, and so my feeling is that this objection has no more force than the objection that in a deterministic universe, none of our decisions have relevance because the outcome is pre-determined. I don't think determinism should be problematic for decision theory, so I don't think the many-worlds interpretation of quantum mechanics should be either.

**tobycrisford**on Should strong longtermists really want to minimize existential risk? · 2022-12-05T18:05:23.244Z · EA · GW

Thanks! Very related. Is there somewhere in the comments that describes precisely the same issue? If so I'll link it in the text.

**tobycrisford**on Should strong longtermists really want to minimize existential risk? · 2022-12-05T18:03:40.463Z · EA · GW

I tried to describe some possible examples in the post. Maybe strong longtermists should have less trust in scientific consensus, since they should act as if the scientific consensus is wrong on some fundamental issues (e.g. on the 2nd law of thermodynamics, faster than light travel prohibition). Although I think you could make a good argument that this goes too far.

I think the example about humanity's ability to coordinate might be more decision-relevant. If you need to act as if humanity *will* be able to overcome global challenges and spread through the galaxy, given the chance, then I think that is going to have relevance for the prioritisation of different existential risks. You will overestimate humanity's ability to coordinate relative to if you didn't make that conditioning, and that might lead you to, say, be less worried about climate change.

I agree that it makes this post much less convincing that I can't describe a clear cut example though. Possibly that's a reason to not be as worried about this issue. But to me, the fact that "allows for a strong future" should almost always dominate "probably true" as a principle for choosing between beliefs to adopt, intuitively feels like it must be decision-relevant.

**tobycrisford**on In Defense of SBF · 2022-11-15T13:44:54.601Z · EA · GW

This seems like an odd post to me. Your headline argument is that you think SBF made an honest mistake, rather than wilfully misusing his users' funds, and most commenters seem to be reacting to that claim. The claim seems likely wrong to me, but if you honestly believe it then I'm glad you're sharing it and that it's getting discussed.

But in your third point (and maybe your second?) you seem to be defending the idea that even if SBF wilfully misused funds, then that's still ok. It was a bad bet, but we should celebrate people who take risky, but positive EV, gambles, even if they strongly violate ethical norms. Is that a fair summary of what you believe, or am I misreading/misunderstanding? If it is, I think this post is very bad and it seems very worrying that it's currently got +ve karma.

**tobycrisford**on Against Anthropic Shadow · 2022-08-24T06:50:54.104Z · EA · GW

I am very confident that the arguments do perfectly cancel out in the sky-colour case. There is nothing philosophically confusing about the sky-colour case, it's just an application of conditional probability.

That doesn't mean we can never learn anything. It just means that if X and Y are independent after controlling for a third variable Z, then learning X can give you no additional information about Y if you already know Z. That's true in general. Here X is the colour of the sky, Y is the probability of a catastrophic event occurring, and Z is the number of times the catastrophic event has occurred in the past.

---

In the Russian roulette example, you can only exist if the gun doesn't fire, but you can still use your existence to conclude that it is more likely that the gun won't fire (i.e. that you picked up the safer gun). The same should be true in anthropic shadow, at least in the one world case.

Fine tuning is helpful to think about here too. Fine tuning can be explained anthropically, but only if a large number of worlds actually exist. If there was only one solar system, with only one planet, then the fine tuning of conditions on that planet for life would be surprising. Saying that we couldn't have existed otherwise does not explain it away (at least in my opinion, for reasons I tried to justify in the 'possible solution #1' section).

In analogy with the anthropic explanation of fine-tuning, anthropic shadow might come back if there are many observer-containing worlds. You learn less from your existence in that case, so there's not necessarily a neat cancellation of the two arguments. But I explored that potential justification for anthropic shadow in the second section, and couldn't make that work either.

**tobycrisford**on Capitalism, power and epistemology: a critique of EA · 2022-08-22T16:43:23.626Z · EA · GW

I'd like to spend more time digesting this properly, but the statistics in this paragraph seem particularly shocking to me:

"For instance, Hickel et al. (2022) calculate that, each year, the Global North extracts from the South enough money to end extreme poverty 70x over. The monetary value extracted from the Global South from 1990 to 2015 - in terms of embodied labour value and material resources - outstripped aid given to the Global South by a factor of 30. "

They also seem hard to reconcile with each other. If the global north extracts every year 70 times what it takes to end extreme poverty (for one year or forever?), and from 1995-2015 the extracted value per year was only 30 times bigger than the aid given per year, then doesn't it follow that the global north is already giving in aid more than double what is needed to end extreme poverty (either at a per year rate or each year it gives double what is needed to end poverty for good)? What am I missing?

It can't be that the figure is 'what it would take to end extreme poverty with no extraction', because that figure would just be zero under this argument wouldn't it?

**tobycrisford**on Nuclear Fine-Tuning: How Many Worlds Have Been Destroyed? · 2022-08-22T15:39:52.417Z · EA · GW

I've replied to your comment on the other post now.

I don't want to repeat myself here too much, but my feeling is that explaining our luck in close calls using our position as observers does have the same problems that I think the anthropic shadow argument does.

It was never guaranteed that observers would survive until now, and the fact that we have is evidence of a low catastrophe rate.

**tobycrisford**on Against Anthropic Shadow · 2022-08-22T15:35:57.589Z · EA · GW

"the arguments ... Are both valid and don't actually conflict. They are entitled to both decrease how likely they take the catastrophes to be(due to no catastrophe changing the color of the sky), but they should also think that they are more likely than their historical record indicates. "

I agree with this. Those are two opposing (but not contradictory) considerations for them to take into account. But what I showed in the post was: once both are taken into account, they are left with the same conclusions as if they had just ignored the colour of the sky completely. That's what the bayesian calculation shows. The two opposing considerations precisely cancel. The historical record is all they actually need to worry about.

The same will be true in the anthropic case too (so no anthropic shadow) unless you can explain why the first consideration doesn't apply any more. Pointing out that you can't observe non-existence is one way to try to do this, but it seems odd. Suppose we take your framing of the Russian roulette example. Doesn't that lead to the same problems for the anthropic shadow argument? However you explain it, once you allow the conclusion that your gun is more likely to be the safer one, then don't you have to allow the same conclusion for observers in the anthropic shadow set-up? Observers are allowed to conclude that their existence makes higher catastrophe frequencies less likely. And once they're allowed to do that, that consideration is going to cancel out the observer-selection bias in their historical record. It becomes exactly analogous to the blue/green-sky case, and then they can actually just ignore anthropic considerations completely, just as observers in the blue/green sky world can ignore the colour of their sky.

**tobycrisford**on Nuclear Fine-Tuning: How Many Worlds Have Been Destroyed? · 2022-08-19T16:58:21.702Z · EA · GW

This is a really interesting topic.

I believe what you are describing here is the 'Anthropic Shadow' effect, which was described in this Bostrom paper: https://nickbostrom.com/papers/anthropicshadow.pdf

From what I can tell, your arguments are substantially the same as those in the paper, although I could be wrong?

Personally I've become pretty convinced that the anthropic shadow argument doesn't work. I think if you follow the anthropic reasoning through properly, a long period of time without a catastrophe like nuclear war *is* strong Bayesian evidence that the catastrophe rate is low, and I think this holds under pretty much whatever method of anthropic reasoning you favour.

I spelled out my argument in an EA forum post recently, so I'll link to that rather than repeating it here. It's a confusing topic and I'm not very sure of myself, so would appreciate your thoughts on whether I'm right, wrong, or whether it's actually independent of what you're talking about here: https://forum.effectivealtruism.org/posts/A47EWTS6oBKLqxBpw/against-anthropic-shadow

**tobycrisford**on Fanatical EAs should support very weird projects · 2022-06-30T16:47:16.890Z · EA · GW

The problem with neglecting small probabilities is the same problem you get when neglecting small anything.

What benefit does a microlitre of water bring you if you're extremely thirsty? Something so small it is equivalent to zero? Well if I offer you a microlitre of water a million times and you say 'no thanks' each time, then you've missed out! The rational way to value things is for a million microlitres to be worth the same as one litre. The 1000th microlitre doesn't have to be worth the same as the 2000th, but their values have to add to the value of 1 litre. If they're all zero then they can't.

I think the same logic applies to valuing small probabilities. For instance, what is the value of one vote from the point of view of a political party? The chance of it swinging an election is tiny, but they'll quickly go wrong if they assign all votes zero value.

I'm not sure what the solution to pascal's mugging/fanatacism is. It's really troubling. But maybe it's something like penalising large effects with our priors? We don't ignore small probabilities, we instead become extremely sceptical of large impacts (in proportion to the size of the claimed impact).

**tobycrisford**on Against Anthropic Shadow · 2022-06-08T07:09:15.060Z · EA · GW

I think that makes sense!

There is another independent aspect to anthropic reasoning too, which is how you assign probabilities to 'indexical' facts. This is the part of anthropic reasoning I always thought was more contentious. For example, if two people are created, one with red hair and one with blue hair, and you are one of these people, what is the probability that you have red hair (before you look in the mirror)? We are supposed to use the 'Self-Sampling Assumption' here, and say the answer is 1/2, but if you just naively apply that rule too widely then you can end up with conclusions like the Doomsday Argument, or Adam+Eve paradox.

I think that a complete account of anthropic reasoning would need to cover this as well, but I think what you've outlined is a good summary of how we should treat cases where we are only able to observe certain outcomes because we do not exist in others.

**tobycrisford**on Against Anthropic Shadow · 2022-06-06T14:30:16.975Z · EA · GW

I think that's a good summary of where our disagreement lies. I think that your "sample worlds until the sky turns out blue" methodology for generating a sample is very different to the existence/non-existence case, especially if there is actually only one world! If there are many worlds, it's more similar, and this is why I think anthropic shadow has more of a chance of working in that case (that was my 'Possible Solution #2').

I find it very interesting that your intuition on the Russian roulette is the other way round to mine. So if there are two guns, one with 1/1000 probability of firing, and one with 999/1000 probability of firing, and you pick one at random and it doesn't fire, you think that you have no information about which gun you picked? Because you'd be dead otherwise?

I agree that we don't get very far by just stating our different intuitions, so let me try to convince you of my point of view a different way:

Suppose that you really do have no information after firing a gun once and surviving. Then, if told to play the game again, you should be indifferent between sticking with the same gun, or switching to the different gun. Lets say you settle on the switching strategy (maybe I offer you some trivial incentive to do so). I, on the other hand, would strongly favour sticking with the same gun. This is because I think I have extremely strong evidence that the gun I picked is the less risky one, if I have survived once.

Now lets take a birds-eye view, and imagine an outside observer watching the game, betting on which one of us is more likely to survive through two rounds. Obviously they would favour me over you. My odds of survival are approximately 50% (it more or less just depends on whether I pick the safe gun first or not). Your odds of survival are approximately 1 in 1000 (you are guaranteed to have one shot with the dangerous gun).

This doesn't prove that your approach to formulating probabilities is wrong, but if ultimately we are interested in using probabilities to inform our decisions, I think this suggests that my approach is better.

On the fine tuning, if it is different, I would like to understand why. I'd love to know what the general procedure we're supposed to use is to analyse anthropic problems. At the moment I struggle to see how it could both include the anthropic shadow effect, and also have the fine tuning of cosmological constants be taken as evidence for a multiverse.

**tobycrisford**on Against Anthropic Shadow · 2022-06-05T18:34:14.988Z · EA · GW

Thank you for your comment! I agree with you that the difference between the bird's-eye view and the worm's eye view is very important, and certainly has the potential to explain why the extinction case is not the same as the blue/green sky case. It is this distinction that I was referring to in the post when asking whether the 'anthropicness' of the extinction case could explain why the two arguments should be treated differently.

But I'm not sure I agree that you are handling the worm's-eye case in the correct way. I could be wrong, but I think the explanation you have outlined in your comment is effectively equivalent to my 'Possible Solution #1', in the post. That is, because it is impossible to observe non-existence, we should treat existence as a certainty, and condition on it.

My problem with this solution is as I explained in that section of the post. I think the strongest objection comes from considering the anthropic explanation of fine tuning. Do you agree with the following statement?:

"The fine tuning of the cosmological constants for the existence of life is (Bayesian) evidence of a multiverse."

My impression is that this statement is generally accepted by people who engage in anthropic reasoning, but you can't explain it if you treat existence as a certainty. If existence is never surprising, then the fine tuning of cosmological constants for life cannot be evidence for anything.

There is also the Russian roulette thought experiment, which I think hits home that you should be able to consider the unlikeliness of your existence and make inferences based on it.

**tobycrisford**on Don’t Be Comforted by Failed Apocalypses · 2022-05-18T06:27:56.496Z · EA · GW

I can see that is a difference between the two cases. What I'm struggling to understand is why that leads to a different answer.

My understanding of the steps of the anthropic shadow argument (possibly flawed or incomplete) is something like this:

You are an observer -> We should expect observers to underestimate the frequency of catastrophic events on average, if they use the frequency of catastrophic events in their past -> You should revise your estimate of the frequency of catastrophic events upwards

But in the coin/tile case you could make an exactly analogous argument:

You see a blue tile -> We should expect people who see a blue tile to underestimate the frequency of heads on average, if they use the frequency of heads in their past -> You should revise your estimate of the frequency of heads upwards.

But in the coin/tile case, this argument is wrong, even though it appears intuitively plausible. If you do the full bayesian analysis, that argument leads you to the wrong answer. Why should we trust the argument of identical structure in the anthropic case?

**tobycrisford**on Don’t Be Comforted by Failed Apocalypses · 2022-05-17T18:52:58.889Z · EA · GW

In the tile case, the observers who see a blue tile are underestimating on average. If you see a blue tile, you then know that you belong to that group, who are underestimating on average. But that still should not change your estimate. That's weird and unintuitive, but true in the coin/tile case (unless I've got the maths badly wrong somewhere).

I get that there is a difference in the anthropic case. If you kill everyone with a red tile, then you're right, the observers on average will be biased, because it's only the observers with a blue tile who are left, and their estimates were biased to begin with. But what I don't understand is, why is finding out that you are alive any different to finding out that your tile is blue? Shouldn't the update be the same?

**tobycrisford**on Don’t Be Comforted by Failed Apocalypses · 2022-05-17T17:45:46.214Z · EA · GW

Thanks for your reply!

If 100 people do the experiment, the ones who end up with a blue tile will, on average, have fewer heads than they should, for exactly the same reason that most observers will live after comparitively fewer catastrophic events.

But in the coin case that still does not mean that seeing a blue tile should make you revise your naive estimate upwards. The naive estimate is still, in bayesian terms, the correct one.

I don't understand why the anthropic case is different.

**tobycrisford**on Don’t Be Comforted by Failed Apocalypses · 2022-05-17T14:22:39.520Z · EA · GW

I've never understood the bayesian logic of the anthropic shadow argument. I actually posted a question about this on the EA forum before, and didn't get a good answer. I'd appreciate it if someone could help me figure out what I'm missing. When I write down the causal diagram for this situation, I can't see how an anthropic shadow effect could be possible.

Section 2 of the linked paper shows that the probability of a catastrophic event having occurred in some time frame in the past *given that we exist now:* P(B_2|E), is smaller than its actual probability of occurring in that time frame, P. The two get more and more different the less likely we are to survive the catastrophic event (they call our probability of survival Q). It's easy to understand why that is true. It is more likely that we would exist now if the event did not occur than if it did occur. In the extreme case where we are certain to be wiped out by the event, then P(B_2|E) = 0.

This means that if you re-ran the history of the world thousands of times, the ones with observers around at our time would have fewer catastrophic events in their past, on average, than is suggested by P. I am completely happy with this.

But the paper then leaps from this observation to the conclusion that our naive estimate of the frequency of catastrophic events (i.e. our estimate of P) must be biased downwards. This is the point where I lose the chain of reasoning. Here is why.

What we care about here is *not* P(B_2|E). What we care about is our estimate of P itself. We would ideally like to calculate the posterior distribution of *P*, given both B_1,2 (the occurrence/non-occurrence of the event in the past), and our existence, E. The causal diagram here looks like this:

P -> B_2 -> E

This diagram means: P influences B_2 (the catastrophic event occurring), which influences E (our existence). But P does not influence E except through B_2.

*This means if we condition on B_2, the fact we exist now should have no further impact on our estimate of P*

To sum up my confusion: The distribution of (P|B_2,E) should be equivalent to the distribution of (P|B_2). I.e., there is no anthropic shadow effect.

In my original EA forum question I took the messy anthropics out of it and imagined flipping a biased coin hundreds of times and painting a blue tile red with probability 1-Q (extinction) if we ever get a head. If we looked at the results of this experiment, we could estimate the bias of the coin by simply counting the number of heads. The colour of the tile is irrelevant. And we should go with the naive estimate, *even though it is again true that people who see a blue tile will have fewer heads on average than is suggested by the bias of the coin*.

What this observation about the tile frequencies misses is that the tile is more likely to be blue when the probability of heads is smaller (or we are more likely to exist if P is smaller), and we should take that into account too.

Overall it seems like our naive estimate of P based on the frequency of the catastrophic event in our past is totally fine when all things are considered.

I'm struggling at the moment to see why the anthropic case should be different to the coin case.

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-05-11T07:19:52.044Z · EA · GW

"I would say exactly the same for this. If these people are being freshly created, then I don't see the harm in treating them as identical."

I think you missed my point. How can 1,000 people be identical to 2,000 people? Let me give a more concrete example. Suppose again we have 3 possible outcomes:

(A) (Status quo): 1 person exists at high welfare +X

(B): Original person has welfare reduced to X - 2, 1000 new people are created at welfare +X

(C): Original person has welfare reduced only to X - , 2000 new people are created, 1000 at welfare , and 1000 at welfare X + .

And you are forced to choose between (B) and (C).

How do you pick? I think you want to say 1000 of the potential new people are "effectively real", but which 1000 are "effectively real" in scenario (C)? Is it the 1000 at welfare ? Is it the 1000 at welfare X+? Is it some mix of the two?

If you take the first route, (B) is strongly preferred, but if you take the second, then (C) would be preferred. There's ambiguity here which needs to be sorted out.

"Then, supposedly no one is effectively real. But actually, I'm not sure this is a problem. More thinking will be required here to see whether I am right or wrong."

Thank you for finding and expressing my objection for me! This does seem like a fairly major problem to me.

"Sorry, but this is quite incorrect. The people in (C) would want to move to (B)."

No, they wouldn't, because the people in (B) are different to the people in (C). You can assert that you *treat* them the same, but you can't assert that they *are* the same. The (B) scenario with different people and the (B) scenario with the same people are both distinct, possible, outcomes, and your theory needs to handle them both. It can give the same answer to both, that's fine, but part of the set up of my hypothetical scenario is that the people *are* different.

"Isn't the very idea of reducing people to their welfare impersonal?"

Not necessarily. So called "person affecting" theories say that an act can only be wrong if it makes things worse *for someone*. That's an example of a theory based on welfare which is not impersonal. Your intuitive justification for your theory seemed to have a similar flavour to this, but if we want to avoid the non-identity problem, we need to reject this appealing sounding principle. It is possible to make things worse even though there is no one who it is worse *for*. Your 'effectively real' modification does this, I just think it reduces the intuitive appeal of the argument you gave.

**tobycrisford**on The COILS Framework for Decision Analysis: A Shortened Intro+Pitch · 2022-05-10T17:00:48.195Z · EA · GW

Where would unintended consequences fit into this?

E.g. if someone says:

"This plan would cause X, which is good. (Co) X would not occur without this plan, (I) We will be able to carry out the plan by doing Y, (L) the plan will cause X to occur, and (S) X is morally good."

And I reply:

"This plan will also cause Z, which is morally bad, and outweights the benefit of X"

Which of the 4 categories of claim am I attacking? Is it 'implementation'?

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-05-07T14:08:54.308Z · EA · GW

You can assert that you consider the 1000 people in (B) and (C) to be identical, for the purposes of applying your theory. That does avoid the non-identity problem in this case. But the fact is that they are not the same people. They have different hopes, dreams, personalities, memories, genders, etc.

By treating these different people as equivalent, your theory has become more *impersonal. * This means you can no longer appeal to one of the main arguments you gave to support it: that your recommendations always align with the answer you'd get if you asked the people in the population whether they'd like to move from one situation to the other. The people in (B) would not want to move to (C), and vice versa, because that would mean they no longer exist. But your theory now gives a strong recommendation for one over the other anyway.

There are also technical problems with how you'd actually apply this logic to more complicated situations where the number of future people differs. Suppose that 1000 extra people are created in (B), but 2000 extra people are created in (C), with varying levels of welfare. How do you apply your theory then? You now need 1000 of the 2000 people in (C) to be considered 'effectively real', to continue avoiding non-identity problem like conclusions, but which 1000? How do you pick? Different choices of the way you decide to pick will give you very different answers, and again your theory is becoming more impersonal, and losing more of its initial intuitive appeal.

Another problem is what to do under uncertainty. What if instead of a forced choice between (B) and (C), the choice is between:

0.1% chance of (A), 99.9% chance of (B)

0.1000001% chance of (A), 99.9% chance of (C).

Intuitively, the recommendations here should not be very different to the original example. The first choice should still be strongly preferred. But are the 1000 people still considered 'effectively real' in your theory, in order to allow you to reach that conclusion? Why? They're not guaranteed to exist, and actually, your real preferred option, (A), is more likely to happen with the second choice.

Maybe it's possible to resolve all these complications, but I think you're still a long way from that at the moment. And I think the theory will look a lot less intuitively appealing once you're finished.

I'd be interested to read what the final form of the theory looks like if you do accomplish this, although I still don't think I'm going to be convinced by a theory which will lead you to be predictably in conflict with your future self, even if you and your future self both follow the theory. I can see how that property can let you evade the repugnant conclusion logic while still *sort of* being transitive. But I think that property is just as undesirable to me as non-transitiveness would be.

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-05-04T22:09:49.794Z · EA · GW

"We minimise our loss of welfare according to the methodology and pick B, the 'least worst' option."

But (B) doesn't minimise our loss of welfare. In B we have welfare X-2, and in C we have welfare X - , so wouldn't your methodology tell us to pick (C)? And this is intuitively clearly wrong in this case. It's telling us not tmake a negligible sacrifice to our welfare now in order to improve the lives of future generations, which is the same problematic conclusion that the non-identity problem gives to certain theories of population ethics.

I'm interested in how your approach would tell us to pick (B), because I still don't understand that?

I won't reply to your other comment just to keep the thread in one place from now on (my fault for adding a P.S, so trying to fix the mistake). But in short, yes, I disagree, and I think that these flaws are unfortunately severe and intractable. The 'forcing' scenario I imagined is more like the real world than the unforced decisions. For most of us making decisions, the fact that people will exist in the future is inevitable, and we have to think about how we can influence their welfare. We are therefore in a situation like (2), where we are going to move from (A) to either (B) or (C) and we just get to pick which of (B) or (C) it will be. Similarly, figuring out how to incorporate uncertainty is also fundamental, because all real world decisions are made under uncertainty.

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-05-03T16:46:10.279Z · EA · GW

I understood your rejection of the total ordering on populations, and as I say, this is an idea that others have tried to apply to this problem before.

But the approach others have tried to take is to use the lack of a precise "better than" relation to evade the logic of the repugnant conclusion arguments, while still ultimately concluding that population Z is worse than population A. If you only conclude that Z is not worse than A, and A is not worse than Z (i.e. we should be indifferent about taking actions which transform us from world A to world Z), then a lot of people would still find that repugnant!

Or are you saying that your theory tells us *not* to transform ourselves to world Z? Because we should only ever do anything that will make things actually better?

If so, how would your approach handle uncertainty? What probability of a world Z should we be willing to risk in order to improve a small amount of real welfare?

And there's another way in which your approach still contains some form of the repugnant conclusion. If a population stopped dealing in hypotheticals and actually started taking actions, so that these imaginary people became real, then you could imagine a population going through all the steps of the repugnant conclusion argument process, thinking they were making improvements on the status quo each time, and finding themselves ultimately ending up at Z. In fact it can happen in just two steps, if the population of B is made large enough, with small enough welfare.

I find something a bit strange about it being different when happening in reality to when happening in our heads. You could imagine people thinking

"Should we create a large population B at small positive welfare?"

"Sure, it increases positive imaginary welfare and does nothing to real welfare"

"But once we've done that, they will then be real, and so then we might want to boost their welfare at the expense of our own. We'll end up with a huge population of people with lives barely worth living, that seems quite repugnant."

"It is repugnant, we shouldn't prioritise imaginary welfare over real welfare. Those people don't exist."

"But if we create them they *will* exist, so then we will end up deciding to move towards world Z. We should take action now to stop ourselves being able to do that in future."

I find this situation of people being in conflict with their future selves quite strange. It seems irrational to me!

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-05-03T16:37:29.768Z · EA · GW

It sounds like I have misunderstood how to apply your methodology. I would like to understand it though. How would it apply to the following case?

Status quo (A): 1 person exists at very high welfare +X

Possible new situation (B): Original person has welfare reduced to X - 2 , 1000 people are created with very high welfare +X

Possible new situation (C): Original person has welfare X - , 1000 people are created with small positive welfare .

I'd like to understand how your theory would answer two cases: (1) We get to choose between all of A,B,C. (2) We are forced to choose between (B) and (C), because we know that the world is about to instantaneously transform into one of them.

This is how I had understood your theory to be applied:

- Neither (B) nor (C) are better than (A), because an instanataneous change from (A) to (B) or (C) would reduce real welfare (of the one already existing person).
- (A) is not better than (B) or (C) because to change (B) or (C) to (A) would cause 1000 people to disappear (which is a lot of negative real welfare).
- (B) and (C) are neither better or worse than each other, because an instantaneous change of one to the other would involve the loss of 1000 existing people (negative real welfare) which is only compensated by the creation of imaginary people (positive imaginary welfare). It's important here that the 1000 people in (B) and (C) are not the same people. This is the non-identity problem.

From your reply it sounds like you're coming up with a different answer when comparing (B) to (C), because both ways round the 1000 people are always considered imaginary, as they don't literally exist in the status quo? Is that right?

If so, that still seems like it gives a non-sensical answer in this case, because it would then say that (C) is better than (B) (real welfare is reduced by less), when it seems obvious that (B) is actually better? This is an even worse version of the flaw you've already highlighted, because the existing person you're prioritising over the imaginary people is already at a welfare well above the 0 level.

If I've got something wrong and your methodology can explain the intuitively obvious answer that (B) is better than (C), and should be chosen in example (2) (regardless of their comparison to A), then I would be interested to understand how that works.

**tobycrisford**on Is the reasoning of the Repugnant Conclusion valid? · 2022-04-24T11:38:02.987Z · EA · GW

P.S. Thinking about this a bit more, doesn't this approach fail to give sensible answers to the non-identity problem as well? Almost all decisions we make about the future will change not just the welfare of future people, but *which future people exist*. That means every decision you could take will reduce real welfare, and so under this approach no decision can be be better than any other, which seems like a problem!