Posts

When should EAs allocate funding randomly? An inconclusive literature review. 2018-11-17T14:53:38.803Z · score: 34 (22 votes)

Comments

Comment by max_daniel on What are the best arguments for an exclusively hedonistic view of value? · 2019-10-21T20:07:50.264Z · score: 5 (5 votes) · EA · GW

Here are some overviews:

https://plato.stanford.edu/entries/hedonism/#EthHed

https://plato.stanford.edu/entries/well-being/#Hed

My guess is that ultimately you'll just find yourself in an irresolvable standoff of differing intuitions with people who favor a different view of value. Philosophers have debated this question for millennia to decades (depending on how we count) and haven't reached agreement, so I think in the absence of some methodological revolution settling this question is hopeless. (Though of course, you clarifying your own thinking, or arriving at a view you feel more confident in yourself, seem feasible.)

Comment by max_daniel on Reality is often underpowered · 2019-10-20T18:34:08.619Z · score: 4 (3 votes) · EA · GW

I agree with your points, but from my perspective they somewhat miss the mark.

Specifically, your discussion seems to assume that we have a fixed, exogenously given set of propositions or factors X, Y, ..., and that our sole task is to establish relations of correlation and causation between them. In this context, I agree on preferring "wide surveys" etc.

However, in fact, doing research also requires the following tasks:

  • Identify which factors X, Y, ... to consider in the first place.
  • Refine the meaning of the considered factors X, Y, ... by clarifying their conceptual and hypothesized empirical relationships to other factors.
  • Prioritize which of the myriads of possible correlational or causal relationships between the factors X, Y, ... to test.

I think that depth can help with these three tasks in ways in which breadth can't.

For instance, in Will's example, my guess is that the main value of considering the history of Objectivism does not come from moving my estimate for the strength of the hypothesis "X = romantic involvement between movement leaders -> Y = movement collapses". Rather, the source of value is including "romantic involvement between movement leaders" into the set of factors I'm considering in the first place. Only then am I able to investigate its relation to outcomes of interests, whether by a "wide survey of cases" or otherwise. Moreover, I might only have learned about the potential relevance of "romantic involvement between movement leaders" by looking at some depth into the history of Objectivism. (I know very little about Objectivism, and so don't know if this is true in this instance; it's certainly possible that the issue of romantic involvement between Objectivist leaders is so well known that it would be mentioned in any 5-sentence summary one would encounter during a breadth-first process. But it also seems possible that it's not, and I'm sure I could come up with examples where the interesting factor was buried deeply.)

My model here squares well with your observation that a "common feature among superforecasters is they read a lot", and in fact makes a more specific prediction: I expect that we'd find that superforecasters read a fair amount (say, >10% of their total reading) of deep, small-n case studies - for example, historical accounts of a single war, economic policy, or biographies.

[My guess is that my comment is largely just restating Will's points from his above comment in other words.]


(FWIW, I think some generators of my overall model here are:

  • Frequently experiencing disagreements I have with others, especially around AI timelines and takeoff scenarios, as noticing a thought like "Uh... I just think your overall model of the world lacks depth and detail." rather than "Wait, I've read about 50 similar cases, and only 10 of them are consistent with your claim".
  • Semantic holism, or at least some of the arguments usually given in its favor.
  • Some intuitive and fuzzy sense that, in the terminology of this Julia Galef post, being a "Hayekian" has worked better for me than being a "Planner", including for making epistemic progress.
  • Some intuitive and fuzzy sense of what I've gotten out of "deep" versus "broad" reading. E.g. my sense is that reading Robert Caro's monumental, >1,300-page biography of New York city planner Robert Moses has had a significant impact on my model of how individuals can attain political power, albeit by adding a bunch of detail and drawing my attention to factors I previously wouldn't have considered rather than by providing evidence for any particular hypothesis.)



Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-09T19:20:10.529Z · score: 5 (3 votes) · EA · GW

I agree. However, your reply makes me think that I didn't explain my view well: I do, in fact, believe that it is not obvious that, say, setting up seed banks is "better than doing nothing" - and more generally, that nothing is obviously better than doing nothing.

I suspect that my appeal to "diverting attention and funding" as a reason for this view might have been confusing. What I had in mind here was not an argument about opportunity cost: while true, I did not want to say that an actor that set up a seed bank could perhaps have done better by doing something else instead (say, donating to ALLFED).

Instead, I was thinking of effects on future decisions (potentially by other actors), as illustrated by the following example:

  • Compare the world in which, at some time t0, some actor A decides to set up a seed bank (say, world w1) with the world w2 in which A decides to do nothing at t0.
  • It could be the case that, in w2, at some later time t1, a different actor B makes a decision that:
    • Causes a reduction in the risk of extinction from nuclear war that is larger than the effect of setting up a seed bank at t0. (This could even be, say, the decision to set up two seed banks.)
    • Happened only because A did not set up a seed bank at t0, and so in particular does not occur in world w1. (Perhaps a journalist in w2 wrote a piece decrying the lack of seed banks, which inspired B - who thus far was planning to become an astronaut - to devote her career to setting up seed banks.)

Of course, this particular example is highly unlikely. And worlds w1 and w2 would differ in lots of other aspects. But I believe considering the example is sufficient to see that extinction risk from nuclear war might be lower in world w2 than in w1, and thus that setting up a seed bank is not obviously better than doing nothing.

Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-09T17:14:15.698Z · score: 6 (3 votes) · EA · GW

I agree, and in fact "nothing is obviously good" describes my (tentative) view reasonably well, at least if (i) the bar for 'obviously' is sufficiently high and (ii) 'good' is to be understood as roughly 'maximizing long-term aggregate well-being.'

Depending on the question one is trying to answer, this might not be a useful perspective. However, I think that when our goal is to actually select and carry out an altruistic action, this perspective is the right one: I'd want to simply compare the totality of the (wellbeing-relevant) consequences with the relevant counterfactual (e.g., no action, or another action), and it would seem arbitrary to me to exclude certain effects because they are due to a general or indirect mechanism.

(E.g., suppose for the sake of the argument that I'm going to die in a nuclear war that would not have happened in a world without seed banks. - I'd think that my death makes the world worse, and I'd want someone deciding about seed banks today to take this disvalue into account; this does not depend on whether the mechanism is that nuclear bombs can be assembled from seeds, or that seed banks have crowded out nuclear deproliferation efforts, or whatever.)

Comment by max_daniel on My experience on a summer research programme · 2019-10-09T16:59:05.466Z · score: 4 (3 votes) · EA · GW
Minor: you may want to have shorter paragraphs - to partition the text more - to increase readability.

Thank you, I much appreciate specific suggestions. This makes sense to me, and I'll try to keep it in mind for future posts.

Comment by max_daniel on My experience on a summer research programme · 2019-10-09T12:17:43.751Z · score: 24 (7 votes) · EA · GW

[I co-directed the summer programme described in the OP, together with Rose Hadshar. Views are my own.]

Hi Denise and Siebe, thank you very much for your comments. I think they are valuable contributions to the conversation about the opportunities and risks of work at EA organizations.

I'm actually concerned (and have been so for a while) that many people in the community overestimate the value of work at EA organizations compared to alternatives. For example, if a friend told me that the only option they've considered for their summer was this summer programme, I'd strongly advise them to also look at opportunities outside the EA community. I think that widely promoted messages, e.g. about the value of 'direct work', have contributed to this (what I think is a) misperception (NB I don't believe that the sources of these messages actually intended to promote this perception, more that the nuance of their intentions got inadvertently lost as messages were propagated through the community). I therefore particularly appreciate that you point out some specific downsides a summer programme such as the one discussed here, and give some actionable reasons to consider alternatives. (And that you do this in a public place, accessible to those with limited opportunity to have high-bandwidth conversations with people working at EA organizations or other relevant experience.)

That being said, my current view is that this summer programme can be a good choice for some people in some circumstances. Based on my experience this summer and the feedback from our summer fellows, I'd also guess that for several of them the summer fellowship was actually more valuable than what they'd have done otherwise. I'd like to gesture at my reasoning, in the hope that this is helpful for people considering applying to a similar programme. (Caveat: as the OP also points out, this programme might look quite different next year, if it'll be run at all. I recommend checking with whoever will be directing it.)

  • Just to prevent misunderstandings, I'd like to mention that this year's programme differed in major ways from previous versions (and was run by different people). In fact, I had been participating in the programme in 2018, and had a mixed experience; when planning this year's programme, I've deliberately tried to learn from this experience (I had also reached out to my fellow fellows to ask them about their experience). E.g., I had appreciated the intellectual autonomy and the opportunity to informally talk to staff at CEA, FHI, and GPI (which shared an office at the time), but I had also often felt lonely and isolated and would have liked more supervision for one of my two projects. As a result, for this year we designed several activities aimed at fostering a collaborative and open culture among the summer fellows, giving them opportunities to introduce themselves and their projects to other staff at the office, and paired each summer fellow with a mentor. 
  • Overall, I think my experience last year was very unlike the experience of this year's fellows: For example, after a prompt to consider what they'd have done otherwise, this year 10 out of 11 people answered 9 or 10 on a 1-10 scale for "How much better or worse do you feel about the summer fellowship than about what you would have done otherwise?" (and the other one with an 8); on the same scale for "If the same program (same organizers, same structure etc.) happened next year, how strongly would you recommend a friend (with similar background to you before the fellowship) to apply?", 9 answered with a 9 or 10 (and the other ones with 7 and 8). While I have concerns about the validity of such self-reported impressions for assessing the value of the programme (which I think will depend on things like future career outcomes), I'm confident that I'd have given significantly less positive responses about my experience last year. 
  • Denise, you're right that this year's programme did "not by default result in job offers being made to the best performers," and wasn't intended to do so. I don't believe this is necessarily a downside, however. In my view, one of the main benefit of this programme is to help people decide whether they'll want to aim for similar work in the future. I think this will often be valuable for people who wouldn't be available for full employment after the programme, e.g. because they'll continue their university studies. In addition, I believe that several of the benefits of this year's programme were enabled by us fostering a relatively non-evaluative atmosphere, which I worry would not have been possible in a programme designed to identify future hires. So overall I think that (a) programmes that are essentially extended "work trials" and (b) programmes that won't usually lead to future employment have different upsides and downsides, and that there is room for both. (I do believe it's important to be transparent about which of the two one is running. I'm not aware of any of this year's summer fellows erroneously believing the programme would result in job offers, but I'd consider it a severe communications mistake on our part if this has happened.) As an aside, I think that a programme like this can to some extent further people's careers. For example, for the summer fellows I mentored I'm now able to provide a credible reference for applications to jobs in EA, and in fact expect to do so over the next few weeks in 1-2 cases. I do, however, agree that these benefits are importantly different from the prospect of a relevant job offer, and I doubt that these benefits alone would make the summer programme worthwhile.
  • My impression is that our mentoring setup worked quite well, at least from the fellows' perspective: 7 said they found mentoring "very valuable", 3 "valuable", and 1 "~neutral;" similarly, 10 out of 11 people said they'd like their mentor to mentor other people in the future, 9 of them maximally strongly. This seems consistent with responses on specific ways in which mentoring provided value and ways in which it could have been better. That being said, I do think there is room for improvement - for example, we weren't in all cases able to find a mentor with relevant domain expertise. My loose impression from this year was that cases in which the mentor had domain expertise were a much "safer bet" in terms of mentoring being valuable. In other cases, I think we were able to overcome this challenge to some extent: for example, I worked with two of my mentees to help them reach out to various domain experts, as a result of which they had several conversations; I still don't expect this provided as much value as direct and regular supervision from someone with domain expertise would have. (On the other hand, I think domain expertise is helpful all else equal, but far from sufficient for good mentoring. We did try to provide guidance to mentors, though I think with mixed success.) Anecdotally, based on my own experience in academia (and the distribution among my friends), my guess is that we've provided significantly more mentoring than the median academic internship, though except in 1-4 cases not as much quality-weighted mentorship as in the right tail (say, >95th percentile) of the academic mentorship distribution. (My wild guess is the situation is better in industry, though with other downsides, e.g. less autonomy.)
  • As mentioned earlier, I think the programme did a decent job at helping people to decide if they want to work in similar roles in the future - and does so in a way that is less costly to people than, say, applying for a full-time job at these organizations. So in this sense I do think it had career-related benefits, albeit by improving future decisions rather than by getting a job immediately. For instance, I think I would have benefitted significantly from the opportunity to work at CEA or FHI for a few weeks before deciding whether to accept the offer I got for FHI's Research Scholars Programme, a 2-year role. (As I said, I participated in last year's summer programme - however, I needed to decide to apply to RSP before the summer programme started, and I got the RSP offer quite early during the summer.) In fact, I believe that in at least one case the summer fellowship helped a fellow to decide that they do not want to accept an offer to work at an EA organization, and I think this was among the most tangible, valuable outcomes of this summer. Siebe, I agree with you that this has to weighed against the potential to get more information about other paths by doing internships elsewhere. However, I think that conversely these other internships would provide less information on how much one would like to work at, say, FHI. I think that it depends on the specifics of someone's circumstances which of these things is more valuable to explore.
  • Summer fellows were paid. (With the exception of a very small number of people who couldn't be paid for legal reasons.) That being said, I do believe there might be room for improvement here: for instance, on a scale from 1 to 5 on whether they felt the amount of financial support was adequate 8 out of 11 fellows responded with 4 or 5 - I think that ideally this would be 11 out of 11. I'm also concerned that 10 out of 11 fellows indicated that their savings stayed constant or (in 3 cases) decreased; while I don't think this is catastrophic given the short duration of the programme, I think it suggests the programme is by far not as accessible to people with a less comfortable financial situation than I would like.
  • Siebe, I agree that "EA-branded organizations are young and often run these programs without much prior experience." For example, my highest degree is a master's in an unrelated field, and I've had maybe 2 years of work experience relevant to me running such a programme and mentoring researchers (though potentially somewhat less or much more depending on how you count things I did during my university years outside of a paid role). I agree this is a drawback, all else being equal. However, I think it might be relevant to note that (i) Rose Hadshar, the co-director of this year's programme, has been the Project Manager for the Research Scholars Programme since last fall, which I think is very relevant work experience (more so than my previous one), (ii) we consulted with several people with significantly more relevant experience, including staff involved in running previous versions of the summer programme. I also think the "all else being equal" clause is important, and possibly not true in expectation: for example, I think it's fair to say that Rose and I really cared about this programme and were highly motivated to make it go as well as we could. By contrast, I've heard many stories from my friends in which they did an internship at an established institution, working with people that had decades of work experience, but came away very frustrated because they felt the internship programme had been poorly thought out, consisted of boring work, or suffered from an uninspiring organizational culture. Clearly, many internship programmes outside of the EA community won't suffer from these problems (and some at EA organizations will), but overall I believe it's worth finding out as much as possible about the specifics of an internship programme one is considering rather than relying too much on proxies such as prior experience. 

I think most of these points provide complementary information to the considerations you raised, rather than undermining them. I think they're relevant if someone wanted to form an overall view on whether to apply to such a programme, but I'm not trying to dispute the value of your comments. I'm curious to what extent the points I raised change your overall view on the upsides and downsides of this summer programme, or if you have any further questions.

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-09T11:36:23.819Z · score: 5 (3 votes) · EA · GW

Thanks, this is helpful!

I hadn't considered the possibility that techniques prior to Gendlin might have included focusing-like techniques, and especially that he's claiming to have synthesized what was already there. This makes me less confident in my impression. What you say about the textbooks you read definitely also moves my view somewhat.

(By contrast, what you wrote about studies on focusing probably makes me somewhat reduce my guess on the strength of the evidence of focusing, but obviously I'm highly uncertain here as I'm extrapolating from weak cues - studies by Gendlin himself, correlational claim of intuitively dubious causal validity - rather than having looked at the studies themselves.)

This all still doesn't square well with my own experiences with and models of therapy, but they may well be wrong or idiosyncratic, so I don't put much weight on them. In particular, 20-30% of sessions still seems higher than what I would guess, but overall this doesn't seem sufficiently important or action-relevant that I'd be interested to get at the bottom of this.

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-08T23:24:10.643Z · score: 9 (5 votes) · EA · GW

I don't think this is very important for my overall view on CFAR's curriculum, but FWIW I was quite surprised by you describing Focusing as

a technique that forms the basis of a significant fraction of modern therapeutic techniques and I consider a core skill for doing emotional processing. I've benefited a lot from this, and it also has a pretty significant amount of evidence behind it (both in that it's pretty widely practiced, and in terms of studies), though only for the standards of behavioral psychology, so I would still take that with a grain of salt.

Maybe we're just using "significant fraction" differently, but my 50% CI would have been that focusing is part of 1-3 of the 29 different "types of psychotherapy" I found on this website (namely "humanistic integrative psychotherapy", and maybe "existential psychotherapy" or "person-centred psychotherapy and counselling"). [Though to be fair on an NHS page I found, humanistic therapy was one of 6 mentioned paradigms.] Weighting by how common the different types of therapy are, I'd expect an even more skewed picture: my impression is that the most common types of therapy (at least in rich, English-speaking countries and Germany, which are the countries I'm most familiar with) are cognitive-behavioral therapy and various kinds of talking therapy (e.g. psychoanalytic, i.e. broadly Freudian), and I'd be surprised if any of those included focusing. My guess is that less than 10% of psychotherapy sessions happening in the above countries include focusing, potentially significantly less than that.

My understanding had been that focusing was developed by Eugene Gendlin, who after training in continental philosophy and publications on Heidegger became a major though not towering (unlike, say, Freud) figure in psychotherapy - maybe among the top decile but not the top percentile in terms of influence among the hundreds of people who founded their own "schools" of psychotherapy.

I've spent less than one hour looking into this, and so might well be wrong about any of this - I'd appreciate corrections.

Lastly, I'd appreciate some pointers to studies on focusing. I'm not doubting that they exist - I'm just curious because I'm interested in psychotherapy and mental health, but couldn't find them quickly (e.g. I searched for "focusing Gendlin" on Google Scholar).

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-08T22:35:56.691Z · score: 1 (1 votes) · EA · GW

Thank you for raising this, I think I was too quick here in at least implicitly suggesting that this defence would work in all cases. I definitely agree with you that we have some desires that are about the future, and that it would misdescribe our desires to conceive all of them to be about present causal antecedents.

I think a more modest claim I might be able to defend would be something like:

The justification of everyday actions does not require an appeal to preferences with the property that, epistemically, we ought to be clueless about their content.

For example, consider the action of not wandering blindly into the road. I concede that some ways of justifying this action may involve preferences about whose contents we ought to be clueless - perhaps the preference to still be alive in 40 years is such a preference (though I don't think this is obvious, cf. "dodge the bullet" above). However, I claim there would also be preferences, sufficient for justification, that don't suffer from this cluelessness problem, even though they may be about the future - perhaps the preference to still be alive tomorrow, or to meet my friend tonight, or to give a lecture next week.

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-08T10:23:18.538Z · score: 7 (5 votes) · EA · GW

Seconded.

(I'm wondering whether this phenomenon could also be due to people using downvotes for different purposes. For example, I use votes roughly to convey my answer to the question "Would I want to see more posts like this on the Forum?", and so I frequently upvote comments I disagree with. By contrast, someone might use votes to convey "Do I think the claims made in this comment are true?".)

Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-07T23:26:54.101Z · score: 3 (2 votes) · EA · GW

Thanks for this suggestion. I agree that in general brainstorming and debates are best kept separate. I also wouldn't want to discourage anyone from posting an answer to this question - in fact, I'm unusually interested in more answers to this question. I'm not sure if you were saying that you in particular feel discouraged from ideating as a response to seeing my comment, but I'm sorry if so. I'm wondering if you would have liked me to explain why I was expressing my disagreement, and to explicitly say that I value you suggesting answers to the original question (which I do)?

Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-07T18:03:22.356Z · score: 9 (6 votes) · EA · GW

Again, I disagree this is obvious. Just some ways in which this could be negative:

  • It could turn out that some of the research required for rapid vaccine development can be misused or exacerbate other risk.
  • The availability of rapid vaccine manufacturing methods could lead to a false sense of confidence among decisionmakers, leading to them effectively neglecting other important prevention and mitigation measures against biorisk.
Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-07T18:01:17.524Z · score: 8 (5 votes) · EA · GW

Again, I disagree that any of these is obvious:

  • Ease of communication also opens up more opportunities for rash decisions and premature messages, can reduce the time available for decisions, and the potential for this infrastructure to be misused by malign actors.
  • Biorisk mitigation being higher status could contribute to making the dangers of bioweapons more widely known among malign actors, thus making it more likely that they're being developed.
  • Pakistan not having nukes would alter the geopolitical situation in South Asia in major ways, with repercussions for the relationships between the major powers India, China, and the US. I find it highly non-obvious what the net effect of this would be.
Comment by max_daniel on What actions would obviously decrease x-risk? · 2019-10-07T17:57:47.500Z · score: 9 (6 votes) · EA · GW

If "uncontroversially" means something like "we can easily see that their net effect is to reduce extinction risk," then I disagree. To give just two examples, the known availability of alternative foods might decrease the perceived cost of nuclear war, thus making it more likely; and it might instill a sense of false confidence in decision-makers, effectively diverting attention and funding from more effective risk-reduction measures. I'm perhaps willing to believe that, after weighing up all these considerations, we can all agree that the net effect is to reduce risk, but I think this is far from obvious.

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-07T17:37:22.035Z · score: 14 (6 votes) · EA · GW

Thanks, that helps me understand where you're coming from, though it doesn't change my views on CFAR. My guess is we disagree about various more general claims around the costs and benefits of legibility, but unfortunately I don't have time right now to articulate my view on this.

Very roughly, I think I (i) agree with you that excessive optimization for easily measurable metrics has harmed the public education system, and in particular has reduced benefits from the right tail, (ii) disagree with your implied criterion of using something like "quality-weighted sum of generated research" is an appropriate main criterion for assessing the education system, and thus by extension disagree with the emphasis on right-tail outcomes when evaluating the public education system as a whole, (iii) don't think this tells us much about CFAR as I both think that CFAR's environment makes increased legibility less risky (due to things like high goal-alignment with important stakeholders such as funders, a more narrow target audience, ...) and also that there are plenty of ways to become more legible that don't incur risks similar to standardized testing or narrow optimization for quantitative metrics (examples: qualitatively describe what you're trying to teach, and why you think this is a good idea; monitor and publish data such as number of workshops run, attendance etc., without narrowly optimizing for any of these; maintain a list of lessons learned).

(I upvoted your reply, not sure why it was downvoted by someone else.)

Comment by max_daniel on Arguments for moral indefinability · 2019-10-07T13:30:25.459Z · score: 7 (3 votes) · EA · GW

Here is a vaguely related rough project proposal I once wrote (apologies for the academic philosophy jargon):

"Implications of evaluative indeterminacy and ‘ontological crises’ for AI alignment

There is a broadly Wittgensteinian/Quinean view which says that we just can’t make meaningful judgments about situations we’re too unfamiliar with (could apply to epistemic judgments, evaluative judgments, or both). E.g. Parfit briefly mentions (and tries to rebut) this in Reasons and Persons before discussing the more science-fiction-y thought experiments about personal identity.

A more moderate variant would be the claim that such judgments at least are underdetermined; e.g. perhaps there are adequacy conditions (say, consistency, knowing all relevant facts, ...) on the process by which the judgment is being made, but the judgment’s content is sensitive to initial conditions or details of the process that are left open.

One reason to believe in such views could be the anticipation that some of our current concepts will be replaced in the future. E.g. perhaps ‘folk psychology’ will be replaced by some sort of scientific theory of consciousness. In LW terminology, this is known as ‘ontological crisis’.

Due to its unusual spatiotemporal and conceptual scope, the question of how to best shape the long-term future – and so by extension AI alignment – depends on many judgments that seem likely candidates for being un(der)determined if one of those views is true, in areas such as: Population ethics on astronomical scales; consciousness and moral patienthood of novel kinds of minds; axiological value of alien or posthuman civilizations; potential divergence of currently contingently relatively convergent axiologies (e.g. desire satisfaction vs. hedonism); ethical implications of creating new universes; ethical implications of ‘acausal effects’ of our actions across the multiverse, etc. (Some of these things might not even make sense upon scrutiny.)

Some of these issues might be about metaethics; some might be theoretical challenges to consequentialism or other specific theoretical views; some might have practical implications for AI alignment or EA efforts to shape the long-term future more broadly."

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-07T12:18:48.841Z · score: 16 (9 votes) · EA · GW

Just a brief reaction:

This makes sense to me as a response to Halstead's question. However, it actually makes me a bit less confident that (what you describe as) CFAR's reluctance to increase legibility is a good idea. An educational institution strikes me as something that can be made legible way more easily and with fewer downsides than an institution doing cutting-edge research in an area that is hard to communicate to non-specialists.

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-07T12:02:33.681Z · score: 1 (1 votes) · EA · GW
AMF is not usually justified from a long-termist point of view, so it is not really surprising that its benefits seem less obvious when you consider it from that point of view.

I agree in principle. However, there are a few other reasons why I believe making this point is worthwhile:

  • GiveWell has in the past advanced an optimistic view about the long-term effects of economic development.
  • Anecdotally, I know many EAs who both endorse long-termism and donate to AMF. In fact, my guess is that a majority of long-termist EAs donate to organizations that have been selected for their short-term benefits. As I say in another comment, I'm not sure this is a mistake because 'symbolic' considerations may outweigh attempts to directly maximize the impact of one's donations. However, it at least suggests that a conversation about the long-termist benefits of organizations like AMF is relevant for many people.
  • More broadly, at the level of organizations and norms, various actors within EA seem to endorse the conjunction of longtermism and recommending donations to AMF over donations to the Make-A-Wish foundation. It's unclear whether this is some kind of political compromise, a marketing tool, or done because of a sincere belief that they are compatible.
  • The point might serve as guidance for developing the ethical and epistemological foundations of EA. To explain, we might simply be unwilling to give up our intuitive commitments and insist that a satisfying ethical and epistemological basis would make longtermism and "AMF over Make-A-Wish" compatible. This would then be one criterion to reject proposed ethical or epistemological theories.
Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-07T11:07:06.515Z · score: 1 (1 votes) · EA · GW
The preference we have are a product of the beliefs we have about what will make our lives better over the long-run. My preference not to smoke is entirely a product of the fact that I believe that it will increase my risk of premature death.

I think this is precisely what I'm inclined to dispute. I think I simply have a preference against premature death, and that this preference doesn't rest on any belief about my long-run wellbeing. I think my long-run wellbeing is way too weird (in the sense that I'm doing things like hyperbolic discounting anyway) and uncertain to ground such preferences.

Nevertheless, I think it is irrational to smoke.

Maybe this points to a crux here: I think on sufficiently demanding notions of rationality, I'd agree with you that considerations analogous to cluelessness threaten the claim that smoking is irrational. My impression is that perhaps the key difference between our views is that I'm less troubled by this.

I don't think a Parfitian understanding of identity would help here

I'm inclined to agree. Just to clarify though, I wasn't referring to Parfit's claims about identity, which if I remember correctly are in the second or third part of Reasons and Persons. I was referring to the first part, where he among other things discusses what he calls the "self-interest theory S" (or something like this).

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-06T18:34:11.312Z · score: 3 (2 votes) · EA · GW

I agree this is a non-obvious question. There is a good reason why consequentialists at least since Sidgwick have asked to what extent the correct moral theory might imply to keep its own principles secret.

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-06T16:21:03.809Z · score: 4 (3 votes) · EA · GW
Cluelessness seems to imply that altruists should be indifferent between all possible actions that they can take. Is this implication of the view embraced?

As I say in another comment, I think that a few effects - such as reducing the risk of human extinction - can be rescued from cluelessness. Therefore, I'm not committed to being indifferent between literally all actions.

I do, however, think that consequentialism provides a reason for only very few actions. In particular, I do not think there is a valid argument for donating to AMF instead of the Make-a-Wish Foundation based on consequentialism alone.

This is actually one example of where I believe cluelessness has practical import. Here is a related thing I wrote a few months ago in another discussion:

"Another not super well-formed claim:
- Donating 10% of one's income to GiveWell charities, prioritizing to reduce chicken consumption over reducing beef consumption, and similar 'individual' actions by EAs that at first glance seem optimized for effectiveness are valuable almost entirely for their 'symbolic' and indirect benefits such as signalling and maintaining community norms.
- Therefore, they are analogous to things like: environmentalists refusing to fly or reducing the waste produced by their household; activists participating in a protest; party members attending weekly meetings of their party; religious people donating money for missionary purposes or building temples.
- Rash criticism of such actions in other communities that appeals to their direct short-term consequences is generally unjustified, and based on a misunderstanding of the role of such actions both within EA and in other communities. If we wanted to assess the 'effectiveness' of these other movements, the crucial question to ask (ignoring higher-level questions such as cause prioritization) about, say, an environmentalist insisting to always switch of the lights when they leave a room, would not be how much CO2 emissions are avoided; instead, the relevant questions would be things like: How does promoting a norm of switching off lights affect that community's ability to attract followers and other resources? How does promoting a norm of switching off lights affect that community's actions in high-stakes situations, in particular when there is strategic interdependence -- for example, what does it imply about the psychology and ability to make credible commitments of a Green party leader negotiating a coalition government?
- It is not at all obvious that promoting norms that are ostensibly about maximizing the effectiveness of all individual 'altruistic' decisions is an optimal or even net positive choice for maximizing a community's total impact. (Both because of and independently of cluelessness.) I think there are relatively good reasons to believe that several EA norms of that kind actually have been impact-increasing innovations, but this is a claim about a messy empirical question, not a tautology."

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-06T16:15:17.624Z · score: 4 (2 votes) · EA · GW
I don't see how you could make a general argument for cluelessness with respect to all decisions made by the community.

I agree. More specifically, I think the argument for cluelessness is defeatable, and tentatively think that we know of defeaters in some cases. Concretely, I think that we are justified in believing in the positive expected value of (i) avoiding human extinction and (ii) acquiring resources for longtermist goals. (Though I do think that for none of these it is obvious that their expected value is positive, and that considering either to be obvious would be a serious epistemic error.)

[...] I don't see how this could ever generalise to an argument that all of our decisions are clueless, since the level of uncertainty will always be almost entirely dependent on the facts about the particular case. Why would uncertainty about the effects of AMF have any bearing on uncertainty about the effects of MIRI or the Clean Air Task Force?

I think you overstate your case here. I agree in principle that "the level of uncertainty will always be almost entirely dependent on the facts about the particular case," and so that whether we are clueless about any particular decision is a contingent question. However, I think that inspecting the arguments for cluelessness about, say, the effects of donations to AMF do suggest that cluelessness will be pervasive, for reasons we are in principle able to isolate. To name just one example, many actions will have small but in expectation non-zero, highly uncertain effect on the pace of technological growth; this in turn will have an in expectation non-zero, highly uncertain net effect on the risk of human extinction, which in turn ... - I believe this line of reasoning alone could be fleshed out into a decisive argument for cluelessness about a wide range of decisions.

Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-06T16:04:11.272Z · score: 13 (4 votes) · EA · GW
Similar arguments for complex cluelessness also seems to apply to my own decisions about what would be in my rational self-interest to do. Nevertheless, I will not be wandering blindly into the road outside my hotel room in 10 minutes.

I appreciate you making this point, as I think it's interesting and I hadn't come across it before. However, I don't currently find it that compelling, for the following reasons [these are sketches, not fully fleshed out arguments I expect to be able to defend in all respects]:

  • I think there is ample room for biting the bullet regarding rational self-interest, while avoiding counter-intuitive conclusions. To explain, I think that the common sense justification for not wandering blindly into the road simply is that I currently have a preference against being hit by a car. I don't think the intuition that it'd be crazy to wander blindly into the road is driven by any theory that appeals exclusively to long-term consequences on my well-being, nor do I think it needs such a philosophical fundament. I think a theory of self-interest that just appeals to consequences for my time-neutral lifetime wellbeing is counter-intuitive and faced with various problems anyway (see e.g. the first part of Reasons and Persons). If it was the case that I'm clueless about the long-term consequences of my actions on my wellbeing, I think that would merely be yet another problem for the rational theory of self-interest; but I was inclined to discard that theory anyway, and don't think that discarding it would undermine any of my common sense beliefs. So while I agree that there might be a problem analogous to cluelessness for philosophers who want to come up with a defensible theory of self-interest, I don't think we get a common-sense-based argument against cluelessness.
  • However, I think one may well be able to dodge the bullet, at least to some extent. I think it's simply not true that we are as clueless about our own future wellbeing as we are about the consequences of our actions for long-run impartial goodness, for the following reasons:
    • Roughly speaking, my own future predictable influence over my own future wellbeing is much greater than my own future influence over impartial goodness. Whatever happens to me, I'll know how well off I am, and I'll be able to react to it; something pretty drastic would need to happen to have a very large and lasting effect on my wellbeing. By contrast, I usually simply won't know how impartial goodness has changed as a result of my actions, and even if I did, it would often be beyond my power to do something about it. If the job I enthusiastically took 10 years ago is now bad for me, I can quit. If the person I rescued from drowning when they were a child is now a dictator wrecking Europe, that's too bad but I'm stuck with it.
    • The time horizon is much shorter, and there is limited opportunity for the indirect effects of my actions to affect me. Suppose I'll still be alive in 60 years. It will, e.g., still be true that my actions will have far-reaching effects on the identities of people that will be born in the next 60 years. However, the number of identities affected, and the indirect effects flowing from this will be much more limited compared to time-horizons that are orders of magnitudes longer; more importantly, most of these indirect effects won't affect me in any systematic way. While there will be some effects on me depending on which people will be born in, say, Nepal in 40 years, I think the defence that these effects will "cancel out" in expectation works, and similarly for most other indirect effects on my wellbeing.
    • Maybe most importantly: I think that a large part of the force of the "new problem of cluelessness" (i.e., instances where the defence that "indirect effects cancel out in expectation" doesn't work) comes from the contingent fact that (according to most plausible axiologies) impartial goodness is freaking weird. I'm not sure how to make this precise, but it seems to me that an important part of the story is that impartial goodness, unlike my own wellbeing, hinges on heavy-tailed phenomena spread out over different scales - e.g., maybe I'm just barely able to guess the sign of the impact of AMF on population size, but assessing the impacts on impartial goodness would also require me to assess the impacts of population size on economic growth, technological progress, the trajectory of farmed animal populations, risks of human extinction, etc. That is, small indirect net effects of my actions on impartial goodness might blow up due to their effects on much larger known and unknown levers, giving rise to the familiar phenomenon of "crucial considerations." For all I know, in an idealized epistemic state I'd realize that the effects of my actions are dominated by their indirect effects on electron suffering (using this as a token example of "something really weird I haven't considered", not to suggest we ought to in fact take electron suffering seriously) - by contrast, I don't think there could be similar "crucial considerations" for my own well-being. It is not plausible that, say, actually, the effect of my walking into the road on my wellbeing will be dominated by the increased likelihood of seeing a red car; it seems that the "worst" kind of issues I'll encounter are things like "does drinking one can of Coke Zero per day increase or decrease my life expectancy?", which is a challenging but not hopeless problem; it's something I'm uncertain, but not clueless about.
Comment by max_daniel on [link] Andreas Mogensen's "Maximal Cluelessness" · 2019-10-06T14:16:27.920Z · score: 1 (1 votes) · EA · GW

Thanks for this! - My tentative view is that cluelessness is an important issue with practical implications, and so I'm particularly interested in thoughtful arguments for opposing views.

I'll post some reactions in separate comments to facilitate discussion.

Knightian uncertainty seems to me never rational. There are strong arguments that credence functions should be sharp. [...]

I agree that are strong arguments that credence functions should be sharp. So I don't think the case for cluelessness is a slam dunk. (Granting that, roughly speaking, considering cluelessness to be an interesting problem commits one to a view using non-sharp credence functions. I'm not in fact sure if one is thus committed.) It just seems to me that the arguments for taking cluelessness seriously as a problem are stronger. Still, I'm curious what you think the best arguments for credence functions being sharp are, or where I can read about them.

Comment by max_daniel on Why do social movements fail: Two concrete examples. · 2019-10-06T13:17:00.231Z · score: 2 (2 votes) · EA · GW
The Spanish Inquisition generally made life hard for people who had observations to make against religion, tradition, etc. The Catholic Church had the first Encyclopedia (by D'Alambert and Diderot) in their list of banned books in 1759.

[I'm not a historian, low confidence:]

In his book A Culture of Growth, the influential economic historian Joel Mokyr claims that attempts to suppress intellectual activity in this period were somewhat toothless because they were rarely enforced consistently across countries. Intellectuals were thus able to evade suppression by moving, sometimes capitalizing on the competing interests of different rulers. More broadly, the book made me think that state-led conservative forces - including the Inquisition - were much less of a big deal than I had previously believed, even though they of course had some impacts including in some well-known cases such as the one you cite or the execution of Giordano Bruno.

So I wonder whether the fact that, for whatever reason, the movement was geographically tied to Spain is a crucial part of the full explanation here.

Comment by max_daniel on Why do social movements fail: Two concrete examples. · 2019-10-06T13:04:45.290Z · score: 6 (6 votes) · EA · GW
The lesson would seem to be something like: if you can, try to do things outside the political sphere; it is too unstable (??).

FWIW, my first thought was: "don't put all your eggs in one basket" / "don't rely on a single supplier of crucial resources." In fact, I almost immediately thought "this seems somewhat similar to EA relying a lot on a single source of funding, i.e., Open Phil."

[Not saying that I am in fact worried about this upon reflection, or that the analogy is useful. Just reporting my immediate reaction.]

Comment by max_daniel on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-05T10:00:52.188Z · score: 8 (4 votes) · EA · GW

Thank you, this speaks to some tentative concerns I had after reading the grant recommendations. FWIW, I feel that the following was particularly helpful for me deciding how much I trust the decision-making behind grant decisions, how willing I feel to donate to the Fund etc. - I think I would have liked to see this information in the top-level post.

Whenever there is a potential cause for a conflict of interest, the relevant person (or a separate fund member who suspects a COI) leaves a comment in the relevant row in the spreadsheet with the details of the COI, then the other fund members look at the description of the COI and decide whether it makes sense for that person to withdraw from voting.
Comment by max_daniel on The Long-Term Future: An Attitude Survey · 2019-09-17T22:41:46.583Z · score: 6 (2 votes) · EA · GW

(I think "if the contributive axiological value of people is negative, then preferring smaller populations is consistent with - indeed, implied by - totalism in population ethics" is a valid point, and obviously so. It is also mentioned by Spears in the paper I cite above. I therefore find it quite irritating that the parent comment was apparently strongly downvoted. Curious if I'm missing a reason for this?

NB I also think the point is trivial and has an implausible premise, but IMO it is the hallmark of good philosophy that each individual statement seems trivial - e.g., Reasons and Persons features an ample amount of such claims that might strike some readers as trivial or pedantic.)

Comment by max_daniel on The Long-Term Future: An Attitude Survey · 2019-09-16T22:38:36.238Z · score: 15 (7 votes) · EA · GW

Interesting, thanks for publishing!

Just a quick note: Your finding about population ethics - i.e. that many prefer smaller populations - is consistent with findings reported in the following paper.

Spears, D. (2017, 05). Making people happy or making happy people? Questionnaire-experimental studies of population ethics and policy. Social Choice and Welfare, 49(1), 145-169. doi:10.1007/s00355-017-1055-7

Among other things, Spears also finds that women prefer smaller populations more strongly than men.

I learned all of this from an unpublished literature review by David Althaus, which might be interesting for your purposes if you haven't seen it. [ETA: Actually David has publicly linked to the lit review - see subsection "An experimental study of population ethics and policy", pp. 23ff., for a summary of Spears (2017) - in this EA Forum post.]

(I'm not mentioning this to argue for any view. I'm very sympathetic to the total view in population ethics, and I agree with your interpretation that many subjects simply failed to understand the scenario in the intended way.)

Comment by max_daniel on Are we living at the most influential time in history? · 2019-09-15T13:27:14.048Z · score: 1 (1 votes) · EA · GW
It seems that this quote predicts a lower rate than there has ever† been before.

Just to make sure I understand - you're saying that, historically, the chance of funds (that were not intended just to advance mutual self-interest) being appropriated has always been higher than 2% per year?

If so, I'm curious what this is based on. - Do you have specific cases of appropriation in mind? Are you mostly appealing to charities with clear founding values and religious groups, both of which you mention later? [Asking because I feel like I don't have a good grasp on the probability we're trying to assess here.]

Comment by max_daniel on Are we living at the most influential time in history? · 2019-09-13T11:18:43.264Z · score: 2 (2 votes) · EA · GW

I think you make good points, and overall I feel quite sympathetic to the view you expressed. Just one quick thought pushing a bit in the other direction:

†† I'm excluding 'maximise profits' as a value!

But perhaps this example is quite relevant? To put it crudely, perhaps we can get away with keeping the value "do the most good" stable. This seems more analogous to "maximize profits" than to any specification of value that refers to a specific content of "doing good" (e.g., food aid to country X, or "abolish factory farming", or "reduce existential risk").

More generally, the crucial point seems to be: the content and specifics of values might change, but some of this change might be something we endorse. And perhaps there's a positive correlation between the likelihood of a change in values and how likely we'd be to agree with it upon reflection. [Exploring this fully seems quite complex both in terms of metaethics and empirical considerations.]

Comment by max_daniel on Alien colonization of Earth's impact the the relative importance of reducing different existential risks · 2019-09-08T14:33:58.811Z · score: 2 (2 votes) · EA · GW
Do you think that there are relatively probably existential catastrophes from rogue AI that would allow for alien colonization of Earth?

Yes, that's what I think.

First, consider the 'classic' Bostrom/Yudkowsky catastrophe scenario in which a single superintelligent agent with misaligned goals kills everyone and then, for instrumental reasons, expands into the universe. I agree that this would be a significant obstacle for alien civilization (though not totally impossible - e.g. there's some, albeit perhaps tiny, chance that an expanding alien civilization could be a more powerful adversary, or there could be some kind of trade, or ...).

However, I don't think we can be highly confident that this is how an existential catastrophe due to AI would look like. Cf. Christiano's What failure looks like, Drexler's Reframing Superintelligence, and also recent posts on AI risk arguments/scenarios by Tom Sittler and Richard Ngo. On some of the scenarios discussed there, I think it's hard to see whether they'd result in an obstacle to alien civilizations or not.

More broadly, I'd be wary to assign very high confidence to any feature of a post-AI catastrophe world. AI that could cause an existential catastrophe is a technology we don't currently possess and cannot anticipate in all its details - therefore, I think it's quite likely that an actual catastrophe based on such AI would in at least some ways have unanticipated properties, i.e., would at least not completely fall into any category of catastrophe we currently anticipate. Relatively robust high-level considerations such as Omohundro's convergent instrumental goal argument can give us good reasons to nevertheless assign significant credence to some properties (e.g., a superintelligent AI agent seems likely to acquire resources), but I don't think they suffice for >90% credence in anything.

Comment by max_daniel on Alien colonization of Earth's impact the the relative importance of reducing different existential risks · 2019-09-06T23:57:11.824Z · score: 2 (2 votes) · EA · GW

I think your reasoning is basically correct (as far as I can tell), at least your conclusion that "we should devote more effort into averting existential risks that make such colonization less likely" (given your premise that alien civs would be somewhat valuable). I feel like your appeal to evolution is a somewhat convincing first-pass argument for your premise that alien civilizations would likely instantiate value, though I wouldn't be that surprised if that premise turned out to be wrong after more scrutiny. I feel less sure about your claims regarding which catastrophes would make alien civs more or less likely, though I would agree at least crudely/directionally. (But e.g. I think there are many 'AI catastrophes' that would be quite compatible with alien civs.)

Anecdotally, I've frequently talked about this and similar considerations with EAs interested in existential risk strategy (or "cause prioritization" or "macrostrategy"). (I don't recall having talked about this issue with Nick Bostrom specifically, but I feel maybe 95% confident that he's familiar with it.) My guess is that the extent to which people have considered questions around aliens is significantly underrepresented in published written sources, though on the other hand I'm not aware of thinking that goes much beyond your post in depth.

Comment by max_daniel on Alien colonization of Earth's impact the the relative importance of reducing different existential risks · 2019-09-06T23:51:54.738Z · score: 3 (2 votes) · EA · GW

Regarding the likelihood (not the value) of intergalactic alien civilizations you might be intrested in this post on Quantifying anthropic effects on the Fermi paradox by Lukas Finnveden. E.g., he concludes:

If you accept the self-indication assumption, you should be almost certain that we’ll encounter other civilisations if we leave the galaxy. In this case, 95 % of the reachable universe will already be colonised when Earth-originating intelligence arrives, in expectation.
Comment by max_daniel on Are we living at the most influential time in history? · 2019-09-04T15:26:06.989Z · score: 5 (4 votes) · EA · GW

On a second thought, maybe what we should do is: take some person at ti (bracketing for a moment whether we draw someone uniformly at random, or take the one with most influence, or whatever) and then look at the difference between their actual actions (or the actions we'd expect them to take in the possible world we're considering if the values of the person are also determined by our sampling procedure) and the actions they'd take if we "intervene" to assume this person in fact was a longtermist altruist.

This definition would suggest that hinginess in the periods I mentioned wasn't that high: It's true that one of 70 people helping to hunt a bison made a big difference when compared to doing nothing; however, probably there is approximately zero difference between what that person has actually done and what they would have done if there had been a longtermist altruists: they'd have helped hunting a bison in both cases.

Comment by max_daniel on Are we living at the most influential time in history? · 2019-09-04T15:20:09.374Z · score: 3 (3 votes) · EA · GW

[Epistemic status: have never thought about this issue specifically in a focused way.]

I think as a super rough first pass it makes sense to think that, all else equal, smaller populations mean more hinginess.

I feel uncertain to what extent this is just because we should then expect any single person to own a greater share of total resources at some point in time. One extreme assumption would be that the relative distribution of resources at any given point in time is the prior for everyone's influence over the long-run future, perhaps weighted by how much they care about the long run. On that extreme assumption, this would probably mean that the maximum influence over all agents is higher today because global inequality is presumably higher than during population bottlenecks or in fact any past period. However, I think that assumption is too extreme: it's not the case that every generation can propagate their values indefinitely, with the share of their influence staying constant; for example, it might be that certain developments are determined by environmental conditions or other factors that are independent from any human's values. This turns on quite controversial questions around environmental/technological determinism that probably have a nuanced rather than simple answer.

Comment by max_daniel on Are we living at the most influential time in history? · 2019-09-04T11:22:15.070Z · score: 27 (14 votes) · EA · GW

Just a quick thought: I wonder whether the hingiest times were during periods of potential human population bottlenecks. E.g., Wikipedia says:

A 2005 study from Rutgers University theorized that the pre-1492 native populations of the Americas are the descendants of only 70 individuals who crossed the land bridge between Asia and North America.
[...]
In 2000, a Molecular Biology and Evolution paper suggested a transplanting model or a 'long bottleneck' to account for the limited genetic variation, rather than a catastrophic environmental change.[6] This would be consistent with suggestions that in sub-Saharan Africa numbers could have dropped at times as low as 2,000, for perhaps as long as 100,000 years, before numbers began to expand again in the Late Stone Age.

(Note that the Wikipedia article doesn't seem super well done, and also that it appears there has been significant scholarly controversy around population bottleneck claims. I don't want to claim that there in fact were population bottlenecks; I'm just curious what the implications in terms of hinginess would be if there were.)

As a first pass, it seems plausible to me that e.g. the action of any one of those 70 humans could have made the difference between this group surviving or not, with potentially momentous consequences. (What if the Vikings, or even later European colonialists, had found a continent without a human population?) Similarly, compared to any human today, if at some point the global human population really was just 2,000, then as a first pass - just based on a crude prior determined by the total population - it seems that one of these 2,000 people could have been enormously influential. Depending on how concentrated the population was and how much of a "close call" it was that modern humans didn't go extinct, it might even be the case that some of these people's actions had - without them realizing it - significant impacts on the probability of human survival (say, shifting the probability by more than 0.1%).

Some unstructured closing thoughts:

  • Scholars often use "history" in a narrow sense to refer to the period of time for which we have written descriptions. My impression is this would exclude periods of population bottlenecks - they'd all be in "prehistory." It's not clear to me if you intended to exclude prehistory based on the title of your post.
  • Even if there were drastic population bottlenecks and these were in fact the hingiest times, it's not clear what would follow from this. E.g., it might be defensible to claim that prehistory is outside the relevant reference class.
  • During a human population bottleneck, the distinction between "direct work" and "investing" on which your definitions rest might cease to make sense. Quite possibly, the best thing one of the 70 people in North America could have done is helping to hunt a bison or some other garden-variety action that helps the group survive - this seems good from the point of view of "longtermist altruism"/direct work, "investment", and selfish self-interest. This is a drastic example, but the direct work vs. investing distinction might also be quite blurry in less drastic times.
  • The potential example of 70 people settling North America also makes me wonder about the distribution of influence across people for any given period of time. Your definition currently talks about "a longtermist altruist living at ti" - but if different longtermist altruists would have vastly different amounts of influence at time ti, it becomes unclear how to understand this definition. Do I randomly draw a member of the human population at that time according to a uniform distribution, and then imagine they are a longtermist altruist? Do we refer to the person with the median influence? The maximum influence? Etc. (A more contemporary example: If I'm someone who could launch a nuclear weapon, then presumably I have a lot more influence than a poor peasant in the Chinese or Indian countryside. The latter observation points to a potential problem with spelling our your definition in terms of the median member of the world population: Todays "longtermist altruists" are very unusual people relative to the world population; it's not clear how much influence a rural farmer in China, India, or Bangladesh has today even if, say, the Bostrom/Yudkowsky story about AI is correct.)
Comment by max_daniel on Existential Risk and Economic Growth · 2019-09-03T13:40:34.799Z · score: 33 (13 votes) · EA · GW

I thought this was one of the most exciting pieces of research I've seen in the last few years. It also makes me really eager to see GPI hopefully making more work in a similar vein happen.

[Disclaimer: I co-organized the summer research fellowship, as part of which Leopold worked on this research, though I didn't supervise him.]

Comment by max_daniel on [Link] "Revisiting the Insights model" (Median Group) · 2019-07-16T08:24:44.978Z · score: 14 (6 votes) · EA · GW

Thanks for linkposting this as I might not have seen it otherwise. FWIW, my own intuition is that work like this is among the most marginally valuable things we can do. Here, "work like this" roughly means something like "build a legible model with implications for something that's clearly an important parameter when thinking about the long-term future and, crucially, have some way to empirically ground that model". However, I didn't look at this model in detail yet and so cannot currently comment on its specific predictions.

Comment by max_daniel on A case for strategy research: what it is and why we need more of it · 2019-07-11T16:35:06.472Z · score: 5 (4 votes) · EA · GW

Thank you for your response, David! One quick observation:

I think the idea cluster of existential risk reduction was formed through something I'd call "research". I think, in a certain way, we need more work of this type.

I agree that the current idea cluster of existential risk reduction was formed through research. However, it seems that one key difference between our views is: you seem to be optimistic that future research of this type (though different in some ways, as you say later) would uncover similarly useful insights, while I tend to think that the space of crucial considerations we can reliably identify with this type of research has been almost exhausted. (NB I think there are many more crucial considerations "out there", it's just that I'm skeptical we can find them.)

If this is right, then it seems we actually make different predictions about the future, and you could prove me wrong by delivering valuable strategy research outputs within the next few years.

Comment by max_daniel on Critique of Superintelligence Part 1 · 2019-07-08T07:32:48.058Z · score: 7 (3 votes) · EA · GW

[Sorry for picking out a somewhat random point unrelated to the main conversation. This just struck me because I feel like it's similar to a divergence in intuitions I often notice between myself and other EAs and particularly people from the 'rationalist' community. So I'm curious if there is something here it would be valuable for me to better understand.]

To give a silly human example, I'll name Tim Ferriss, who has used the skills of "learning to learn", "ignoring 'unwritten rules' that other people tend to follow", and "closely observing the experience of other skilled humans" to learn many languages, become an extremely successful investor, write a book that sold millions of copies before he was well-known, and so on. His IQ may not be higher now than when he begin, but his end results look like the end results of someone who became much more "intelligent".
Tim has done his best to break down "human-improving ability" into a small number of rules. I'd be unsurprised to see someone use those rules to improve their own performance in almost any field, from technical research to professional networking.

Here is an alternative hypothesis, a bit exaggerated for clarity:

  • There is a large number of people who try to be successful in various ways.
  • While trying to be successful, people tend to confabulate explicit stories for what they're doing and why it might work, for example "ignoring 'unwritten rules' that other people tend to follow".
  • These confabulations are largely unrelated to the actual causes of success, or at least don't refer to them in a way nearly as specific as they seem to do. (E.g., perhaps a cause could be 'practicing something in an environment with frequent and accurate feedback', while a confabulation would talk about quite specific and tangential features of how this practice was happening.)
  • Most people actually don't end up having large successes, but a few do. We might be pulled to think that their confabulations about what they were doing are insightful or worth emulating, but in fact it's all a mix of survivorship bias and people with certain innate traits (IQ, conscientiousness, perhaps excitement-seeking, ...) not occurring in the confabulations doing better.

Do you think we have evidence that this alternative hypothesis is false?

Comment by max_daniel on Effective Altruism is an Ideology, not (just) a Question · 2019-07-06T17:25:55.522Z · score: 2 (2 votes) · EA · GW

I already did this. - I was implicitly labelling this "double upvote" and was trying to say something like "I wish I could upvote this post even more strongly than with a 'strong upvote'". But thanks for letting me know, and sorry that this wasn't clear. :)

Comment by max_daniel on X-risks of SETI and METI? · 2019-07-04T13:19:42.671Z · score: 1 (1 votes) · EA · GW

I agree that information we received from aliens would likely spread widely. So in this sense I agree it would clearly be a potential info hazard.

It seems unclear to me whether the effect of such information spreading would be net good or net bad. If you see reasons why it would probably be net bad, I'd be curious to learn about them.

Comment by max_daniel on X-risks of SETI and METI? · 2019-07-04T06:53:59.594Z · score: 5 (2 votes) · EA · GW

([This is not a serious recommendation and something I might well change my mind about if I thought about it for one more hour:] Yes, though my tentative view is that there are fairly strong, and probably decisive, irreversability/option value reasons for holding off actions like SETI until their risks and benefits are better understood. NB the case is more subtle for SETI than METI, but I think the structure is the same: once we know there are aliens there is no way back to our previous epistemic state, and it might be that knowing about aliens is an info hazard.)

Comment by max_daniel on A case for strategy research: what it is and why we need more of it · 2019-07-03T14:23:29.725Z · score: 1 (1 votes) · EA · GW
These are all about AI (except maybe the one about China). Is that because you believe the easy and valuable wins are only there, or because you're most aware of those?

My guess is that AI examples were most salient to me because AI has been the area I've thought about the most recently. I strongly suspect there are easy wins in other areas as well.

Comment by max_daniel on Information security careers for GCR reduction · 2019-07-02T10:49:30.733Z · score: 4 (3 votes) · EA · GW

That's helpful, thanks!

Do you have a sense of whether the required talent is relatively generic quantitative/technical talent that would e.g. predict success in fields like computer science, physics, or engineering, or something more specific? And also what the bar is?

Currently I'm not sure if what you're saying is closer to "if you struggled with maths in high school, this career is probably not for you" or "you need to be at a +4 std level of ability in these specific things" (my guess is something closer to the former).

No worries if that was beyond the depth of your investigation.

Comment by max_daniel on Announcing the launch of the Happier Lives Institute · 2019-06-29T13:08:47.429Z · score: 1 (1 votes) · EA · GW

Hi Michael, thank you for your thoughtful reply. This all makes a lot of sense to me.

FWIW, my own guess is that explicitly defending or even mentioning a specific population ethical view would be net bad - because of the downsides you mention - for almost any audience other than EAs and academic philosophers. However, I anticipate my reaction being somewhat common among, say, readers of the EA Forum specifically. (Though I appreciate that maybe you didn't write that post specifically for this Forum, and that maybe it just isn't worth the effort to do so.) Waiting and checking if other people flag similar concerns seems like a very sensible response to me.

One quick reply:

Holding one's views on population ethics or the badness of death fixed, if one has a different view of what value is, or how it should be measured (or how it should be aggregated) that is clearly opens up scope for a new approach to prioritisation. The motivation to set up HLI came from the fact if we use self-reported subjective well-being scores are the measure of well-being, that does indicate potentially different priorities.

I agree I didn't make intelligible why this would be confusing to me. I think my thought was roughly:

(i) Contingently, we can have an outsized impact on the expected size of the total future population (e.g. by reducing specific extinction risks).

(ii) If you endorse totalism in population ethics (or a sufficiently similar aggregative and non-person-affecting view), then whatever your theory of well-being, because of (i) you should think that we can have an outsized impact on total future well-being by affecting the expected size of the total future population.

Here, I take "outsized" to mean something like "plausibly larger than through any other type of intervention, and in particular larger than through any intervention that optimized for any measure of near-term well-being". Thus, loosely speaking, I have some sense that agreeing with totalism in population ethics would "screen off" questions about the theory of well-being, or how to measure well-being - that is, my guess is that reducing existential risk would be (contingently!) a convergent priority (at least on the axiological, even though not necessarily normative level) of all bundles of ethical views that include totalism, in particular irrespective of their theory of well-being. [Of course, taken literally this claim would probably be falsified by some freak theory of well-being or other ethical view optimized for making it false, I'm just gesturing at a suitably qualified version I might actually be willing to defend.]

However, I agree that there is nothing conceptually confusing about the assumption that a different theory of well-being would imply different career priorities. I also concede that my case isn't decisive - for example, one might disagree with the empirical premise (i), and I can also think of other at least plausible defeaters such as claims that improving near-term happiness correlates with improving long-term happiness (in fact, some past GiveWell blog posts on flow-through effects seem to endorse such a view).

Comment by max_daniel on Information security careers for GCR reduction · 2019-06-28T22:31:35.635Z · score: 12 (8 votes) · EA · GW

I've seen some people advise against this career path, and I remember a comment by Luke elsewhere that he's aware of some people having that view. Given this, I'm curious if there are any specific arguments against pursuing a career in information security that you've come across?

(It's not clear to me that there must be any. - E.g. perhaps all such advice was based on opaque intuitions, or was given for reasons not specific to information security such as "this other career seems even better".)

Comment by max_daniel on Information security careers for GCR reduction · 2019-06-28T22:27:17.412Z · score: 11 (7 votes) · EA · GW

Could you elaborate on why you "expect the training [for becoming an information security professional] to be very challenging"?

Based on the OP, I could see the answer being any combination of the following, and I'm curious if you have more specific views.

  • a) The training and work is technically challenging.
  • b) The training and work has idiosyncratic aspects that may be psychologically challenging, e.g. the requirement to handle confidential information over extended periods of time.
  • c) The training and work requires an unusually broad combination of talents, e.g. both technical aptitude and the ability to learn to manage large teams.
  • d) You don't know of any specific reasons why the training would be challenging, but infer that it must be for structural reasons such as few people pursuing that career despite lucrative pay.
Comment by max_daniel on Effective Altruism is an Ideology, not (just) a Question · 2019-06-28T13:03:08.008Z · score: 20 (14 votes) · EA · GW

[Disclaimer: I used to be the Executive Director of the Foundational Research Institute, and currently work at the Future of Humanity Institute, both of which you mention in your post. Views are my own.]

Thank you so much for writing this! I wish I could triple-upvote this post. It seems to fit very well with some thoughts and unarticulated frustrations I've had for a while. This doesn't mean I agree with everything in the OP, but I feel excited about conversations it might start. I might add some more specific comments over the next few days.

[FWIW, I'm coming roughly from a place of believing that (i) at least some of the central 'ideological tenets' of EA are conducive to the community causing good outcomes, (ii) the overall ideological and social package of EA making me more optimistic about the EA community causing good outcomes per member than about any other major existing social and ideological package. However, I think these are messy empirical questions we are ultimately clueless about. And I do share a sense that in at least some conversations within the community it's not being acknowledged that these are debatable questions, and that the community's trajectory is being and will be affected by these implicit "ideological" foundations. (Even though I probably wouldn't have chosen the term "ideology".)

I do think that an awareness of EA's implicit ideological tenets sometimes points to marginal improvements I'd like the community to make. This is particularly true for more broadly investigating potential long-termist cause areas, including ones that don't have to do with emerging technologies. I also suspect that qualitative methodologies from the social sciences and humanities are currently being underused, e.g. I'd be very excited to see thoroughly conducted interviews with AI researchers and certain government staff on several topics.

Of course, all of this reflects that I'm thinking about this in a sufficiently outcome-oriented ethical framework.

My perception also is that within the social networks most tightly coalescing around the major EA organizations in Oxford and the Bay Area it is more common for people to be aware of the contingent "ideological" foundations you point to than one would maybe expect based on published texts. As a random example, I know of one person working at GPI who described themselves as a dualist, and I've definitely seen discussions around "What if certain religious views are true?" - in fact, I've seen many more discussions of the latter kind than in other mostly secular contexts and communities I'm familiar with.]