I'm keen for the language around this to convey the correct vibe about the epistemic status of the framework: currently I think this is "here are some dimensions that I and some other people feel like are helpful for our thinking". But not "we have well-validated ways of measuring any of these things" nor "this is definitely the most helpful carving up in the vicinity" nor "this was demonstrated to be helpful for building a theory of change for intervention X which did verifiably useful things". I think the animal names/pictures are kind of playful and help to convey that this isn't yet attempting to be in epistemically-solid land?
I guess I'm interested in the situations where you think an abbreviation would be helpful. Do you want someone to make an EA personality test based on this?
I think this is a good point which I wasn't properly appreciating. It doesn't seem particularly worse for (2) than for (1), except insofar as terminology is more locked in for (1) than (2).
Of course, a possible advantage of "clueless" is that it strikes a self-deprecating tone; if we're worried about being perceived as arrogant then having the language err on the side of assigning blame to ourselves rather than the universe might be a small help
I think that bare terms like "unpredictability" or particularly "uncertainty" are much too weak; they don't properly convey the degree of epistemic challenge, and hence don't pick out what's unusual about the problem situation that we're grappling with.
"Unforseeability" is a bit stronger, but still seems rather too weak. I think "unknowability", "radical uncertainty", and "cluelessness" are all in the right ballpark for their connotations.
I do think "unknowability" for (2) and "absolute/total unknowability" for (1) is an interesting alternative. Using "unknowable" rather than "clueless" puts the emphasis on the decision situation rather than the agent; I'm not sure whether that's better.
To me it sounds slightly odd to use the word "clueless" for (2), however, given the associations that word has (cf. Cambridge dictionary).
In everyday language I actually think this fits passably well. The dictionary gives the definition "having no knowledge of something". For (2) I feel like informally I'd be happy with someone saying that the problem is we have no knowledge of how our actions will turn out, so long as they clarified that they didn't mean absolutely no knowledge. Of course this isn't perfect; I'd prefer they said "basically no knowledge" in the first place. But it's also the case that informally "clueless" is often modified with superlatives (e.g. "totally", "completely"), so I think that a bare "clueless" doesn't really connote having no idea at all.
(1) is not a gradable concept - if we're clueless, then in Hilary Greaves' words, we "can never have even the faintest idea" which of two actions is better.
(2), on the other hand, is a gradable concept - it can be more or less difficult to find the best strategies. Potentially it would be good to have a term that is gradable, for that reason.
I appreciate you making this distinction. Although I find that it all the more makes me want to use one term (e.g. clueless) for (2), and a modified version (absolutely clueless, or totally clueless, or perhaps infinitely clueless) for (1). I think that the natural relation between the two concepts is that (1) is something like a limiting case of (2) taken to the extreme, so it's ideal if the terminology reflects that.
A couple of other remarks around this:
I think the fact that "totally clueless" is a common English phrase suggests that "clueless" is grammatically seen as a gradable concept.
I agree that in principle (2) is a gradable concept so we might want to have language that can express it.
In practice my instinct is that most of the time one will point at the problem and put attention on possible responses, and it won't be that helpful to discuss exactly how severe the problem is.
However, I like the idea that being able to express gradations might make it easier to notice the concept of gradations.
I can dream that eventually we could find a natural metric for degree-of-cluelessness ...
One possibility is something relating to (un)predictability or (un)foreseeability. That has the advantage that it relates to forecasting.
Hmm, I'm unsure whether the link to forecasting is more of an advantage or a disadvantage. It's suggestive of the idea that one deals with the problem by becoming better at forecasting, which I think is something which is helpful, but probably only a small minority of how we should address it.
I suggest that (1) should be called "the problem of absolute cluelessness" and that (2) should be called "the practical problem of cluelessness".
When context is clear one could drop the adjective. My suspicion is that with time (1) will come to be regarded as a solved problem, and (2) will still want a lot of attention. I think it's fine/desirable if at that point it gets to use the pithier term of "cluelessness". I also think that it's probably good if (1) and (2) have names which make it clear that there's a link between them. I think there may be a small transition cost from current usage, but (a) there just isn't that much total use of the terms now, and (b) current usage seems inconsistent about whether it includes (2).
As a clarification: I don't think "here are some good effects that would come out of getting lots of upvotes" would count as such an argument.
I am now feeling like the legitimate use cases for such arguments might be narrow enough, and their benefits small enough, that it might be better to have a norm that disallows them, for the sake of being a cleaner rule. Or maybe it should be okay to make arguments so long as you explicitly cancel any implicature that you're asking people to upvote? Confused about what's best here.
I didn't downvote (because as you say it's providing relevant information), but I did have a negative reaction to the comment. I think the generator of that negative reaction is roughly: the vibe of the comment seems more like a political attempt to close down the conversation than an attempt to cooperatively engage. I'm reminded of "missing moods"; it seems like there's a legitimate position of "it would be great to have time to hash this out but unfortunately we find it super time consuming so we're not going to", but it would naturally come with a mood of sadness that there wasn't time to get into things, whereas the mood here feels more like "why do we have to put up with you morons posting inaccurate critiques?". And perhaps that's a reasonable position, but it at least leaves a kind of bad taste.
Yeah my quick guess is that (as for many complex skills) g is very helpful, but that it's very possible to be high g without being very good at the thing I'm pointing at (partially because feedback loops are poor, so people haven't necessarily has a good training signal for improving).
I guess I significantly agree with all of the above, and I do think it would have been reasonable for me to mention these considerations. But since I think the considerations tend to blunt rather than solve the issues, and since I think the audience for my post will mostly be well aware of these considerations, it still feels fine to me to have omitted mention of them? (I mean, I'm glad that they've come up in the comments.)
I guess I'm unsure whether there's an interesting disagreement here.
Yeah, I totally agree that if you're much more sophisticated than your (potential) donors you want to do this kind of analysis. I don't think that applies in the case of what I was gesturing at with "~community projects", which is where I was making the case for implicit impact markets.
Assuming that the buyers in the market are sophisticated:
in the straws case, they might say "we'll pay $6 for this output" and the straw org might think "$6 is nowhere close to covering our operating costs of $82,000” and close down
I think too much work is being done by your assumption that the cost effectiveness can't be increased. In an ideal world, the market could create competition which drives both orgs to look for efficiency improvements
This kind of externality should be accounted for by the market (although it might be that the modelling effectively happens in a distributed way rather than anyone thinking about it all).
So you might get VCs who become expert in judging when early-stage projects are a good bet. Then people thinking of starting projects can somewhat outsource the question to the VCs by asking "could we get funding for this?"
Moral trade is definitely relevant here. Moral trade basically deals with cases with fundamental-differences-in-values (as opposed to coordination issues from differences in available information etc.).
I haven't thought about this super carefully, but it seems like a nice property of impact markets is that they'll manage to simultaneously manage the moral trade issues and the coordination issues. Like in the example of donors wishing to play donor-of-last-resort it's ambiguous whether this desire is driven by irreconcilably different values or different empirical judgements about what's good.
I agree that these considerations would blunt the coordination issues some.
So I think that a proposal for "Implicit impact markets without infrastructure" should probably include as one element a reminder for people to take these considerations into account.
I guess I think that it should include that kind of reminder if it's particularly important to account for these things under an implicit impact markets set-up. But I don't think that; I think they're important to pay attention to all of the time, and I'm not in the business (in writing this post) of providing reminders about everything that's important.
In fact I think it's probably slightly less important to take them into account if you have (implicit or explicit) impact markets, since the markets would relieve some of the edge that it's otherwise so helpful to blunt via these considerations.
Yeah, Shapley values are a particular instantiation of a way that you might think the implicit credit split would shake out. There are some theoretical arguments in favour of Shapley values, but I don't think the case is clear-cut. However in practice they're not going to be something we can calculate on-the-nose, so they're probably more helpful as a concept to gesture with.
Of course "non-EA funding" will vary a lot in its counterfactual value. But roughly speaking I think that if you are pulling in money from places where it wouldn't have been so good, then on the implicit impact markets story you should get a fraction of the credit for that fundraising. Whether or not that's worth pursuing will vary case-to-case.
Basically I agree with Michael that it's worth considering but not always worth doing. Another way of looking at what's happening is that starting a project which might appeal to other donors creates a non-transferrable fundraising opportunity. Such opportunities should be evaluated, and sometimes pursued.
I agree that in principle that you could model all of this out explicitly, but it's the type of situation where I think explicit modelling can easily get you into a mess (because there are enough complicated effects that you can easily miss something which changes the answer), and also puts the cognitive work in the wrong part of the system (the job of funders is to work out what would be the best use of their resources; the job of the charities is to provide them with all relevant information to help them make the best decision).
I think impact markets (implicit or otherwise) actually handle this reasonably well. When you're starting a charity, you're considering investing resources in pursuit of a large payoff (which may not materialise). Because you're accepting money to do that, you have to give up a fraction of the prospective payoff to the funders. This could change the calculus of when it's worth launching something.
I like the jumping in! I think using vignettes as a starting point for discussion of norms has some promise.
In these cases, I imagine it being potentially fruitful to have more-discussion-per-vignette about both whether the idea captured is a good one (I think it's at least unclear in some of your examples), as well as how good it would be if the norm were universalised ... we don't want to spend too much attention on promoting norms that while positive just aren't a very big deal.
Maybe we should try to set default expectations of how much credit for a project goes to different contributors? With the idea that not commenting is a tacit endorsement that the true credit split is probably in that ballpark (or at least that others can reasonably read it that way).
One simple suggestion might be four equal parts credits: to founding/establishing the org and setting it in a good direction (including early funders and staff); to current org leadership; to current org staff; to current funders. I do expect substantial deviation from that in particular cases, but it's not obvious to me that any of the buckets is systematically too big or too small, so maybe it's reasonable as a starting point?
Inefficiencies from inconsistent estimates of value
Broadening from just considering donations, there's a worry that the community as a whole might be able to coordinate to get better outcomes than we're currently managing. For instance opinions about the value of earning to give vary quite a bit; here's a sketch to show how that can go wrong:
This doesn't quite fit in the hierarchy of approaches to donor coordination, but it is one of the issues that fully explicit impact markets should be able to help resolve. How much would implicit impact markets help? Maybe if they were totally implicit and "strengths of ask" were always made qualitatively rather than quantitatively it wouldn't help so much (since everyone would understand "strength" relative to what they think of as normal for the importance of money or direct work, and Alice and Beth have different estimates of that 'normal'). But if a fraction of projects move to providing quantitative estimates (while still not including any formal explicit market mechanisms), that might be enough to relieve the inefficiencies.
Definitely didn't mean to shut down conversation! I felt like I had a strong feeling that it was not an option on the table (because of something like coherence reasons -- cf. my reply to Jonas -- not because it seemed like a bad or too-difficult idea). But I hadn't unpacked my feeling. I also wasn't sure whether I needed to, or whether when I posted everyone would say something like "oh, yeah, sure" and it would turn out to be a boring point. This was why I led with "I don't know how much of an outlier I am"; I was trying to invite people to let me know if this was a boring triviality after it was pointed out, or if it was worth trying to unpack.
P.S. I appreciate having what seemed bad about the phrasing pointed out.
Hmm, no, I didn't mean something that feels like pessimism about coordination ability, but that (roughly speaking) thing you get if you try to execute a "change the name of the movement" operation is not the same movement with a different name, but a different (albeit heavily overlapping) movement with the new name. And so it's better understood as a coordinated heavy switch to emphasising the new brand than it is just a renaming (although I think the truth is actually somewhere in the middle).
I don't think that's true if the name change is minor so that the connotations are pretty similar. I think that switching from "effective altruism" to "efficient do-gooding" is a switch which could more or less happen (you'd have a steady trickle of people coming in from having read old books or talked to people who were familiar with the old name, but "effective altruism, now usually called efficient do-gooding" would mostly work). But the identity of the movement is (at least somewhat) characterised by its name and how people understand it and relate to it. If you shifted to a name like "global priorities" with quite different connotations, I think that it would change people's relationship with the ideas, and you would probably find a significant group of people who said "well I identify with the old brand, but not with the new brand", and then what do you say to them? "Sorry, that brand is deprecated" doesn't feel like a good answer.
(I sort of imagine you agree with all of this, and by "change the name of the movement" you mean something obviously doable like getting a lot of web content and orgs and events and local groups to switch over to a new name. My claim is that that's probably better conceived of in terms of its constituent actions than in terms of changing the name of the movement.)
I don't know how much of an outlier I am, but I feel like "change the name of the movement" is mostly not an option on the table. Rather there's a question about how much (or when) to emphasise different labels, with the understanding that the different labels will necessarily refer to somewhat different things. (This is a different situation than an organisation considering a rebrand; in the movement case people who preferred the connotations of the older label are liable to just keep using it.)
Anyhow, I like your defence of "effective altruism", and I don't think it should be abandoned (while still thinking that there are some contexts where it gets used but something else might be better).
Maybe the obvious suggestion then is "new enlightenment"? I googled, and the term has some use already (e.g. in a talk by Pinker), but it feels pretty compatible with what you're gesturing at. I guess it would suggest a slightly broader conception (more likely to include people or groups not connected to the communities you named), but maybe that's good?
Thanks, makes sense. This makes me want to pull out the common characteristics of these different groups and use those as definitional (and perhaps realise we should include other groups we're not even paying attention to!), rather than treat it as a purely sociological clustering. Does that seem good?
Like maybe there's a theme about trying to take the world and our position in it seriously?
Could you say a little more about the context(s) where a name seems useful?
(I think it's often easier to think through what's wanted from a name when you understand the use case, and sometimes when you try to do this you realise it was a slightly different thing that you really wanted to name anyway.)
Note that I think that the mechanisms I describe aren't specific to economics, but cover academic research generally-- and will also include most of how most AI safety researchers (even those not in academia) will have impact.
There are potentially major crux moments around AI, so there's also the potential to do an excellent job engineering real transformative systems to be safe at some point (but most AI safety researchers won't be doing that directly). I guess that perhaps the indirect routes to impact for AI safety might feel more exciting because they're more closely connected to the crucial moments -- e.g. you might hope to set some small piece of the paradigm that the eventual engineers of the crucial systems are using, or hope to support a culture of responsibility among AI researchers, to make it less likely that people at the key time ignore something they shouldn't have done.
Finally, I imagine quant trading is a non-starter for a longtermist who is succeeding in academic research. As a community, suppose we already have significant ongoing funding from 3 or so of the world's 3k billionaires. What good is an extra one-millionaire? Almost anyone's comparative advantage is more likely to lie in spending the money, but even more so if one can do so within academic research.
It seems quite wrong to me to present this as so clear-cut. I think if we don't get major extra funding the professional longtermist community might plateau at a stable size in perhaps the low thousands. A successful quantitative trader could support several more people at the margin (a very successful trader could support dozens). If you're a good fit for the crowd, it might also be a good group to network with.
If you're particularly optimistic about future funding growth, or pessimistic about community growth, you might think it's unlikely we end up in that world in a realistic timeframe, but there's likely to still be some hedging value.
To be clear, I mostly wouldn't want people in the OP's situation to drop the PhD to join a hedge fund. But it's worth understanding that e.g. the main routes to impact in academic research are probably:
Providing leadership for the academic field from within the field, including:
Helping students orient to what's important, and providing space for them to work on more important projects
Using academia as a springboard to affect non-academic projects (e.g. being an advisor on particular policy topics, or providing solid support for claims that are broadly useful)
I think for some people those just aren't going to be a great personal fit (even if they can achieve conventional "success" in academia!), so it's worth considering other options.
In this particular case, I'm kind of excited about getting more longtermist economists. But it might depend e.g. how disillusioned the OP is with the field as to whether it might make sense for them to be such a person.
I guess I wouldn't recommend the donor lottery to people who wouldn't be happy entering a regular lottery for their charitable giving (but I would usually recommend them to be happy with that regular lottery!).
Btw, I'm now understanding your suggestions as not really alternatives to the donor lottery, since I don't think you buy into its premises, but alternatives to e.g. EA Funds.
(In support of the premise of respecting individual autonomy about where to allocate money: I think that making requests to pool money in a way that rich donors expect to lose control would risk making EA pattern match at a surface level to a scam, and might drive people away. For a more extreme version of this, imagine someone claiming that as soon as you've decided to donate some money you should send it all to the One True EA Collective fund, so that it can be fairly distributed, and it would be a weird propagation of wealth to allow rich people to take any time to think about where to give their money; whether or not you think an optimal taxation system would equalise wealth much more, I think it's fairly clear that the extreme bid that everyone pool donations would be destructive because it would put off donors.
By dominant action I mean "is ~at least as good as other actions on ~every dimension, and better on at least one dimension".
My confusion is something like: there's no new money out there! Its a group of donors deciding to give individually or give collectively. So the perspective of "what will lead to optimal allocation of resources at the group level?" is the right one.
I don't think donor lotteries are primarily about collective giving. As a donor lottery entrant, I'd be just as happy giving $5k for a 5% chance of controlling a $100k pot of pooled winnings as entering a regular lottery where I could give $5k for a 5% chance of winning $100k (which I would then donate)*. In either case I think I'll do more than 20x as much good with $100k than $5k (mostly since I can spend longer thinking and investigating), so it's worthwhile in expectation.
* Except that I usually don't have good access to that kind of lottery (maybe there would also be tax implications, although perhaps it's fine if the money is all being donated). So the other donors are a logistical convenience, but not an integral part of the idea.
My understanding is that past people selected to allocate the pool haven't tended to delegate that allocation power. And indeed if you're strongly expecting to do so, why not just give the allocation power to that person beforehand, either over your individual donation (e.g. through an EA fund) or over a pool. Why go through the lottery stage?
I don't know that they should strongly expect to do so. But in any case the reason for going through the lottery stage is simple: maybe you'd want to take 50 hours thinking about whether to delegate and to whom, and vetting possible people to delegate to. That time might not be worth spending for a $5k donation, but become worth spending for a $100k donation. (Additionally the person you want to delegate to might be more likely to take the duty seriously for a larger amount of money.)
I think your analysis of the alternatives is mostly from the perspective of "what will lead to optimal allocation of resources at the group level?"
But the strongest case for donor lotteries, in my view, isn't in these terms at all. Rather, it's that entering a lottery is often a dominant action from the perspective of the individual donor (if most other things they would consider giving to don't exhibit noticeably diminishing returns over the amount they are attempting to get in the lottery). The winner of a lottery need not be the allocator for the money; they can instead e.g. decide to take longer thinking about whom they want to delegate allocation power to (I actually think this might often be the "technically correct" move; I don't know how often lottery winners act this way). This dominance argument would go through for a much smaller proportion of possible donors for your alternatives. I'm interested if you see another reason that people would donate to these?
I spent a little while thinking about this. My guess is that of the activities I list:
Alice and Bob's efforts look comparable to donating (in external benefit/effort) when the longtermist portfolio is around $100B-$1T/year
Clara's efforts looks comparable to donating when the longtermist portfolio is around $1B-$10B/year
Diya's efforts look comparable to donating when the longtermist portfolio is around $10B-$100B/year
Elmo's efforts are harder to say because they're closer to directly trying to grow longtermist support, so the value diminishes as the existing portfolio gets larger just as for donations, and it more depends on underlying quality
All of those numbers are super crude and I might well disagree with myself if I came back later and estimated again. They also depend on lots of details (like how good the individuals are at executing on those strategies).
Perhaps most importantly, they're excluding the internal benefits -- if these activities are (as I suggest) partly good for practicing some longtermist judgement, then I'd really want to see them as a complement to donation rather than just a competitor.
One argument goes via something like the reference class of global autopoeitic information-processing systems: life has persisted since it started several billion years ago; multicellular life similarly; sexual selection similarly. Sure, species go extinct when they're outcompeted, but the larger systems they're part of have only continued to thrive.
The right reference class (on this story) is not "humanity as a mammalian species" but "information-based civilization as the next step in faster evolution". Then we might be quite optimistic about civilization in some meaningful sense continuing indefinitely (though perhaps not about particular institutions or things that are recognisably human doing so).
Some fixed models also support macroscopic probabilities of indefinite survival: e.g. if in each generation each individual has a number of descendants drawn from a Poisson distribution with parameter 1.1, then there's a finite chance of extinction in each generation but these diminish fast enough (as the population gets enormous) that if you make it through an initial rocky period you're pretty much safe.
That model is clearly too optimistic because it doesn't admit crises with correlated problems across all the individuals in a generation. But then there's a question about how high is the unavoidable background rate of such crises (i.e. ones that remain even if you have a very sophisticated and well-resourced attempt to prevent them).
On current understanding I think the lower bounds for the rate of exogenous such events rely on things like false vacuum decay (and maybe GRBs while we're local enough), and those lower bounds are really quite low, so it's fairly plausible that the true rate is really low (though also plausible it's higher because there are risks that aren't observed/understood).
Bounding endogenous risk seems a bit harder to reason about. I think that you can give kind of fairytale/handwaving existence proofs of stable political systems (which might however be utterly horrific to us). Then it's at least sort of plausible that there would be systems which are simultaneously extremely stable and also desirable.
I want the world in 30 years time to be in as good a state as it can be in order to face whatever challenges that will come next.
I like this! I sometimes use a perspective which is pretty close (though often think about 50 years rather than 30 years, and hold it in conjunction with "what are the challenges we might need to face in the next 50 years?"). I think 30 vs 50 years is a kind-of interesting question. I've thought about 50 because if I imagine e.g. that we're going to face critical junctures with the development of AI in 40 years, that's within the scope where I can imagine it being impacted by causal pathways that I can envision -- e.g. critical technology being developed by people who studied under professors who are currently students making career decisions. By 60 years it feels a bit too tenuous for me to hold on to.
I kind of agree that if looking at policy specifically a shorter time horizon feels good.
I have two different responses (somewhat in tension with each other):
Finding "everyday" things to do will necessitate identifying what's good to do in various situations which aren't the highest-value activity an individual can be undertaking
This is an important part of deepening the cultural understanding of longtermism, rather than have all of the discussion be about what's good to do in a particular set of activities that's had strong selection pressure on it
This is also important for giving people inroads to be able to practice different aspects of longtermism
I think it's a bit like how informal EA discourse often touches on how to do everyday things efficiently (e.g. "here are tips for batching your grocery shopping") -- it's not that these are the most important things to be efficient about, but that all-else-equal it's good, and it's also very good to give people micro-scale opportunities to put efficiency-thinking into practice
Note however that my examples would be better if they had more texture:
Discussion of the nuance of better or worse versions of the activities discussed could be quite helpful for conveying the nuance of what is good longtermist action
To the extent that these are far from the highest value activities those people could be undertaking, it seems important to be up-front about that: keeping tabs on what's relatively important is surely an important part of the (longtermist) EA culture
I'm not sure how much I agree with "probably much less positive than some other things that could be done even by "regular people', even once there are millions or tens of millions of longtermists"
I'd love to hear your ideas for things that you think would be much more positive for those people in that world
My gut feeling is that they are at the level of "competitive uses of time/attention (for people who aren't bought into reorienting their whole lives) by the time there are tens of millions of longtermists"
It seems compatible with that feeling that there could be some higher-priority things for them to be doing as well -- e.g. maybe some way of keeping immersed in longtermist culture, by being a member of some group -- but that those reach saturation or diminishing returns
I think I might be miscalibrated about this; think it would be easier to discuss with some concrete competition on the table
Of course to the extent that these actually are arguably competitive actions, if I believe my first point, maybe I should have been looking for even more everyday situations
e.g. could ask "what is the good longtermist way to approach going to the shops? meeting a romantic partner's parents for the first time? deciding how much to push yourself to work when you're feeling a bit unwell?"
Be more cooperative. (There are arguments about increasing cooperation, especially from people working on reducing S-risks, but I couldn't find any suitable resource in a brief search)
Take a strong stance against narrow moral circles.
Have a good pitch prepared about longtermism and EA broadly. Balance confidence with adequate uncertainty.
Have a well-structured methodology for getting interested acquaintances more involved with EA.
Help friends in EA/longtermism more.
Strengthen relationships with friends who have a high potential to be highly influential in the future.
I basically like all of these. I think there might be versions which could be bad, but they seem like a good direction to be thinking in.
I'd love to see further exploration of these -- e.g. I think any of your six suggestions could deserve a top-level post going into the weeds (& ideally reporting on experiences from trying to implement it). I feel most interested in #3, but not confidently so.
I think that the suggestions here, and most of the arguments, should apply to "Everyday EA " which isn't necessarily longtermistic. I'd be interested in your thoughts about where exactly should we make a distinction between everyday longtermist actions and non-longtermist everyday actions.
I agree that quite a bit of the content seems not to be longtermist-specific. But I was approaching it from a longtermist perspective (where I think the motivation is particularly strong), and I haven't thought it through so carefully from other angles.
I think the key dimension of "longtermism" that I'm relying on is the idea that the longish-term (say 50+ years) indirect effects of one's actions are a bigger deal in expectation than the directly observable effects. I don't think that that requires e.g. any assumptions about astronomically large futures. But if you thought that such effects were very small compared to directly observable effects, then you might think that the best everyday actions involved e.g. saving money or fundraising for charities you had strong reason to believe were effective.
I have a sense that a large part of the success of scientific norms comes down to their utility being immediately visible.
I agree with this. I don't think science has the attractor property I was discussing, but it has this other attraction of being visibly useful (which is even better). I was trying to use science as an example of the self-correction mechanism.
Or perhaps I am having a semantic confusion: is science self-propagating in that scientists, once cultivated, go on to cultivate others?
Yes, this is the sense of self-propagating that I intended.
In my words, what you've done is point out that approximate-consequentialism + large-scale preferences is an attractor.
I think that this is a fair summary of my first point (it also needs enough truth seeking to realise that spreading the approach is valuable). It doesn't really speak to the point about being self-correcting/improving.
I'm not trying to claim that it's obviously the strongest memeplex in the long term. I'm saying that it has some particular strengths (which make me more optimistic than before I was aware of those strengths).
I think another part of my thinking there is that actually quite a lot of people have altruistic preferences already, so it's not like trying to get buy-in for a totally arbitrary goal.