"To see the world as it is, rather than as I wish it to be."
I work for the EA research nonprofit Rethink Priorities. Despite my official title, I don't really think of the stuff I do as "research." In particular, when I think of the word "research", I think of people who are expanding the frontiers of the world's knowledge, whereas often I'm more interested in expanding the frontiers of my knowledge, and/or disseminating it to the relevant parties.
I'm also really interested in forecasting.
People may or may not also be interested in my comments on Metaculus and Twitter:
Hi. I'm glad you appear to have gained a lot from my quick reply, but for what it's worth I did not intend my reply as an admonishment.
I think the core of what I read as your comment is probably still valid. Namely, that if I misidentified problems as biases when almost all of the failures are due to either a) noise/error or b) incompetence unrelated to decision quality (eg mental health, insufficient technical skills, we aren't hardworking enough), then the bias identification isn't true or useful. Likewise, debiasing is somewhere between neutral to worse than useless if the problem was never bias to begin with.
I'm suspicious of 1), especially if taken too far, because I think if taken too far it would justify way too much complacency in worlds where foreseeable moral catastrophes are not only possible but probable.
Some quick thoughts: I would guess that Open Phil is better at this than other EA orgs, both because of individually more competent people and much better institutional incentives (ego not wedded to specific projects working). For your specific example, I'm (as you know) new to AI governance, but I would naively guess that most (including competence-weighted) people in AI governance are more positive about AI interventions than you are.
Happy to be corrected empirically.
(I also agree with Larks that publishing a subset of these may be good for improving the public conversation/training in EA, but I understand if this is too costly and/or if the internal analyses embed too much sensitive information or models)
You might also like Aaron Schwartz's notes on productivity:
Assigned problems are problems you’re told to work on. Numerous psychology experiments have found that when you try to “incentivize” people to do something, they’re less likely to do it and do a worse job. External incentives, like rewards and punishments, kills what psychologists call your “intrinsic motivation” — your natural interest in the problem. (This is one of the most thoroughly replicated findings of social psychology — over 70 studies have found that rewards undermine interest in the task.)5 People’s heads seem to have a deep avoidance of being told what to do.6
[LZ Sidenote: I think I'd want to actually read the studies or at least a meta-analysis of recent replications first before being sure of this]
The weird thing is that this phenomenon isn’t just limited to other people — it even happens when you try to tell yourself what to do! If you say to yourself, “I should really work on X, that’s the most important thing to do right now” then all of the sudden X becomes the toughest thing in the world to make yourself work on. But as soon as Y becomes the most important thing, the exact same X becomes much easier.
Create a false assignment
This presents a rather obvious solution: if you want to work on X, tell yourself to do Y. Unfortunately, it’s sort of difficult to trick yourself intentionally, because you know you’re doing it.7 So you’ve got to be sneaky about it.
One way is to get someone else to assign something to you. The most famous instance of this is grad students who are required to write a dissertation, a monumentally difficult task that they need to do to graduate. And so, to avoid doing this, grad students end up doing all sorts of other hard stuff.
The task has to both seem important (you have to do this to graduate!) and big (hundreds of pages of your best work!) but not actually be so important that putting it off is going to be a disaster.
Hmm, did you read the asterisk in the quoted comment?
*The natural Gricean implicature of that claim is that I'm saying that EA orgs are an exception. I want to disavow that implication. For context, I think this is plausibly the second or third biggest limitation for my own work.
(No worries if you haven't, I'm maybe too longwinded and it's probably unreasonable to expect people to carefully read everything on a forum post with 76 comments!)
If you've read it and still believe that I "sound breathtakingly arrogant ", I'd be interested in whether you can clarify whether "breathtakingly arrogant" means either a) what I say is untrue or b) what I say is true but insufficiently diplomatic.
More broadly, I mostly endorse the current level of care and effort and caveats I put on the forum. (though I want to be more concise, working on it!)
I can certainly make my writing more anodyne and less likely to provoke offense, e.g. by defensive writing and pre-empting all objections I can think of, by sprinkling the article heavily with caveats throughout, by spending 3x as much time on each sentence, or just by having much less public output (the last of which is empirically what most EAs tend to do).
I suspect this will make my public writing worse however.
Thanks a lot! Is there a writeup of this somewhere? I tend to be a pretty large fan of explicit rationality (at least compared to EAs or rationalists I know), so evidence that reasoning in this general direction is empirically kind of useless would be really useful to me!
Yeah I'm surprised by this as well. Both classical utilitarianism (in the extreme version, "everything that is not morally obligatory is forbidden") and longtermism just seem to have many lower degrees of freedom than other commonly espoused ethical systems, so it would naively be surprising if these worldviews can justify a broader range of actions than close alternatives.
I think what you said is insightful and worth considering further. Nonetheless, I will only address a specific subpoint for now, and revisit this later.
Basically, have you considered the perspective that “some EA orgs aren’t very good” to be a better explanation for the problems?
Hmm I'm not sure what you mean, and I think it's very likely we're talking about different problems. But assuming we're talking about the same problems, at a high-level any prediction problem can be decomposed to bias vs error (aka noise, aka variance).
I perceive that many of the issues I've mentioned to be better explained by bias than error. In particular I just don't think we'll see equivalently many errors in the opposite direction. This is an empirical question however, and I'd be excited to see more careful followups to test this hypothesis.
(as a separate point, I do think some EA orgs aren't very good, with "very good" defined as I'd rather the $s be spent on their work rather than in Open Phil coffers, or my own bank account. I imagine many other EAs would feel similarly about my own work).
Speaking for myself, I think I have strong ideological reasons to think that predictably doing (lots of) good is possible.
I also have a bias towards believing that things that are good for X reason are also good for Y reason, and this problem rears itself up even when I try to correct for it. E.g. I think Linch(2012-2017) too easily bought in to the "additional consumption $s don't make you happier " narratives, and I'm currently lactovegetarian even though I started being vegetarian for very different reasons than what I currently believe to be the most important. I perceive other EAs as on average worse at this (not sure the right term. Decoupling?) than me, which is not necessarily true of the other biases on this list.
A specific instantiation of this is that it's easier for me to generate solutions to problems that are morally unambiguous by the standards of non-EA Western morality, even though we'd expect the tails to come apart fairly often.
To a lesser extent, I have biases towards thinking that doing (lots of) good comes from things that I and my friends are predisposed to be good at (eg cleverness, making money).
Another piece of evidence is that EAs seem far from immune from ideological capture for non-EA stuff. My go-to example is the SSC/NYT thing.
(I'm exaggerating my views here to highlight the differences, I think my all-things-considered opinion on these positions are much closer to yours than the rest of the comment will make it sound)
I think my strongest disagreement with your comment is the framing here:
I'm not sure how you're distinguishing between the two hypotheses:
Longtermists don't like quantitative modeling in general.
Longtermist questions are not amenable to quantitative modeling, and so longtermists don't do much quantitative modeling, but they would if they tackled questions that were amenable to quantitative modeling.
(Unless you want to defend the position that longtermist questions are just as easy to model as, say, those in global poverty? That would be... an interesting position.)
If we peel away the sarcasm, I think the implicit framing is that
If X is less amenable than Y to method A of obtaining truth, and X is equally or more amenable to methods B, C, and D relative to Y, we should do less method A to obtain truth in X (relative to Y), and more methods B, C, and D.
X is less amenable than Y to method A of obtaining truth.
Thus, we should use method A less in X than in Y.
Unless I'm missing something, I think this is logically invalid. The obvious response here is that I don't think longtermist questions are more amenable to explicit quantitative modeling than global poverty, but I'm even more suspicious of other methodologies here.
Medicine is less amenable to empirical testing than physics, but that doesn't mean that clinical intuition is a better source of truth for the outcomes of drugs than RCTs. (But medicine is relatively much less amenable to theorems than physics, so it's correct to use less proofs in medicine than physics.)
More minor gripes:
(and I'll note that even GiveWell isn't doing this).
I think I'm willing to bite the bullet and say that GiveWell (or at least my impression of them from a few years back) should be more rigorous in their modeling. Eg, weird to use median staff member's views as a proxy for truth, weird to have so few well-specified forecasts, and so forth.
The first piece of advice in that post is to use techniques like assumption based planning, exploratory modeling, and scenario planning, all of which sound to me like "explicit modeling
I think we might just be arguing about different things here? Like to me, these seem more like verbal arguments of questionable veracity than something that has a truth-value like cost-effectiveness analyses or forecasting. (In contrast, Open Phil's reports on AI, or at least the ones I've read, would count as modeling).
because it is generally a terrible idea to just do what a purely quantitative model tells you
What's the actual evidence for this? I feel like this type of reasoning (and other general things like it in the rough cluster of "injunctions against naive consequentialism") are pretty common in our community and tend to be strongly held, but when I ask people to defend it, I just see weird thought experiments and handwaved intuitions (rather than a model or a track record)?
This type of view also maps in my head to being the type of view that's a) high-status and b) diplomatic/"plays nicely" with high-prestige non-EA Western intellectuals, which makes me doubly suspicious that views of this general shape are arrived at through impartial truth-seeking means.
I also think in practice if you have a model telling you to do one thing but your intuitions tell you to do something else, it's often worth making enough updates to form a reflective equilibrium. There are at least two ways to go about this:
1) Use the model to maybe update your intuitions, and go with your intuitions in the final decision, being explicit about how your final decisions may have gone against the naive model.
2) Use your intuitions to probe which pieces your model is making, update your model accordingly, and then go with your (updated) model in the final decision, being explicit about how your final model may have been updated for unprincipled reasons.
I think you (and by revealed preferences, the EA community, including myself) usually goes with 1) as the correct form of intuition vs model reflective equilibrium. But I don't think this is backed by too much evidence, and I think we haven't really given 2) a fair shake.
Now I think in practice 1) and 2) might end up getting the same result much of the time anyway. But a) probably not all the time and b) this is an empirical question.
I like that this post has set out the sketch of a theory of organisation truthfulness. In particular "In worlds where motivated reasoning is commonplace, we’d expect to see:
Red-teaming will discover errors that systematically slant towards an organization’s desired conclusion.
Deeper, more careful reanalysis of cost-effectiveness or impact analyses usually points towards lower rather than higher impact."
Presumably, in worlds where motivated reasoning is rare, red-teaming will discover errors that slant towards and away from an organisation's desired conclusion and deeper, more careful reanalysis of cost-effectiveness points towards lower and higher impact equally often.
I think this is first-order correct (and what my post was trying to get at). Second-order, I think there's at least one important caveat (which I cut from my post) with just tallying total number (or importance-weighted number of) errors towards versus away from the desired conclusion as a proxy for motivated reasoning. Namely, you can't easily differentiate "motivated reasoning" biases from perfectly innocent traditional optimizer's curse.
Suppose an organization is considering 20 possible interventions and do initial cost-effectiveness analyses for each of them. If they have a perfectly healthy and unbiased epistemic process, then the top 2 interventions that they've selected from that list would a) in expectation be better than the other 18 and b) in expectation will have more errors slanted towards higher impact rather than lower impact.
If they then implement the top 2 interventions and do an impact assessment 1 year later, then I think it's likely the original errors (not necessarily biases) from the initial assessment will carry through.
External red-teamers will then discover that these errors are systematically biased upwards, but at least on first blush "naive optimizer's curse issues" looks importantly different in form, mitigation measures, etc, from motivated reasoning concerns.
I think it's likely that either formal Bayesian modeling or more qualitative assessments can allow us to differentiate the two hypotheses.
I haven't reread this post (I find it aversive/painful), but for outside view reasons I think you should heavily discount any conclusion or analysis reached by the author of this post, for reasons outlined here and here.
I would guess that the operational details and factual claims are relatively more trustworthy.
You should assume that the author was pretty junior to this type of analysis and not very good at impact assessments or related points.
I'm not sure the evidence you present is all that strong though, since it too is subject to a lot of selection bias
Oh I absolutely agree. I generally think the more theoretical sections of my post are stronger than the empirical sections. I think the correct update from my post is something like "there is strong evidence of nonzero motivated reasoning in effective altruism, and some probability that motivated reasoning + selection bias-mediated issues are common in our community" but not enough evidence to say more than that.
I think a principled follow-up work (maybe by CEA's new epistemics project manager?) would look like combing through all (or a statistically representative sample of) impact assessments and/or arguments made in EA, and try to catalogue them for motivated reasoning and other biases.
I think you're (unintentionally) running a motte-and-bailey here.
I think this is complicated. It's certainly possible I'm fighting against strawmen!
But I will just say what I think/believe right now, and others are free to correct me. I think among committed longtermists, there is a spectrum of trust in explicit modeling, going from my stereotype of weeatquince(2020)'s views to maybe 50% (30%?) of the converse of what you call the "motte."(Maybe Michael Dickens(2016) is closest?). My guess is that longtermist EAs ( like almost all humans) have never been that close to purely quantitative models guiding decisions, and we've moved closer in the last 5 years to reference classes of fields like the ones that weeatquince's post pulls from.
I also think I agree with MichaelStJules' point about the amount of explicit modeling that actually happens relative to effort given to other considerations. "Real" values are determined not by what you talk about, but by what tradeoffs you actually make.
This was my initial reaction, that suspiciousness of existing forecasts can justify very wide error bars but not certainty in >50 year timelines. But then I realized I didn't understand what probability OP gave to <50 years timelines, which is why I asked a clarifying question first.
Hmm I sort of agree with this. I think when I run back-of-the-envelope calculations on the value of information that you can gain from "gold standard" studies or models on questions that are of potential interest in developed-world contexts (eg high-powered studies on zinc on common cold symptom, modeling how better ventilation can stop airborne disease spread at airports, some stuff on social platforms/infrastructures for testing vaccines, maybe some stuff on chronic fatigue), it naively seems like high-quality but simple research (but not implementation) for developed world health research (including but not limited to the traditional purview of public health) is plausibly competitive with Givewell-style global health charities even after accounting for the 100x-1000x multiplier.
I think the real reason people don't do this more is because we're limited more here on human capital than on $s. In particular, people with a) deep health backgrounds and b) strong EA alignment have pretty strong counterfactuals in working or attempting to work on either existential biorisk reduction or public health research for developing world diseases, both of which are probably more impactful (for different reasons).
I feel like the main reasons you shouldn't trust forecasts from subject matter experts are something like:
external validity: do experts in ML have good forecasts that outperform a reasonable baseline?
AFAIK this is an open question, probably not enough forecasts have resolved yet?
internal validity: do experts in ML have internally consistent predictions? Do they give similar answers at slightly different times when the evidence that has changed is minimal? Do they give similar answers when not subject to framing effects?
AFAIK they've failed miserably
base rates: what's the general reference class we expect to draw from?
I'm not aware of any situation where subject matter experts not incentivized to have good forecasts do noticeably better than trained amateurs with prior forecasting track records.
So like you and steve2152 I'm at least somewhat skeptical of putting too much faith in expert forecasts.
However, in contrast I feel like a lack of theoretical understanding of current ML can't be that strong evidence against trusting experts here, for the very simple reason that conservation of expected evidence means this implies that we ought to trust forecasts from experts with a theoretical understanding of their models more. And this seems wrong because (among others) it would've been wrong 50 years ago to trust experts on GOFAI for their AI timelines!
First of all, I'm really excited for this contest! Using fiction to communicate EA messages has always seemed a priori plausible to me (along the lines of eg 4.2 here), and I'm excited to see various possible different takes here!
Certainly the success of introductions like HPMOR lends additional nontrivial evidence to this theory, so I'm excited to see more experiments like this one and others.
Secondly, really cool that CEA is taking the initiative to encourage these things.
We want lots of people to read and discuss your submissions — we think the Forum will be a really fun place if good stories start showing up. However, we won’t use upvotes or comments as part of our process for choosing a winner.
If you’re wary of sharing your work in public, remember that winning the contest guarantees your work being shared in public (with many, many people). That said, you are welcome to use a pseudonym if you’d prefer!
I think I personally will have a preference for fiction to not show up as top-level posts on the Forum, unless they've been previously vetted as unusually good or they're unusually culturally significant. But obviously a) different people have different tastes, and b) this is your forum!
Thanks for the link to your FAQ, I'm excited to read it further now!
Re: the rest of your comment, I think you're reading more into my comment than I said or meant. I do not think researchers should generally be deferential; I think they should have strong beliefs, that may in fact go against expert consensus. I just don't think this is the right attitude while you are junior
To be clear, I think Geoffrey Hinton's advice was targeted at very junior people. In context, the interview was conducted for Andrew Ng's online deep learning course, which for many people would be their first exposure to deep learning. I also got the impression that he would stand by this advice for early PhDs (though I could definitely have misunderstood him), and by "future Geoffrey Hintons and Eliezer Yudkowskys" I was thinking about pretty junior people rather than established researchers.
I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase).[...](Note also that AI @ Berkeley is a very selective program.)
What % do you think this is true for, quality-weighted?
I remember an interview with Geoffrey Hinton where (paraphrased) Hinton was basically like "just trust your intuitions man. Either your intuitions are good or they're bad. If they are good you should mostly trust your intuitions regardless of what other people say, and if they're bad, well, you aren't going to be a good researcher anyway."
And I remember finding that logic really suspicious and his experiences selection-biased like heck (My understanding is that Hinton "got lucky" by calling neural nets early but his views aren't obviously more principled than his close contemporaries).
But to steelman(steel-alien?) his view a little, I worry that EA is overinvested in outside-view/forecasting types (like myself?), rather than people with strong and true convictions/extremely high-quality initial research taste, which (quality-weighted) may be making up the majority of revolutionary progress.
And if we tell the future Geoffrey Hintons (and Eliezer Yudkowskys) of the world to be more deferential and trust their intuitions less relative to elite consensus or the literature, we're doing the world/our movement a disservice, even if the advice is likely to be individually useful/good for most researchers in terms of expected correctness of beliefs or career advancement.
I disagree that QRI's comparative advantage, such as it is, is figuring out the correctness of moral realism or hedonistic utilitarianism. "Your philosophers were so preoccupied with whether or not they should, they didn't even stop to think if they could."
This is a message I received in private conversation by someone who I trust reasonably highly in terms of general epistemics. I'm reposting it here because it goes against the general "vibe" of the EAF and it's good to get well-informed contrarian opinions.
I used to be very very sceptical of their work (a lot of red flags for 'woo', including lack of positive evidence and being so confusingly/indirectly expressed as to be difficult to even evaluate).Then I read their 2019 neural annealing work (https://opentheory.net/2019/11/neural-annealing-toward-a-neural-theory-of-everything/) and found that it did seem to make some sense and seemed to generate some specific novel predictions. But, as I commented at the time, the things that seemed sensible and useful were almost all related to predictive processing, not their core STV theory and the connection to the major novel parts of their theory seemed unclear.Their responses in the Forum thread were a large negative update for a variety of reasons, but largely because they seemed unable or unwilling to spell out core parts of their theory.Their responses seemed fairly inexplicably bad to me though, because it seemed like there were many cases where (even based on my very slight knowledge of their theory) they could have given much more convincing responses rather than be super evasive or waffly.For example, if they had given the response that they gave in one of the final comments in the discussion, right at the beginning (assuming Abby would have responded similarly) the response to their exchange might have been very different i.e. I think people would have concluded that they gave a sensible response and were talking about things that Abby didn't have expertise to comment on:
_______ Abby Hoskin: If your answer relies on something about how modularism/functionalism is bad: why is source localization critical for your main neuroimaging analysis of interest? If source localization is not necessary: why can't you use EEG to measure synchrony of neural oscillations?
Mike Johnson: The harmonic analysis we’re most interested in depends on accurately modeling the active harmonics (eigenmodes) of the brain. EEG doesn’t directly model eigenmodes; to infer eigenmodes we’d need fairly accurate source localization. It could be there are alternative ways to test STV without modeling brain eigenmodes, and that EEG could give us. I hope that’s the case, and I hope we find it, since EEG is certainly a lot easier to work with than fMRI.
Abby Hoskin: Ok, I appreciate this concrete response. I don't know enough about calculating eigenmodes with EEG data to predict how tractable it is.
While I think celebrating progress is good, and having a clearer "sense" of the data is good, I think the changes in the post are both qualitatively and quantitatively tiny compared to eg, changes my family members in China experienced between 1980 and 2000 or between 2000 and 2020. So I do think having your priors be formed by typical experiences in Western countries would give you a (relative) general sense of global stagnation.
Another potential difference is that you don't get do-overs: the more senior person can't later write a paper that follows exactly the same idea but that's written to a much higher standard, because there's more of a requirement that each paper brings original ideas.
Hmm taking a step back, I wonder if the crux here is that you believe(?) that the natural output for research is paper-shaped^, whereas I would guess that this would be the exception rather than the norm, especially for a field that does not have many very strong non-EA institutions/people (which I naively would guess to be true of EA-style TAI governance).
This might be a naive question, but why is it relevant/important to get papers published if you're trying to do impactful research? From the outside, it seems unlikely that all or most good research is in paper form, especially in a field like (EA) AI governance where (if I understand it correctly) the most important path to impact (other than career/skills development) is likely through improving decision quality for <10(?) actors.
If you are instead trying to play the academia/prestige game, wouldn't it make more sense to optimize for that over direct impact? So instead of focusing on high-quality research on important topics, write the highest-quality (by academic standards) paper you can in a hot/publishable/citable topic and direction.
^ This is a relevant distinction because originality is much more important in journal articles than other publication formats, you absolutely can write a blog post that covers the same general idea as somebody else but better, and AFAIK there's nothing stopping a think tank from "revising" a white paper covering the same general point but with much better arguments.
Hmm I have conflicting feelings about this. I think whenever you add additional roadblocks or other limitations on criticism, or suggestions that criticisms can be improved, you
a) see the apparent result that criticisms that survive the process will on average be better.
b) fail to see the (possibly larger) effect that there's an invisible graveyard of criticisms that people choose not to voice because it's not worth the hassle.
At the same time, being told that your life work is approximately useless is never a pleasant feeling, and it's not always reasonable to expect people to handle it with perfect composure (Thankfully nothing of this magnitude has ever happened to me, but I was pretty upset when an EA Forum draft I wrote in only a few days had to be scrapped or at least rewritten because it assumed a mathematical falsehood). So while I think Mike's responses to Abby are below a reasonable bar of good forum commenting norms, I think I have more sympathy for his feelings and actions here than Greg seems to.
So I'm pretty conflicted. My own current view is that I endorse Abby's comments and tone as striking the right balance for the forum, and I endorse Greg's content but not the tone.
But I think reasonable people can disagree here, and we should also be mindful that when we ask people to rephrase substantive criticisms to meet a certain stylistic bar (see also comments here), we are implicitly making criticisms more onerous, which arguably has pretty undesirable outcomes.
Ideally, someone senior tells you what to work on. But this is time-expensive for them, and they don’t want to give away their best ideas to somebody who might execute them badly. So more realistically…
This seems very surprising to me. Unless by "best ideas" you mean "literally somebody's top idea" or by "someone senior" you mean Nick Bostrom?
My impression from talking to friends working in ML is that usually faculty have ideas that they'd be excited to see their senior grad students to work on, senior grad students have research ideas that they'd love for junior grad students to implement, and so forth.
Math and theoretical CS likewise have lists of open problems.
Similarly, in (non-academic EA) research I have way too many ideas that I can't work on myself, and I've frequently seen buffets of potential research topics/ideas that more senior researchers propose.
My general impression is that this is the norm in EA research? When people choose not to work on other people's ideas, it's usually due to a combination of personal fit and arrogance in believing your own ideas are more important (or depending on the relevant incentives, other desiderata like "publishable", "appealing to funders", or "tractable"), not because of a lack of ideas!
I think of the metrics I mentioned above as proxies rather than as the underlying targets, which is some combination of:
a) Is STV true? b) Conditional upon STV being true, is it useful?
What my forecasting questions aimed to do is shedding light on a). I agree that academia and citations isn't the best proxy. They may in some cases have conservatism bias (I think trusting the apparent academic consensus on AI risk in 2014 would've been a mistake for early EAs), but are also not immune to falseties/crankery (cf replication crisis). In addition, standards for truth and usefulness are different within EA circles than academia, partially because we are trying to answer different questions.
This is especially an issue as the areas that QRI is likely to interact with (consciousness, psychedelics) seem from the outside to be more prone than average to falseness and motivated cognition, including within academia.
This is what I was trying to get at with "will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?" because Luke is a non-QRI affiliated person within EA who's a) respected and b) have thought about concepts adjacent to QRI's work. Bearing in mind that Luke is very far from a perfect oracle, I would still trust Luke's judgement on this more than an arbitrarily selected academic in an adjacent field.
I think the actual question I'm interested in is something like "In X year, will a panel of well-respected EAs a) not affiliated with QRI and b) have very different thoughts from each other and c)who have thought about things adjacent to QRI's work have updated to believing STV to be substantively true" but I was unable to come up with a clean question operationalization in the relatively brief amount of time I gave myself to come up with this.
People are free to counterpropose and make their own questions.
Note that the 2nd question is about total citations rather than of one paper, and 3k citations doesn't seem that high if you're introducing an entirely new subfield (which is roughly what I'd expect if STV is true). The core paper of Friston's free energy principle has almost 5,000 citations for example, and it seems from the outside that STV (if true) ought to be roughly as big a deal as free energy.
For a sense of my prior beliefs about EA-encouraged academic subfields, I think 3k citations in 10 years is an unlikely but not insanely high target for wild animal welfare (maybe 20-30%?), and AI risk is likely already well beyond that (eg >1k citations for Concrete Problems alone).
Hi, all. Talk is cheap, and EA Forum karma may be insufficiently nuanced to convey substantive disagreements.
I've taken the liberty to sketch out several forecasting questions that might reflect underlying differences in opinion. Interested parties may wish to forecast on them (which the EA Forum should allow you to do directly, at least on desktop) and then make bets accordingly.
Feel free to also counterpropose (and make!) other questions if you think the existing question operationalizations are not sufficient (I'm far from knowledgeable in this field!).
For what it's worth, I read this comment as constructive rather than non-constructive.
If I write a long report and an expert in the field think that the entire premise is flawed for specific technical reasons, I'd much rather them point this out rather than for them to worry about niceness and then never getting around to mentioning it, thus causing my report to languish in obscurity without me knowing why (or worse, for my false research to actually be used!)
Geometric mean is just a really useful tool for estimations in general. It also makes a lot of sense for aggregating results other than probabilities, eg for different Fermi estimates of real quantities.
That said, I think it's worth pointing out the case where arithmetic mean of probabilities is exactly right to use: if you think that exactly one of the estimates is correct but you don't know which (rather than the usual situation of thinking they all provide evidence about what the correct answer is).
To extend this and steelman the case for arithmetic mean of probabilities (or something in that general direction) a little, in some cases this seems a more intuitive formulation of risk (which is usually how these things are talked about in EA contexts), especially if we propagate further to expected values or value of information concerns.
Eg, suppose that we ask 3 sources we trust equally about risk from X vector of an EA org shutting down in 10 years. One person says 10%, 1 person says 0.1%, 1 person says 0.001%.
Arithmetic mean of probabilities gets you ~ 3.4%, geometric mean of odds gets you ~0.1%. 0.1% seems comfortably below the background rate of organizations dying, that in many cases it's not worth the value of information to investigate further. Yet naively this seems to be too cavalier if one out of three sources thinks there's a 10% chance of failure from X vector alone!
Also as a mild terminological note, I'm not sure I know what you mean by "correct answer" when we're referring to probabilities in the real world. Outside of formal mathematical examples and maybe some quantum physics stuff, probabilities are usually statements about our own confusions in our maps of the world, not physically instantiated in the underlying reality.
Hmm one recent example is that somebody casually floated to me an idea that can potentially entirely solve an existential risk (though the solution might have downside risks of its own) and I realized then that I had no idea how much to price the solution in terms of EA $s, like whether it should be closer to 100M, 1B or $100B.
My first gut instinct was to examine the solution and also to probe the downside risks, but then I realized this is thinking about it entirely backwards. The downside risks and operational details don't matter if even the most optimistic cost-effectiveness analyses isn't enough to warrant this being worth funding!
General suspicion of the move away from expected-value calculations and cost-effectiveness analyses.
This is a portion taken from a (forthcoming) post about some potential biases and mistakes in effective altruism that I've analyzed via looking at cost-effectiveness analysis. Here, I argue that the general move (at least outside of human and animal neartermism) away from Fermi estimates, expected values, and other calculations just makes those biases harder to see, rather than fix the original biases.
I may delete this section from the actual post as this point might be a distraction from the overall point.
I’m sure there are very good reasons (some stated, some unstated) for moving away from cost-effectiveness analysis. But I’m overall pretty suspicious of the general move, for a similar reason that I’d be suspicious of non-EAs telling me that we shouldn’t use cost-effectiveness analyses to judge their work, in favor of say systematic approaches, good intuitions, and specific contexts like lived experiences (cf. Beware Isolated Demands for Rigor):
I’m sure you have specific arguments for why in your case quantitative approaches aren’t very necessary and useful, because your uncertainties span multiple orders of magnitude, because all the calculations are so sensitive to initial assumptions, and so forth. But none of these arguments really point to verbal heuristics suddenly (despite approximately all evidence and track records to the contrary) performing better than quantitative approaches.
In addition to the individual epistemic issues with verbal assessments unmoored by numbers, we also need to consider the large communicative sacrifices made by not having a shared language (mathematics) to communicate things like uncertainty and effect sizes. Indeed, we have ample evidence that switching away from numerical reasoning when communicating uncertainty is a large source of confusion.
To argue that in your specific situation, verbal judgment is superior without numbers than with numbers, never mind that your proposed verbal solutions obviates the biases associated with trying to do numerical cost-effectiveness modeling of the same, the strength of your evidence and arguments needs to be overwhelming. Instead, I get some simple verbal heuristic-y arguments, and all of this is quite suspicious.
Or more succinctly:
It’s easy to lie with numbers, but it’s even easier to lie without them
So overall I don’t think moving away from explicit expected value calculations and cost-effectiveness analyses is much of a solution, if at all, for the high-level reasoning mistakes and biases that are more clearly seen in cost-effectiveness analyses. Most of what the shift away from EV does is makes things less grounded in reality, less transparent and harder to critique (cf. “Not Even Wrong”).
I've read his post on this before. I think this question is substantively easier for heavy SFE-biased views, especially if you pair it with some other beliefs Brian has.
that's pretty damning for the WAW movement, unless climate change prevention is not the highest-leverage influence against WAS (a priori, it seems unlikely that climate change prevention would have the highest positive influence).
I read it as more damming for climate change people for what it's worth, or at least the ones who claim to be doing it for the animals.