The motivated reasoning critique of effective altruism 2021-09-14T20:43:14.571Z
How valuable is ladder-climbing outside of EA for people who aren't unusually good at ladder-climbing or unusually entrepreneurial? 2021-09-01T00:47:31.983Z
What are examples of technologies which would be a big deal if they scaled but never ended up scaling? 2021-08-27T08:47:16.911Z
What are some key numbers that (almost) every EA should know? 2021-06-18T00:37:17.794Z
Epistemic Trade: A quick proof sketch with one example 2021-05-11T09:05:25.181Z
[Linkpost] New Oxford Malaria Vaccine Shows ~75% Efficacy in Initial Trial with Infants 2021-04-23T23:50:20.545Z
Some EA Forum Posts I'd like to write 2021-02-23T05:27:26.992Z
RP Work Trial Output: How to Prioritize Anti-Aging Prioritization - A Light Investigation 2021-01-12T22:51:31.802Z
Some learnings I had from forecasting in 2020 2020-10-03T19:21:40.176Z
How can good generalist judgment be differentiated from skill at forecasting? 2020-08-21T23:13:12.132Z
What are some low-information priors that you find practically useful for thinking about the world? 2020-08-07T04:38:07.384Z
David Manheim: A Personal (Interim) COVID-19 Postmortem 2020-07-01T06:05:59.945Z
I'm Linch Zhang, an amateur COVID-19 forecaster and generalist EA. AMA 2020-06-30T19:35:13.376Z
Are there historical examples of excess panic during pandemics killing a lot of people? 2020-05-27T17:00:29.943Z
[Open Thread] What virtual events are you hosting that you'd like to open to the EA Forum-reading public? 2020-04-07T01:49:05.770Z
Should recent events make us more or less concerned about biorisk? 2020-03-19T00:00:57.476Z
Are there any public health funding opportunities with COVID-19 that are plausibly competitive with Givewell top charities per dollar? 2020-03-12T21:19:19.565Z
All Bay Area EA events will be postponed until further notice 2020-03-06T03:19:24.587Z
Are there good EA projects for helping with COVID-19? 2020-03-03T23:55:59.259Z
How can EA local groups reduce likelihood of our members getting COVID-19 or other infectious diseases? 2020-02-26T16:16:49.234Z
What types of content creation would be useful for local/university groups, if anything? 2020-02-15T21:52:00.803Z
How much will local/university groups benefit from targeted EA content creation? 2020-02-15T21:46:49.090Z
Should EAs be more welcoming to thoughtful and aligned Republicans? 2020-01-20T02:28:12.943Z
Is learning about EA concepts in detail useful to the typical EA? 2020-01-16T07:37:30.348Z
8 things I believe about climate change 2019-12-28T03:02:33.035Z
Is there a clear writeup summarizing the arguments for why deep ecology is wrong? 2019-10-25T07:53:27.802Z
Linch's Shortform 2019-09-19T00:28:40.280Z
The Possibility of an Ongoing Moral Catastrophe (Summary) 2019-08-02T21:55:57.827Z
Outcome of GWWC Outreach Experiment 2017-02-09T02:44:42.224Z
Proposal for an Pre-registered Experiment in EA Outreach 2017-01-08T10:19:09.644Z
Tentative Summary of the Giving What We Can Pledge Event 2015/2016 2016-01-19T00:50:58.305Z
The Bystander 2016-01-10T20:16:47.673Z


Comment by Linch on EA needs consultancies · 2021-09-18T21:48:09.624Z · EA · GW

"Y" is a strictly stronger claim than "If X, then Y", but many people get more emotional with "If X, then Y."

Consider "Most people around 2000 years ago had a lot of superstitions and usually believed wrong things" vs "Before Jesus Christ, people had a lot of superstitions and usually believed wrong things."

In hindsight I wish I'd given your wording, not mine, but oh well

Oh what an interesting coincidence.

Comment by Linch on EA needs consultancies · 2021-09-17T19:50:57.809Z · EA · GW

I tried answering your question on the object level a few times but I notice myself either trying to be reconciliatory or defensive, and I don't think I will endorse either response upon reflection. 

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-17T08:26:30.063Z · EA · GW

Hi. I'm glad you appear to have gained a lot from my quick reply, but for what it's worth I did not intend my reply as an admonishment.

I think the core of what I read as your comment is probably still valid. Namely, that if I misidentified problems as biases when almost all of the failures are due to either a) noise/error or b) incompetence unrelated to decision quality (eg mental health, insufficient technical skills, we aren't hardworking enough), then the bias identification isn't true or useful. Likewise, debiasing is somewhere between neutral to worse than useless if the problem was never bias to begin with.

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-17T05:57:24.982Z · EA · GW

I'm suspicious of 1), especially if taken too far, because I think if taken too far it would justify way too much complacency in worlds where foreseeable moral catastrophes are not only possible but probable

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-17T05:49:47.389Z · EA · GW

Some quick thoughts: I would guess that Open Phil is better at this than other EA orgs, both because of individually more competent people and much better institutional incentives (ego not wedded to specific projects working). For your specific example, I'm (as you know) new to AI governance, but I would naively guess that most (including competence-weighted) people in AI governance are more positive about AI interventions than you are. 

Happy to be corrected empirically. 

(I also agree with Larks that publishing a subset of these may be good for improving the public conversation/training in EA, but I understand if this is too costly and/or if the internal analyses embed too much sensitive information or models)

Comment by Linch on The Importance-Avoidance Effect · 2021-09-17T01:04:57.388Z · EA · GW

You might also like Aaron Schwartz's notes on productivity:

Assigned problems

Assigned problems are problems you’re told to work on. Numerous psychology experiments have found that when you try to “incentivize” people to do something, they’re less likely to do it and do a worse job. External incentives, like rewards and punishments, kills what psychologists call your “intrinsic motivation” — your natural interest in the problem. (This is one of the most thoroughly replicated findings of social psychology — over 70 studies have found that rewards undermine interest in the task.)5 People’s heads seem to have a deep avoidance of being told what to do.6

[LZ Sidenote: I think I'd want to actually read the studies or at least a meta-analysis of recent replications first before being sure of this]

The weird thing is that this phenomenon isn’t just limited to other people — it even happens when you try to tell yourself what to do! If you say to yourself, “I should really work on X, that’s the most important thing to do right now” then all of the sudden X becomes the toughest thing in the world to make yourself work on. But as soon as Y becomes the most important thing, the exact same X becomes much easier.

Create a false assignment

This presents a rather obvious solution: if you want to work on X, tell yourself to do Y. Unfortunately, it’s sort of difficult to trick yourself intentionally, because you know you’re doing it.7 So you’ve got to be sneaky about it.

One way is to get someone else to assign something to you. The most famous instance of this is grad students who are required to write a dissertation, a monumentally difficult task that they need to do to graduate. And so, to avoid doing this, grad students end up doing all sorts of other hard stuff.

The task has to both seem important (you have to do this to graduate!) and big (hundreds of pages of your best work!) but not actually be so important that putting it off is going to be a disaster.


Comment by Linch on EA needs consultancies · 2021-09-16T09:00:16.865Z · EA · GW

Hmm, did you read the asterisk in the quoted comment?

*The natural Gricean implicature of that claim is that I'm saying that EA orgs are an exception. I want to disavow that implication. For context, I think this is plausibly the second or third biggest limitation for my own work.

(No worries if you haven't, I'm maybe too longwinded and it's probably unreasonable to expect people to carefully read everything on a forum post with 76 comments!)

If you've read it and still believe that I "sound breathtakingly arrogant ", I'd be interested in whether you can clarify whether "breathtakingly arrogant" means either a) what I say is untrue or b) what I say is true but insufficiently diplomatic. 

More broadly, I mostly endorse the current level of care and effort and caveats I put on the forum. (though I want to be more concise, working on it!) 

I can certainly make my writing more anodyne and less likely to provoke offense, e.g. by defensive writing and pre-empting all objections I can think of,  by sprinkling the article heavily with caveats throughout, by spending 3x as much time on each sentence, or just by having much less public output (the last of which is empirically what most EAs tend to do).

I suspect this will make my public writing worse however. 

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-16T07:54:18.062Z · EA · GW

Thanks a lot! Is there a writeup of this somewhere? I tend to be a pretty large fan of explicit rationality (at least compared to EAs or rationalists I know), so evidence that reasoning in this general direction is empirically kind of useless would be really useful to me!

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T22:54:17.634Z · EA · GW

Yeah I'm surprised by this as well. Both classical utilitarianism (in the extreme version, "everything that is not morally obligatory is forbidden") and longtermism just seem to have many lower degrees of freedom than other commonly espoused ethical systems, so it would naively be surprising if these worldviews can justify a broader range of actions than close alternatives. 

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T20:58:31.479Z · EA · GW

I think what you said is insightful and worth considering further. Nonetheless, I will only address a specific subpoint for now, and revisit this later.

Basically, have you considered the perspective that “some EA orgs aren’t very good” to be a better explanation for the problems?

Hmm I'm not sure what you mean, and I think it's very likely we're talking about different problems. But assuming we're talking about the same problems, at a high-level any prediction problem can be decomposed to bias vs error (aka noise, aka variance).

I perceive that many of the issues I've mentioned to be better explained by bias than error. In particular I just don't think we'll see equivalently many errors in the opposite direction. This is an empirical question however, and I'd be excited to see more careful followups to test this hypothesis.

(as a separate point, I do think some EA orgs aren't very good, with "very good" defined as I'd rather the $s be spent on their work rather than in Open Phil coffers, or my own bank account. I imagine many other EAs would feel similarly about my own work). 

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T20:47:53.112Z · EA · GW

Speaking for myself, I think I have strong ideological reasons to think that predictably doing (lots of) good is possible. 

I also have a bias towards believing that things that are good for X reason are also good for Y reason, and this problem rears itself up even when I try to correct for it. E.g. I think Linch(2012-2017) too easily bought in to the "additional consumption $s don't make you happier " narratives, and I'm currently lactovegetarian even though I started being vegetarian for very different reasons than what I currently believe to be the most important. I perceive other EAs as on average worse at this (not sure the right term. Decoupling?) than me, which is not necessarily true of the other biases on this list. 

A specific instantiation of this is that it's easier for me to generate solutions to problems that are morally unambiguous by the standards of non-EA Western morality, even though we'd expect the tails to come apart fairly often. 

To a lesser extent, I have biases towards thinking that doing (lots of) good comes from things that I and my friends are predisposed to be good at (eg cleverness, making money).

Another piece of evidence is that EAs seem far from immune from ideological capture for non-EA stuff. My go-to example is the SSC/NYT thing

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T19:22:52.294Z · EA · GW

I didn't mean anything special, just that IDInsight developmental economists started Giving Green and was overly credulous in a bunch of ways, as mentioned in the New causes section.

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T11:31:17.075Z · EA · GW

Thanks, done.

Comment by Linch on EA Forum Creative Writing Contest: Submission thread for work first published elsewhere · 2021-09-15T11:10:36.971Z · EA · GW

One of my favorite pieces of EA fiction is a Tweet I can no longer find, most likely from the Twitter account ASmallFiction:

"It was a difficult job," he thought to himself, "but someone had to do it."

As he walked away, he wondered who that someone will be. 

I actually quote that a lot, including as admonishments to myself and in mentoring discussions about agency and responsibility to more junior EAs.

I also like this one:

"What's this?" 

"That's an ancient map." 

"Then what are these lines?" 

"'Borders.' The ancient ones used them to decide who to care about."

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T10:51:37.561Z · EA · GW

Thanks so much for the response! Upvoted.

(I'm exaggerating my views here to highlight the differences, I think my all-things-considered opinion on these positions are much closer to yours than the rest of the comment will make it sound)

I think my strongest disagreement with your comment is the framing here:

I'm not sure how you're distinguishing between the two hypotheses:

  1. Longtermists don't like quantitative modeling in general.
  2. Longtermist questions are not amenable to quantitative modeling, and so longtermists don't do much quantitative modeling, but they would if they tackled questions that were amenable to quantitative modeling.

(Unless you want to defend the position that longtermist questions are just as easy to model as, say, those in global poverty? That would be... an interesting position.)

If we peel away the sarcasm, I think the implicit framing is that

  1. If X is less amenable than Y to method A of obtaining truth, and X is equally or more amenable to methods B, C, and D relative to Y, we should do less method A to obtain truth in X (relative to Y), and more methods B, C, and D.
  2. X is less amenable than Y to method A of obtaining truth.
  3. Thus, we should use method A less in X than in Y.

Unless I'm missing something, I think this is logically invalid. The obvious response here is that I don't think longtermist questions are more amenable to explicit quantitative modeling than global poverty, but I'm even more suspicious of other methodologies here.

Medicine is less amenable to empirical testing than physics, but that doesn't mean that clinical intuition is a better source of truth for the outcomes of drugs than RCTs.  (But medicine is  relatively much less amenable to theorems than physics, so it's correct to use less proofs in medicine than physics.)

More minor gripes:

 (and I'll note that even GiveWell isn't doing this).

I think I'm willing to bite the bullet and say that GiveWell (or at least my impression of them from a few years back) should be more rigorous in their modeling.  Eg, weird to use median staff member's views as a proxy for truth, weird to have so few well-specified forecasts, and so forth. 

The first piece of advice in that post is to use techniques like assumption based planning, exploratory modeling, and scenario planning, all of which sound to me like "explicit modeling

I think we might just be arguing about different things here? Like to me, these seem more like verbal arguments of questionable veracity than something that has a truth-value like cost-effectiveness analyses or forecasting.  (In contrast, Open Phil's reports on AI, or at least the ones I've read, would count as modeling). 

because it is generally a terrible idea to just do what a purely quantitative model tells you

What's the actual evidence for this? I feel like this type of reasoning (and other general things like it in the rough cluster of "injunctions against naive consequentialism") are pretty common in our community and tend to be strongly held, but when I ask people to defend it, I just see weird thought experiments and handwaved intuitions (rather than a model or a track record)?

This type of view also maps in my head to being the type of view that's a) high-status and b) diplomatic/"plays nicely" with high-prestige non-EA Western intellectuals, which makes me doubly suspicious that views of this general shape are arrived at through impartial truth-seeking means. 

I also think in practice if you have a model telling you to do one thing but your intuitions tell you to do something else, it's often worth making enough updates to form a reflective equilibrium. There are at least two ways to go about this:

1) Use the model to maybe update your intuitions, and go with your intuitions in the final decision, being explicit about how your final decisions may have gone against the naive model.

2) Use your intuitions to probe which pieces your model is making, update your model accordingly, and then go with your (updated) model in the final decision, being explicit about how your final model may have been updated for unprincipled reasons.

I think you (and by revealed preferences, the EA community, including myself) usually goes with 1) as the correct form of intuition vs model reflective equilibrium. But I don't think this is backed by too much evidence, and I think we haven't really given 2) a fair shake.

Now I think in practice 1) and 2) might end up getting the same result much of the time anyway. But a) probably not all the time and b) this is an empirical question. 

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T10:27:15.109Z · EA · GW

Also, ALLFED gets hit particularly hard because of their quantitative estimate, but I don't think they're uniquely terrible, but rather uniquely transparent.

Oh I agree. Do you think it's worth editing my post to make that clearer?

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-15T00:44:19.202Z · EA · GW

Thanks for your extensions! Worth pondering more.

I like that this post has set out the sketch of a theory of organisation truthfulness. In particular 
"In worlds where motivated reasoning is commonplace, we’d expect to see:

  1. Red-teaming will discover errors that systematically slant towards an organization’s desired conclusion.
  2. Deeper, more careful reanalysis of cost-effectiveness or impact analyses usually points towards lower rather than higher impact."

Presumably, in worlds where motivated reasoning is rare, red-teaming will discover errors that slant towards and away from an organisation's desired conclusion and deeper, more careful reanalysis of cost-effectiveness points towards lower and higher impact equally often.

I think this is first-order correct (and what my post was trying to get at). Second-order, I think there's at least one important caveat (which I cut from my post) with just tallying total number (or importance-weighted number of) errors towards versus away from the desired conclusion as a proxy for motivated reasoning. Namely, you can't easily differentiate "motivated reasoning" biases from perfectly innocent traditional optimizer's curse

Suppose an organization is considering 20 possible interventions and do initial cost-effectiveness analyses for each of them. If they have a perfectly healthy and unbiased epistemic process, then the top 2 interventions that they've selected from that list would  a) in expectation be better than the other 18 and b) in expectation will have more errors slanted towards higher impact rather than lower impact. 

If they then implement the top 2 interventions and do an impact assessment 1 year later, then I think it's likely the original errors (not necessarily biases) from the initial assessment will carry through. 

External red-teamers will then discover that these errors are systematically biased upwards, but at least on first blush "naive optimizer's curse issues" looks importantly different in form, mitigation measures, etc, from motivated reasoning concerns. 

I think it's likely that either formal Bayesian modeling or more qualitative assessments can allow us to differentiate the two hypotheses.

Comment by Linch on Tentative Summary of the Giving What We Can Pledge Event 2015/2016 · 2021-09-15T00:05:03.565Z · EA · GW

I haven't reread this post (I find it aversive/painful), but for outside view reasons I think you should heavily discount any conclusion or analysis reached by the author of this post, for reasons outlined here and here

I would guess that the operational details and factual claims are relatively more trustworthy.

You should assume that the author was pretty junior to this type of analysis and not very good at impact assessments or related points.

Comment by Linch on The motivated reasoning critique of effective altruism · 2021-09-14T23:39:21.159Z · EA · GW

I'm not sure the evidence you present is all that strong though, since it too is subject to a lot of selection bias

Oh I absolutely agree. I generally think the more theoretical sections of my post are stronger than the empirical sections. I think the correct update from my post is something like "there is strong evidence of nonzero motivated reasoning in effective altruism, and some probability that motivated reasoning + selection bias-mediated issues are common in our community" but not enough evidence to say more than that.

I think a principled follow-up work (maybe by CEA's new epistemics project manager?) would look like combing through all (or a statistically representative sample of) impact assessments and/or arguments made in EA, and try to catalogue them for motivated reasoning and other biases. 

I think you're (unintentionally) running a motte-and-bailey here.

I think this is complicated. It's certainly possible I'm fighting against strawmen!

But I will just say what I think/believe right now, and others are free to correct me. I think among committed longtermists, there is a spectrum of trust in explicit modeling, going from my stereotype of weeatquince(2020)'s views to maybe 50% (30%?) of the converse of what you call the "motte."(Maybe Michael Dickens(2016) is closest?). My guess is that longtermist EAs ( like almost all humans) have never been that close to purely quantitative models guiding decisions, and we've moved closer in the last 5 years to reference classes of fields like the ones that weeatquince's post pulls from. 

I also think I agree with MichaelStJules' point about the amount of explicit modeling that actually happens relative to effort given to other considerations. "Real" values are determined not by what you talk about, but  by what tradeoffs you actually make.

Comment by Linch on EA Forum Creative Writing Contest: $10,000 in prizes for good stories · 2021-09-14T23:30:23.714Z · EA · GW

(For disclosure: Lizka's interning under me)

Comment by Linch on EA Forum Creative Writing Contest: $10,000 in prizes for good stories · 2021-09-14T10:38:59.210Z · EA · GW

So personally, I will prefer for entries to be replies to a top-level post, and maybe for winners to be reposted to top-level posts. 

But I will hide it for myself for now.

Comment by Linch on AI timelines and theoretical understanding of deep learning · 2021-09-14T03:39:13.718Z · EA · GW

This was my initial reaction, that suspiciousness of existing forecasts can justify very wide error bars but not certainty in >50 year timelines. But then I realized I didn't understand what probability OP gave to <50 years timelines, which is why I asked a clarifying question first. 

Comment by Linch on EA Forum Creative Writing Contest: $10,000 in prizes for good stories · 2021-09-14T03:35:14.846Z · EA · GW

All writing is covered by copyright!

Not writing published 70 years after the author's death, if I understand correctly.

(Which is not a hypothetical example if people are planning to excerpt Kipling). 

Comment by Linch on Public Health Research · 2021-09-13T01:16:31.057Z · EA · GW

Hmm I sort of agree with this. I think when I run back-of-the-envelope calculations on the value of information that you can gain from "gold standard" studies or models on questions that are of potential interest in developed-world contexts (eg high-powered studies on zinc on common cold symptom, modeling how better ventilation can stop airborne disease spread at airports, some stuff on social platforms/infrastructures for testing vaccines, maybe some stuff on chronic fatigue), it naively seems like high-quality but simple research (but not implementation) for developed world health research  (including but not limited to the traditional purview of public health) is plausibly competitive with Givewell-style global health charities even after accounting  for the 100x-1000x multiplier.

I think the real reason people don't do this more is because we're limited more here on human capital than on $s. In particular, people with a) deep health backgrounds and b) strong EA alignment have pretty strong counterfactuals in working or attempting to work on either existential biorisk reduction or public health research for developing world diseases, both of which are probably more impactful (for different reasons).

Comment by Linch on AI timelines and theoretical understanding of deep learning · 2021-09-12T22:30:16.322Z · EA · GW

I have generally been quite skeptical about the view that we are on the cusp of a revolution that will lead us to artificial general intelligence in the next 50 years so.

Can you clarify what you mean by this? Does "quite skeptical" mean

I think there is <20% probability that we'll have AGI in <50 years


I think there is <1% probability that we'll have AGI in <50 years


I think there is <0.01% probability that we'll have AGI in <50 years

or something else? 

Language is quite imprecise, numbers can't resolve uncertainty in the underlying phenomenon, but they help a lot in clarifying and making the strength of your uncertainty more precise. 

Comment by Linch on AI timelines and theoretical understanding of deep learning · 2021-09-12T22:23:53.890Z · EA · GW

I feel like the main reasons you shouldn't trust forecasts from subject matter experts are something like:

  • external validity: do experts in ML have good forecasts that outperform a reasonable baseline?
    • AFAIK this is an open question, probably not enough forecasts have resolved yet?
  • internal validity: do experts in ML have internally consistent predictions? Do they give similar answers at slightly different times when the evidence that has changed is minimal? Do they give similar answers when not subject to framing effects?
    • AFAIK they've failed miserably
  • base rates: what's the general reference class we expect to draw from?
    • I'm not aware of any situation where subject matter experts not incentivized to have good forecasts do noticeably better than trained amateurs with prior forecasting track records.

So like you and steve2152 I'm at least somewhat skeptical of putting too much faith in expert forecasts. 

However, in contrast I feel like a lack of theoretical understanding of current ML can't be that strong evidence against trusting experts here, for the very simple reason that conservation of expected evidence means this implies that we ought to trust forecasts from experts with a theoretical understanding of their models more. And this seems wrong because (among others) it would've been wrong 50 years ago to trust experts on GOFAI for their AI timelines!

Comment by Linch on EA Forum Creative Writing Contest: $10,000 in prizes for good stories · 2021-09-12T21:55:39.261Z · EA · GW

First of all, I'm really excited for this contest! Using fiction to communicate EA messages has always seemed a priori plausible to me (along the lines of eg 4.2 here), and I'm excited to see various possible different takes here!

Certainly the success of introductions like HPMOR lends additional nontrivial evidence to this theory, so I'm excited to see more experiments like this one and others.

Secondly, really cool that CEA is taking the initiative to encourage these things.

How do I submit content?

All stories must be published on the EA Forum and tagged with Creative Writing Contest.

We want lots of people to read and discuss your submissions — we think the Forum will be a really fun place if good stories start showing up. However, we won’t use upvotes or comments as part of our process for choosing a winner.

If you’re wary of sharing your work in public, remember that winning the contest guarantees your work being shared in public (with many, many people). That said, you are welcome to use a pseudonym if you’d prefer!

I think I personally will have a preference for fiction to not show up as top-level posts on the Forum, unless they've been previously vetted as unusually good or they're unusually culturally significant. But obviously a) different people have different tastes, and b)  this is your forum! 

Comment by Linch on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-12T00:09:10.160Z · EA · GW

(As an aside, I read your FAQ and enjoyed it, so thanks for the share!)

Comment by Linch on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-09T21:01:33.486Z · EA · GW

Thanks for the link to your FAQ, I'm excited to read it further now!

Re: the rest of your comment, I think you're reading more into my comment than I said or meant. I do not think researchers should generally be deferential; I think they should have strong beliefs, that may in fact go against expert consensus. I just don't think this is the right attitude while you are junior

To be clear, I think Geoffrey Hinton's advice was targeted at very junior people. In context, the interview was conducted for Andrew Ng's online deep learning course, which for many people would be their first exposure to deep learning. I also got the impression that he would stand by this advice for early PhDs (though I could definitely have misunderstood him), and by "future Geoffrey Hintons and Eliezer Yudkowskys" I was thinking about pretty junior people rather than established researchers.

Comment by Linch on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-09T16:58:55.872Z · EA · GW

I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase).[...](Note also that AI @ Berkeley is a very selective program.)

What % do you think this is true for, quality-weighted? 

I remember an interview with Geoffrey Hinton where (paraphrased) Hinton was basically like "just trust your intuitions man. Either your intuitions are good or they're bad. If they are good you should mostly trust your intuitions regardless of what other people say, and if they're bad, well, you aren't going to be a good researcher anyway."

And I remember finding that logic really suspicious and his experiences selection-biased like heck (My understanding is that Hinton "got lucky" by calling neural nets early but his views aren't obviously more principled than his close contemporaries). 

But to steelman(steel-alien?) his view a little, I worry that EA is overinvested in outside-view/forecasting types (like myself?), rather than people with strong and true convictions/extremely high-quality initial research taste, which (quality-weighted) may be making up  the majority of revolutionary progress. 

And if we tell the future Geoffrey Hintons (and Eliezer Yudkowskys) of the world to be more deferential and trust their intuitions less relative to elite consensus or the literature, we're doing the world/our movement a disservice, even if the advice is likely to be individually useful/good for most researchers in terms of expected correctness of beliefs or career advancement. 

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-09T16:37:04.952Z · EA · GW

I disagree that QRI's comparative advantage, such as it is, is figuring out the correctness of moral realism or hedonistic utilitarianism. "Your philosophers were so preoccupied with whether or not they should, they didn't even stop to think if they could."

Comment by Linch on APPLY NOW | EA Global: London (29-31 Oct) | EAGxPrague (3-5 Dec) · 2021-09-09T16:29:28.699Z · EA · GW

back of the envelope calculations

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-09T16:26:53.033Z · EA · GW

This is a message I received in private conversation by someone who I trust reasonably highly in terms of general epistemics. I'm reposting it here because it goes against the general "vibe" of the EAF and it's good to get well-informed contrarian opinions.

I used to be very very sceptical of their work (a lot of red flags for 'woo', including lack of positive evidence and being so confusingly/indirectly expressed as to be difficult to even evaluate).Then I read their 2019 neural annealing work ( and found that it did seem to make some sense and seemed to generate some specific novel predictions. But, as I commented at the time,  the things that seemed sensible and useful were almost all related to predictive processing, not their core STV theory and the connection to the major novel parts of their theory seemed unclear.Their responses in the Forum thread were a large negative update for a variety of reasons, but largely because they seemed unable or unwilling to spell out core parts of their theory.Their responses seemed fairly inexplicably bad to me though, because it seemed like there were many cases where (even based on my very slight knowledge of their theory) they could have given much more convincing responses rather than be super evasive or waffly.For example, if they had given the response that they gave in one of the final comments in the discussion, right at the beginning (assuming Abby would have responded similarly) the response to their exchange might have been very different i.e. I think people would have concluded that they gave a sensible response and were talking about things that Abby didn't have expertise to comment on:

Abby Hoskin: If your answer relies on something about how modularism/functionalism is bad: why is source localization critical for your main neuroimaging analysis of interest? If source localization is not necessary: why can't you use EEG to measure synchrony of neural oscillations?

Mike Johnson: The harmonic analysis we’re most interested in depends on accurately modeling the active harmonics (eigenmodes) of the brain. EEG doesn’t directly model eigenmodes; to infer eigenmodes we’d need fairly accurate source localization. It could be there are alternative ways to test STV without modeling brain eigenmodes, and that EEG could give us. I hope that’s the case, and I hope we find it, since EEG is certainly a lot easier to work with than fMRI.

Abby Hoskin: Ok, I appreciate this concrete response. I don't know enough about calculating eigenmodes with EEG data to predict how tractable it is.

Comment by Linch on APPLY NOW | EA Global: London (29-31 Oct) | EAGxPrague (3-5 Dec) · 2021-09-09T16:17:41.600Z · EA · GW

I'd be interested in seeing impact assessments/BOTECs (EDIT: Back of the envelope calculations) anyway, though like you(?) I think it is likely that the time costs are greater than the money costs. 

Comment by Linch on My Ordinary Life: Improvements Since the 1990s · 2021-09-09T15:46:26.477Z · EA · GW

While I think celebrating progress is good, and having a clearer "sense" of the data is good, I think the changes in the post are both qualitatively and quantitatively tiny compared to eg, changes my family members in China experienced between 1980 and 2000 or between 2000 and 2020. So I do think having your priors be formed by typical experiences in Western countries would give you a (relative) general sense of global stagnation. 

Comment by Linch on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-08T16:59:01.134Z · EA · GW

Another potential difference is that you don't get do-overs: the more senior person can't later write a paper that follows exactly the same idea but that's written to a much higher standard, because there's more of a requirement that each paper brings original ideas.

Hmm taking a step back, I wonder if the crux here is that you believe(?) that the natural output for research is paper-shaped^, whereas I would guess that this would be the exception rather than the norm, especially for a field that does not have many very strong non-EA institutions/people (which I naively would guess to be true of EA-style TAI governance).

This might be a naive question, but why is it relevant/important to get papers published if you're trying to do impactful research? From the outside, it seems unlikely that all or most good research is in paper form, especially in a field like (EA) AI governance where (if I understand it correctly) the most important path to impact (other than career/skills development) is likely through improving decision quality for <10(?) actors. 

If you are instead trying to play the academia/prestige game, wouldn't it make more sense to optimize for that over direct impact? So instead of focusing on high-quality research on important topics, write the highest-quality (by academic standards) paper you can in a hot/publishable/citable topic and direction. 

^ This is a relevant distinction because originality is much more important in journal articles than other publication formats, you absolutely can write a blog post that covers the same general idea as somebody else but better, and AFAIK there's nothing stopping a think tank from "revising" a white paper covering the same general point but with much better arguments.

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-08T00:02:26.298Z · EA · GW

Hmm I have conflicting feelings about this.  I think whenever you add additional roadblocks or other limitations on criticism, or suggestions that criticisms can be improved, you 

a) see the apparent result that criticisms that survive the process will on average be better. 

b) fail to see the (possibly larger) effect that there's an invisible graveyard of criticisms that people choose not to voice because it's not worth the hassle. 

At the same time, being told that your life work is approximately useless is never a pleasant feeling, and it's not always reasonable to expect people to handle it with perfect composure (Thankfully nothing of this magnitude has ever happened to me, but I was pretty upset when an EA Forum draft I wrote in only a few days had to be scrapped or at least rewritten because it assumed a mathematical falsehood). So while I think Mike's responses to Abby are below a reasonable bar of good forum commenting norms, I think I have more sympathy for his feelings and actions here than Greg seems to.

So I'm pretty conflicted. My own current view is that I endorse Abby's comments and tone as striking the right balance for the forum, and I endorse Greg's content but not the tone. 

But I think reasonable people can disagree here, and we should also be mindful that when we ask people to rephrase substantive criticisms to meet a certain stylistic bar (see also comments here), we are implicitly making criticisms more onerous, which arguably has pretty undesirable outcomes. 

Comment by Linch on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-07T23:24:01.441Z · EA · GW

Ideally, someone senior tells you what to work on. But this is time-expensive for them, and they don’t want to give away their best ideas to somebody who might execute them badly. So more realistically…

This seems very surprising to me. Unless by "best ideas" you mean "literally somebody's top idea" or by "someone senior" you mean Nick Bostrom? 

My impression from talking to friends working in ML is that usually faculty have ideas that they'd be excited to see their senior grad students to work on, senior grad students have research ideas that they'd love for junior grad students to implement, and so forth. 

Math and theoretical CS likewise have lists of open problems.

Similarly, in (non-academic EA) research I have way too many ideas that I can't work on myself, and I've frequently seen buffets of potential research topics/ideas that more senior researchers propose. 

My general impression is that this is the norm in EA research? When people choose not to work on other people's ideas, it's usually due to a combination of personal fit and arrogance in believing your own ideas are more important (or depending on the relevant incentives, other desiderata like "publishable", "appealing to funders", or "tractable"), not because of a lack of ideas! 

Very surprised to hear about your experiences.

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-07T22:20:59.585Z · EA · GW

I think of the metrics I mentioned above as proxies rather than as the underlying targets, which is some combination of:

a) Is STV true?
b) Conditional upon STV being true, is it useful?

What my forecasting questions aimed to do is shedding light on a). I agree that academia and citations isn't the best proxy. They may in some cases have conservatism bias (I think trusting the apparent academic consensus on AI risk in 2014 would've been a mistake for early EAs), but are also not immune to falseties/crankery (cf replication crisis). In addition, standards for truth and usefulness are different within EA circles than academia, partially because we are trying to answer different questions.

This is especially an issue as the areas that QRI is likely to interact with (consciousness, psychedelics) seem from the outside to be more prone than average to falseness and motivated cognition, including within academia.

This is what I was trying to get at with "will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?" because Luke is a non-QRI affiliated person within EA who's a) respected and b) have thought about concepts adjacent to QRI's work. Bearing in mind that Luke is very far from a perfect oracle, I would still trust Luke's judgement on this more than an arbitrarily selected academic in an adjacent field. 

I think the actual question I'm interested in is something like "In X year, will a panel of well-respected EAs a) not affiliated with QRI and b) have very different thoughts from each other and c)who have thought about things adjacent  to QRI's work have updated to believing STV to be substantively true" but I was unable to come up with a clean question operationalization in the relatively brief amount of time I gave myself to come up with this. 

People are free to counterpropose and make their own questions. 

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-07T09:55:32.511Z · EA · GW

Note that the 2nd question is about total citations rather than of one paper, and 3k citations doesn't seem that high if you're introducing an entirely new subfield (which is roughly what I'd expect if STV is true). The core paper of Friston's free energy principle has almost 5,000 citations for example, and it seems from the outside that STV (if true) ought to be roughly as big a deal as free energy. 

For a sense of my prior beliefs about EA-encouraged academic subfields, I think 3k citations in 10 years is an unlikely but not insanely high target for wild animal welfare (maybe 20-30%?), and AI risk is likely already well beyond that (eg >1k citations for Concrete Problems alone).

Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-07T09:16:31.489Z · EA · GW

Hi, all. Talk is cheap, and EA Forum karma may be insufficiently nuanced to convey substantive disagreements. 

I've taken the liberty to sketch out several forecasting questions that might reflect underlying differences in opinion. Interested parties may wish to forecast on them (which the EA Forum should allow you to do directly, at least on desktop) and then make bets accordingly.

Feel free to also counterpropose (and make!) other questions if you think the existing question operationalizations are not sufficient (I'm far from knowledgeable in this field!). 


Comment by Linch on A Primer on the Symmetry Theory of Valence · 2021-09-07T01:21:57.337Z · EA · GW

For what it's worth, I read this comment as constructive rather than non-constructive. 

If I write a long report and an expert in the field think that the entire premise is flawed for specific technical reasons, I'd much rather them point this out rather than for them to worry about niceness and then never getting around to mentioning it, thus causing my report to languish in obscurity without me knowing why (or worse, for my false research to actually be used!)  

Comment by Linch on When pooling forecasts, use the geometric mean of odds · 2021-09-06T20:26:29.368Z · EA · GW

Geometric mean is just a really useful tool for estimations in general. It also makes a lot of sense for aggregating results other than probabilities, eg for different Fermi estimates of real quantities.

Comment by Linch on When pooling forecasts, use the geometric mean of odds · 2021-09-06T20:22:50.528Z · EA · GW

That said, I think it's worth pointing out the case where arithmetic mean of probabilities is exactly right to use: if you think that exactly one of the estimates is correct but you don't know which (rather than the usual situation of thinking they all provide evidence about what the correct answer is).

To extend this and steelman the case for arithmetic mean of probabilities (or something in that general direction) a little, in some cases this seems a more intuitive formulation of risk (which is usually how these things are talked about in EA contexts), especially if we propagate further to expected values or value of information concerns.

Eg, suppose that we ask 3 sources we trust equally about risk from X vector of an EA org shutting down in 10 years. One person says 10%, 1 person says 0.1%, 1 person says 0.001%. 

Arithmetic mean of probabilities gets you ~ 3.4%, geometric mean of odds gets you ~0.1%. 0.1% seems comfortably below the background rate of organizations dying, that in many cases it's not worth the value of information to investigate further. Yet naively this seems to be too cavalier if one out of three sources thinks there's a 10% chance of failure from X vector alone!  

Also as a mild terminological note, I'm not sure I know what you mean by "correct answer" when we're referring to probabilities in the real world. Outside of formal mathematical examples and maybe some quantum physics stuff, probabilities are usually statements about our own confusions in our maps of the world, not physically instantiated in the underlying reality. 

Comment by Linch on When pooling forecasts, use the geometric mean of odds · 2021-09-06T20:05:52.148Z · EA · GW

(I have not read the post)

I endorse these implicit odds, based on both theory and some intuitions from thinking about this in practice. 

Comment by Linch on Linch's Shortform · 2021-09-06T19:21:06.901Z · EA · GW

Hmm one recent example is that somebody casually floated to me an idea that can potentially entirely solve an existential risk (though the solution might have downside risks of its own) and I realized then that I had no idea how much to price the solution in terms of EA $s, like whether it should be closer to 100M, 1B or $100B. 

My first gut instinct was to examine the solution and also to probe the downside risks, but then I realized this is thinking about it entirely backwards. The downside risks and operational details don't matter if even the most optimistic cost-effectiveness analyses isn't enough to warrant this being worth funding! 

Comment by Linch on Linch's Shortform · 2021-09-06T19:17:35.547Z · EA · GW

I agree with the general flavor of what you said, but am unsure about the exact numbers.

Comment by Linch on Linch's Shortform · 2021-09-06T02:29:05.312Z · EA · GW

General suspicion of the move away from expected-value calculations and cost-effectiveness analyses.

This is a portion taken from a (forthcoming) post about some potential biases and mistakes in effective altruism that I've analyzed via looking at cost-effectiveness analysis. Here, I argue that the general move (at least outside of human and animal neartermism) away from Fermi estimates, expected values, and other calculations just makes those biases harder to see, rather than fix the original biases.

I may delete this section from the actual post as this point might be a distraction from the overall point.


I’m sure there are very good reasons (some stated, some unstated) for moving away from cost-effectiveness analysis. But I’m overall pretty suspicious of the general move, for a similar reason that I’d be suspicious of non-EAs telling me that we shouldn’t use cost-effectiveness analyses to judge their work, in favor of say systematic approaches, good intuitions, and specific contexts like lived experiences (cf. Beware Isolated Demands for Rigor):

I’m sure you have specific arguments for why in your case quantitative approaches aren’t very necessary and useful, because your uncertainties span multiple orders of magnitude, because all the calculations are so sensitive to initial assumptions, and so forth. But none of these arguments really point to verbal heuristics suddenly (despite approximately all evidence and track records to the contrary) performing better than quantitative approaches. 

In addition to the individual epistemic issues with verbal assessments unmoored by numbers, we also need to consider the large communicative sacrifices made by not having a shared language (mathematics) to communicate things like uncertainty and effect sizes. Indeed, we have ample evidence that switching away from numerical reasoning when communicating uncertainty is a large source of confusion

To argue that in your specific situation, verbal judgment is superior without numbers than with numbers, never mind that your proposed verbal solutions obviates the biases associated with trying to do numerical cost-effectiveness modeling of the same, the strength of your evidence and arguments needs to be overwhelming. Instead, I get some simple verbal heuristic-y arguments, and all of this is quite suspicious.

Or more succinctly: 

It’s easy to lie with numbers, but it’s even easier to lie without them

So overall I don’t think moving away from explicit expected value calculations and cost-effectiveness analyses is much of a solution, if at all, for the high-level reasoning mistakes and biases that are more clearly seen in cost-effectiveness analyses. Most of what the shift away from EV does is makes things less grounded in reality, less transparent and harder to critique (cf. “Not Even Wrong”).

Comment by Linch on Linch's Shortform · 2021-09-05T00:05:46.835Z · EA · GW

I've read his post on this before. I think this question is substantively easier for heavy SFE-biased views, especially if you pair it with some other beliefs Brian has.

that's pretty damning for the WAW movement, unless climate change prevention is not the highest-leverage influence against WAS (a priori, it seems unlikely that climate change prevention would have the highest positive influence).

I read it as more damming for climate change people for what it's worth, or at least the ones who claim to be doing it for the animals.

Comment by Linch on A Sequence Against Strong Longtermism · 2021-09-04T23:51:56.513Z · EA · GW

Yes, this is what I mean. I was unsure how to be diplomatic about it.