Posts

Samotsvety Nuclear Risk update October 2022 2022-10-03T18:10:42.477Z
Impactful Forecasting Prize Results and Reflections 2022-03-29T16:16:16.052Z
Samotsvety Nuclear Risk Forecasts — March 2022 2022-03-10T18:52:33.570Z
Comparing top forecasters and domain experts 2022-03-06T20:43:36.510Z
Impactful Forecasting Prize meetup 2022-02-10T17:12:39.035Z
Impactful Forecasting Prize meetup 2022-02-10T17:08:17.886Z
We are giving $10k as forecasting micro-grants 2022-02-08T12:20:25.160Z
Impactful Forecasting Prize for forecast writeups on curated Metaculus questions 2022-02-04T20:06:11.797Z
Prediction Markets in The Corporate Setting 2021-12-31T17:10:34.916Z

Comments

Comment by Misha_Yagudin on Questions on databases of AI Risk estimates · 2022-10-02T18:25:24.356Z · EA · GW

Nicole Noemi gathers some forecasts about AI risk (a) from Metaculus, Deepmind co-founders, Eliezer Yudkowsky, Paul Christiano, and Aleja Cotra's report on AI timelines.

h/t Nuño

Comment by Misha_Yagudin on Ingredients for creating disruptive research teams · 2022-10-02T18:22:30.193Z · EA · GW

Terri Griffith [thinks](https://econjwatch.org/File+download/1236/UnderappreciatedWorksSept2022.pdf?mimetype=pdf Research Team Design and Management for Centralized R&D is their most neglected paper. They summarize it as follows:

It is a field study of 39 research teams within a global Fortune 100 science/technology company. As we write in the abstract, we demonstrate that “teams containing breadth of both research and business unit experience are more effective in their innovation efforts under two conditions: 1) there must be a knowledge-sharing climate in the team (arguably allowing the team to have access to the knowledge developed through the members’ breadth of experience) and 2) the team leader also has a breadth of research and business experience allowing for the member breadth to be knowledgably managed.” With 13 years perspective, I still find these results valuable and often share them in my innovation management courses.

Comment by Misha_Yagudin on 9/26 is Petrov Day · 2022-09-26T12:29:29.916Z · EA · GW

And the FLI award is probably worth mentioning.

Comment by Misha_Yagudin on An experiment eliciting relative estimates for Open Philanthropy’s 2018 AI safety grants · 2022-09-12T12:05:22.907Z · EA · GW

A slightly edited section of my comment on the earlier draft:

I lean skeptical about "relative pair-wise comparisons" after participating: I think people were surprised by their aggregate estimates (e.g., I was very surprised!); I think later convergence was due to common sense and mostly came from people moving points between interventions and not from pair-wise anything;

I think this might be because I am unconfident about eliciting distributions with Squiggle. As I don't have good intuition about how a few log-normals with 80% probability between xx and yy would compare to each other after aggregations (probably this is common, see 2a). After I did my point estimates + my CI via Squiggle for everything alltogether, I think they didn't match each other that well. Maybe that's because lognormal is right-skewed and fairly heavy-tailed?

Comment by Misha_Yagudin on Join ASAP (AI Safety Accountability Programme) 🚀 · 2022-09-10T12:33:44.162Z · EA · GW

For sharing updates on Slack I would recommend https://geekbot.com/

Comment by Misha_Yagudin on Shareholder activism · 2022-09-10T11:46:24.848Z · EA · GW

Due to the rise of index funds (they "own" > 1/5 of American public companies), it seems that an alternative strategy might be trying to rise in the ranks of firms like BlackRock, Vanguard, or SSGA. It's not unprecedented for them to take action (partly for selfish reasons); here are examples of BlackRock taking stances on environmental sustainability and coronavirus cure/vaccine.

Here is a paper exploring the potential implications of the rise of index funds and their stewardship: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3282794

Comment by Misha_Yagudin on [Cause Exploration Prizes] Training experts to be forecasters · 2022-09-10T10:09:49.967Z · EA · GW

A few considerations against, tied by generalism enables scale theme:

(1) There are a lot of domains where one can become an expert: it feels infeasible to train and select very capable forecasters in all of them. Being generally thoughtful person/forecaster allows to somewhat successfully go into areas outside your immediate expertise.

Training/selecting experts in a few especially important niches (e.g., AI, biosecurity, and certain topics in geopolitics) seems good and feasible.

(2) But at times of crisis, experts' time is much more valuable than generalist's time. Even now, it's often the case that competent forecasters are quite busy with their main jobs — it's not unlikely that competent forecaster-experts should be doing something different from forecasting.

Comment by Misha_Yagudin on Samotsvety's AI risk forecasts · 2022-09-10T07:58:47.398Z · EA · GW

To add to Eli's comment, I think on such complex topics, it's just common for even personal estimates to fluctuate quite a bit. For example, here is an excerpt from footnote 181 of Carlsmith report:

[...] And my central estimate varies between ~1-10% depending on my mood, what considerations are salient to me at the time, and so forth. This instability is yet another reason not to put too much weight on these numbers.

Comment by Misha_Yagudin on Samotsvety's AI risk forecasts · 2022-09-09T13:09:36.788Z · EA · GW

Good question. I think AI researchers views inform/can inform me. A few examples from the recent NLP Community Metasurvey. I would quote bits from this summary.

Few scaling maximalists: 17% agreed that Given resources (i.e., compute and data) that could come to exist this century, scaled-up implementations of established existing techniques will be sufficient to practically solve any important real-world problem or application in NLP.

This was surprsing and updated me somewhat against shorter timelines (and higher risk) as, for example, it clashes with the "+12 OOMS Enough" premise of the Kokotajlo's argument for short timelines of the Carlsmith report (see also this and his review of Carlsmith report).

NLP is on a path to AGI: 58% agreed that Understanding the potential development of artificial general intelligence (AGI) and the benefits/risks associated with it should be a significant priority for NLP researchers. Related: 57% agreed that Recent developments in large-scale ML modeling (such as in language modeling and reinforcement learning) are significant steps toward the development of AGI.

If these numbers were significantly lower or higher, it would also probably update my views.

AGI could be catastrophic: 36% agreed that It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war. 46% of women and 53% of URM respondents agreed. The comments suggested that people took a pretty wide range of interpretations to this, including things like OOD robustness failures leading to weapons launches.

This number is puzzling and hard to interpret. It seems appropriate in light of AI Impacts' What do ML researchers think about AI in 2022? where "48% of respondents gave at least 10% chance of an extremely bad outcome".

I don't fully understand what this implies about the ML community's views on AI alignment. But I can see myself updating positively if these concerns would lead to more safety culture, alignment research, etc.

Comment by Misha_Yagudin on The Nietzschean Challenge to Effective Altruism · 2022-08-31T13:48:10.697Z · EA · GW

I want to push back on the practical upshot (ii).

Getting rid of extreme suffering (and maybe eradicating suffering altogether) seems like a huge cultural achievement. I think it'd be hard to deny that, even for someone who treats any personal pursuit of trivial comforts as an ultimate distraction. For example, the eradication of smallpox is in the same reference class and is clearly on the list of the peak human achievements of the 20th century. It required worldwide cooperation and logistic, advances in medical science and technology, and probably much more.

Comment by Misha_Yagudin on The Operations team at CEA transforms · 2022-08-02T19:35:51.448Z · EA · GW

Especially excited about "Immigration Specialist."

Comment by Misha_Yagudin on Import your EAG(x) info to your profile & other new features (Forum update June 2022) · 2022-07-06T15:05:15.319Z · EA · GW

I dunno “About the author” looks really annoying to me. It triggers ~disgust for "unwarranted status/authority projection" social games associated with some other outlets. It's fine to say that you are experts and think X, Y, and Z (but your reasons should be good still); it's also fine to have a friendly oneliner to remind a reader who the author is. But having all your bio recalled after each post is a bit "meh".

Unfortunately, it seems there is no way to hide it in Account Settings > Site Customizations. If anyone is in my shoes, I killed it with the uBlock origin element picker.

Comment by Misha_Yagudin on What’s the theory of change of “Come to the bay over the summer!”? · 2022-06-20T14:18:33.848Z · EA · GW

Yeah, it would probably be good if people redirected this energy to climbing ladders in the government/civil service/military or important powerful corporate institutions. But I guess these ladders underpay you in terms of social credit/inner ringing within EA. Should we praise people aiming for 15y-to-high-impact careers more?

Comment by Misha_Yagudin on What are EA's biggest legible achievements in x-risk? · 2022-06-19T18:44:13.538Z · EA · GW

Thank you for you work!

Comment by Misha_Yagudin on Michael Nielsen's "Notes on effective altruism" · 2022-06-16T17:10:18.484Z · EA · GW

A useful piece of context. When asked about recommendations on charitable giving Michael Nielsen writes:

Same answer as @TheZvi's earlier, I'm afraid. I'm pretty Hayekian; I wish there were good price signals here! In some ways I view that as what EA is doing: it is trying to use community argument and institutions to price public goods (& the like) appropriately.

Comment by Misha_Yagudin on Apollo Academic Surveys · 2022-05-20T17:16:23.227Z · EA · GW

Yes, they did, see e.g. https://twitter.com/ApolloSurveys/status/1527362513200787469

Comment by Misha_Yagudin on Potatoes: A Critical Review · 2022-05-10T20:55:07.970Z · EA · GW

Hm, is it just through calories or maybe through micronutrients as well: potatoes are high in potassium, vitamin C, B6, and K1 compared to other staple foods? Footnote 6 tends to suggest that it's mostly about calories.

Comment by Misha_Yagudin on Effective altruism’s odd attitude to mental health · 2022-04-29T19:30:28.539Z · EA · GW

Thanks for clarifying! I think our misunderstanding comes from different framings: EA meta/infrastructure v. global health; within global health, MH v. disease/poverty alleviation. I agree with you and calebp on the former.

Comment by Misha_Yagudin on Effective altruism’s odd attitude to mental health · 2022-04-29T15:55:47.269Z · EA · GW

That's fairly dismissive. You could have written:

If imposter syndrome and other easily preventable/treatable debilitating mental issues were common among EAs, I'd guess that should be a much higher priority to address than poor "physical" health among EAs.

Isn't it in the end about what's more cost-effective? I can interpret you as pointing to "if ~depression was more cost-effective to address than ~smallpox, it would have been addressed first in the developed nations," this might be a good heuristic but a well-thought cost-effectiveness estimate is more convincing to me. Seems like criticizing MichaelPlant's cost-effectiveness estimates could have been more productive here. (TBC: I myself haven't engaged with HLI's work/methal health charities much).

And mental health being comparatively cost-effective doesn't sound ridiculous to me. StrongMinds attracted some donations from EAs in the past including GiveWell staff.

(I sorta feel that this comment and 23 upvotes / 7 votes support OP's observation that people seem to be dismissive about mental health as global health and development intervention.)

Comment by Misha_Yagudin on Impactful Forecasting Prize Results and Reflections · 2022-03-29T17:37:56.041Z · EA · GW

Yes, thank you; that makes sense and is very helpful!

Comment by Misha_Yagudin on Russia-Ukraine Conflict: Forecasting Nuclear Risk in 2022 · 2022-03-25T03:18:47.736Z · EA · GW

Is any forecaster or organizer willing to bet with me on "Will Russia place any nuclear weapons in Belarus before 2023?" at 3:7 odds (with me on "no")?

Comment by Misha_Yagudin on Predicting for Good: Charity Prediction Markets · 2022-03-23T17:24:13.086Z · EA · GW

re: 1 — agree, MM is nicer than Poly. And I view UX as a very important issue for adoption (think of Robinhood).

re: 2 — would be great if you'd work that out!

re: 3 — I think just using the proper scoring rules (like log or Brier) is good enough; what are the problems with these? Smart aggregation (based on track-record and some other info) would allow leveraging non-superforecasters (likely through putting weight on prospective superforecasters). I think another way to participate in prediction pools and markets is to bring new information and considerations, this can be rewarded through Reddit's karma, r/changemyview deltas, or with StackOverflow upvotes (one interesting challenge is to figure that out).

Comment by Misha_Yagudin on Predicting for Good: Charity Prediction Markets · 2022-03-23T17:08:13.603Z · EA · GW

Austin replied to the earlier version of the comment (my bad), which is similar in substance on the relevant market: https://manifold.markets/Austin/will-the-ftx-future-fund-decide-to

Thanks for your comment, Misha (and congrats on the FF grant)! I've been a huge fan of your work, especially your report on Prediction Markets in the Corporate Setting https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/prediction-markets-in-the-corporate-setting

  1. Agreed that the marginal dollar moved between different weird altruistic donors is less impactful than winning a bet on Polymarket; but there are a host of usability issues in Polymarket (crypto onboarding, market availability, UX design) that we think we solve much better in Manifold. Over time, the long-term vision of this would be to draw in more charitable dollars from outside the EA community as well.

  2. The externalities of demanding constant vigilance are a good point, and something we do take seriously; I'd like to work out an interface/design a mechanism that allows a trader to input a true probability and be rewarded, without needing to check in on their position constantly. Maybe this just is being a prediction pool!

  3. I think prediction pools are quite promising, though I'm not sure if a good (easy-to-use, incentive-aligned) mechanism has been worked out; do you have any pointers to setups/implementations/designs of prediction pools that you think are good? I'm especially curious if these are framed in a way that allow a normal person (aka not superforecaster) to understand the system and meaningfully contribute.

Would love to chat more; happy to discuss in this thread, or feel free to find a time on https://calendly.com/austinchen/manifold !

(I am out of funds to reply to them there :P, this should be seen as foot voting for their platform as a whole.)

Comment by Misha_Yagudin on Predicting for Good: Charity Prediction Markets · 2022-03-23T17:06:06.161Z · EA · GW

I am skeptical.

The market will mainly attract altruistic traders as you can only make altruistic gains. Further, prediction markets are for unusual people[1]. So we will end up with unusual altruistic donors.

I see two motivations for such donors: contributing to the information generating system and distributing money to grantmakers with better judgment.

Re: infogain. I am fairly skeptical about low-volume prediction markets being competitive to similar (in the number of participants) prediction pools. See https://bit.ly/arb-prediction-markets-vs-opinion-pools. But I am excited to try it and compare forecasts to predictions pools on other platforms?[2]

Re: allocation. For unusual altruistic donors, the upside feels small: the system reallocates the money of other unusual donors, so donated earnings are not fully contafactual. While money goes to people with better judgment, it's unclear that their returns/earnings would be higher than on other platforms like Polymarket, where money is non-altruistic.[3]

Lastly, I think the externalities are quite big for prediction markets (as opposed to prediction pools): markets require constant vigilance as with every update, you win/lose. Active traders will spend a lot of attention on their trades. This is especially bad given that traders are altruistically minded.[4]

(At the same time, I am fairly excited to see more experiments in the space.)

(I have/had a CoI as I applied/received a FF grant for work on epistemic institutions.)


  1. Even donor lotteries do not attract broad participation despite a strong case for using them. ↩︎

  2. Happy to collaborate message me here or preferably at mike.yagudin@gmail.com ↩︎

  3. This is another issue; even on fairly liquid Polymarket, a lot of things seem mispriced to me. Nuño and I bet there, so hopefully, these are not empty words. ↩︎

  4. Cf. donor lotteries. One of their aims is to cut research costs for donors. These costs are highest for donors with good judgment as their opportunity costs are the highest. ↩︎

Comment by Misha_Yagudin on Is there an EA grants database? · 2022-03-23T14:42:55.508Z · EA · GW

Yes, I intend to but like everything, it might rod. What are your other needs here?

Comment by Misha_Yagudin on Is there an EA grants database? · 2022-03-22T22:43:01.189Z · EA · GW

I recently commissioned one for EA Funds, it would be fairly easy to add OP and SFF into it.

Comment by Misha_Yagudin on Is there an EA grants database? · 2022-03-22T22:42:37.631Z · EA · GW

I recently commissioned one for EA Funds, it would be fairly easy to add OP and SFF into it.

Comment by Misha_Yagudin on Let Russians go abroad · 2022-03-13T15:27:05.397Z · EA · GW

My understanding is that this is due to mandatory legal reasons. I believe Philip's situation will be resolved via another donor soon.

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-12T21:51:01.396Z · EA · GW

So to be 100x of the default rate, one should have put less than 1% on events in Ukraine unfolding as they are now. This feels too low to me (and I registered some predictions in personal communications about a year ago supporting that — thou done in a hurry of email exchange).

I think a reasonable forecaster working on nuclear risk should have put significant worsening of the situation at above 10% (just from crude base rates of Russia's "foreign policy" and past engagement in Ukraine). And I think among Russia-Ukrainian conflicts, this one, while surprising (to me subjectively) in a few ways, is not in the bottom 10% (and not even in the bottom third for me in terms of Russia-NATO tensions — based on the past forecasts). So one should go above baseline but no more than an order of magnitude, imo.

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-12T20:04:28.448Z · EA · GW

I will just note that 10x with 20% (= 25% - 5%) and 100x with 5% would/should dominate EV of your estimate. P = .75 * X + .20 * 10 * X + .05 * 100 * X = .75 X + 7 X = 7.75 X.

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-12T17:43:27.854Z · EA · GW

Thanks that is useful and interesting! (re: edit — I agree but maybe at 90% given some uncertainty about readiness.)

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-12T17:16:55.342Z · EA · GW

One can pull off quite a bit of wordplay/puns (in Russian) with "Samotsvety" (about forecasting), which I find adorably cringy. Alas, even Nuño doesn't remember why the name was chosen.

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-12T00:08:12.156Z · EA · GW

Yes, but you also should update on incidents not leading to a catastrophe. If Nuño and I scratched math correctly, you should feel times more doomed, where n is the number of incidents. If it's the 8th accident, you should only feel like only ~1% more doomed.

Comment by Misha_Yagudin on Samotsvety Nuclear Risk Forecasts — March 2022 · 2022-03-11T03:59:23.580Z · EA · GW

It's not about the odds; it's about Beta distribution. You are right to be suspicious about the addition of odds, but there is nothing wrong with adding shape parameters of Beta distributions.

I don't want to go into many details but a teaser for readers:

  • One want's to figure out , probability of some event happening.
  • One starts with some prior about ; if it's uniform over , the prior is .
  • If one then observes successes and failures, one would update to .
  • If one then wants to get the probability of success next time, one needs to integrate over possible (basically to take expected value). It would lead to .

For Laplace's law of succession, you start with , observe failures and update to . And your estimate is . In this context, Pablo suggests starting with different prior of (which corresponds to the probability of success and odds of ) to then update to after observing failures.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-10T15:34:39.344Z · EA · GW

As a semi-active user of prediction markets and a person who looked up a bunch of studies about them, I don't see that many innovations or at least anything that crucially changes the picture. I would be excited to be proven wrong, and am curious to know what you would characterize as advances in capability and methodology.

I am partly basing my impression on Mellers & Tetlock (2019), they write "We gradually got better at improving prediction polls with various behavioral and statistical interventions, but it proved stubbornly hard to improve prediction markets." And my impression is that they experimented quite a bit with them.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-09T04:14:50.086Z · EA · GW

Appreciate that, Yonatan! :)

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-08T23:00:42.794Z · EA · GW

Thank you, Tim! Likely partly due to is my impressions of what's going on based on existing research; I think we know that it is "likely partly" but probably not much more based on current literature.

The line of reasoning which I find plausible is "GJP PM and GJP All Surveys Logit" is more or less the same pool of people but the one aggregation algorithm is much better than another; it's plausible that "IC All Surveys Logit would improve on ICPM quite dramatically." And because the difference between GJP PM and ICPM is small it feels plausible that if the best aggregation method would be applied to IC, IC would cut the aforementioned 30% gap.

(I am happy to change my mind upon seeing more research comparing strong forecasters and domain experts.)

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-08T18:07:40.672Z · EA · GW

Upd 2022-03-14: Good Judgement Inc representative confirmed that Goldstein et al (2015) didn't have a superforecaster-only pool. Unfortunately, the citations above are indeed misleading; as of now, we are not aware of research comparing superforecasters and ICPM.

Upd 2022-03-08: after some thought, we decided to revisit the post to be more precise. While this study has been referenced multiple times as superforecasters vs ICPM it's unclear whether one of the twenty algorithms compared used only superforecasters (which seems plausible, see below). We still believe that Goldstein et al bear on how well the best prediction pools do, compared to ICPM. The main question about All Surveys Logit, whether the performance gap is due to the different aggregation algorithms used, also applies to claims about superforecasters.

  • Co-investigators of GJP summarize the result that way (comment);
  • Good Judgment Inc. uses this study on their page Superforecasters vs. ICPM (comment);
  • further, in private communications people assumed that narrative;
  • my understanding of data justifies the claim (comment).

Lastly, even if we assume that claims of superforecasters performance in comparison with IC haven't been backed by this (or any other) study[1], the substantive claim hold: the 30% edge is likely partly due to the different aggregation techniques used stands.


  1. As I reassert in this comment, everyone refers to this study as a justification; and upon extensive literature search, I haven't found other comparisons. ↩︎

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-08T15:23:52.781Z · EA · GW

All Survey Logit was the best method out of the many methods the study tried. Their class of methods is flexible enough to include superforecasters as they were trying weighting forecasters by past performance (and as the research was done based on year 3 data the superforecasters were a salient option). By construction ASL is superforecaster level or above.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-08T15:01:24.728Z · EA · GW

Thanks for catching a typo! Appreciate the heads up.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T18:26:31.105Z · EA · GW

Thanks for engaging with our post!

Here is Mellers et al. (2017) about the study:

Each year, the top 2% of subjects were designated "superforecasters" and were assigned to work together in elite teams. In this richer setting, superforecasters became more accurate and resisted regression to the mean, suggesting that their ac-curacy was driven at least in part by skill, rather than luck (Mellers, Stone, Atanasov, Rohrbaugh, Metz, Ungar, Bishop, Horowitz, Merkle & Tetlock, 2015b). Indeed, using Brier scores to measure accuracy, Goldstein, Hartman, Comstockand Baumgarten (2016) found that superforecasters outperformed U.S. intelligence analysts on the same questions by roughly 30%.

(Emphasis mine.)

I believe their assessment of whether it's fair to call one of "GJP best methods" "superforecasters" is more authoritative as the term originated from their research (and comes with a better understanding of methodology).

Anyways, the "GJP best method" used all Brier score boosting adjustments discussed in the literature (maybe excluding teaming), including selecting individuals (see below). And, IIRC, superforecasters are basically forecasters selected based on their performance.

Finally, we compare ICPM accuracy to that of GJP's single most accurate CW method for the set of questions being analyzed—a method called "All Surveys Logit." All Surveys Logit takes the most recent forecasts from a selection of individuals in GJP's survey elicitation condition, weights them based on a forecaster's historical accuracy, expertise, and psychometric profile, and then extremizes the aggregate forecast (towards 1 or 0) using an optimized extremization coefficient.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T17:20:50.543Z · EA · GW

It's indeed the case that GJP was 34.7% better than the ICPM. But it's not the case that GJP participants were 34.7% better than intelligence analysts. The intelligent analyst used prediction markets that are generally worse than prediction pools (see Appendix A), so we are not comparing apples to apples.

It would be fair to judge IC for using prediction markets rather than prediction pools after seeing research coming out of GJP. But we don't know how an intelligence analyst prediction pool would perform compared to the GJP prediction pool. We have reasons to believe that difference might not be that impressive based on ICPM vs GJP PM and based on Sell et al (2021).

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T17:07:57.977Z · EA · GW

Indeed, but the misconception/lack of nuance is specifically about 30% here is Wikipedia on Good Judgement Project. I guess it's either about looking at preliminary data or rounding.

The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T17:01:43.141Z · EA · GW

Good catch, Tim! Well, at least Good Judgement Inc. (and some papers I've seen) cite Goldstein et al (2015) straight after David Ignatius's 30% claim: https://goodjudgment.com/resources/the-superforecasters-track-record/superforecasters-vs-the-icpm/

If you by any chance have another paper[1] or resource in mind regarding the 30% claim, I would love to include it in the review.


  1. Note that Goldstein et al don't make that claim themselves, their discussion and conclusion are nuanced. ↩︎

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T16:06:40.198Z · EA · GW

Thank you! We might consider editing the summary. This particular point is mostly supported by our takes on Goldstein et al (2015) and by Appendix A.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T16:01:26.412Z · EA · GW

The linked story doesn't cite another paper, so it's hard to guess their actual source. Generally, academic research takes a while to be written and get published; the 2015 version of the paper seems to be the latest draft in circulation. It's not uncommon to share and cite papers before they get published.

Comment by Misha_Yagudin on Comparing top forecasters and domain experts · 2022-03-07T13:18:25.405Z · EA · GW

Interesting side-finding: prediction markets seem notably worse than cleverly aggregated prediction pools (at least when liquidity is as low as in the play markets). Not many studies, but see Appendix A for what we've found.

Comment by Misha_Yagudin on We are giving $10k as forecasting micro-grants · 2022-03-06T14:55:23.139Z · EA · GW

Finally, an anonymous benefactor increased the size of this newsletter's microgrants program (a), so if you have a forecasting or epistemics-related project you'd be keen to implement, consider applying. We recently gave our first $5k grant to Clay Graubard, for work related to his quantified journalism (a) on the Ukraine invasion.

via https://forum.effectivealtruism.org/posts/xpkpXq57mXmLbgkSC/forecasting-newsletter-february-2022#Forecasting_Job_Board

Comment by Misha_Yagudin on Nuclear attack risk? Implications for personal decision-making · 2022-02-28T12:06:10.296Z · EA · GW

That seems way too high to me: are you willing to bet at 5%? (For epistemic purposes only. I hope no one reading will be offended.) If so confirm here and PM me on Forum to figure out the details.

Comment by Misha_Yagudin on Nuclear attack risk? Implications for personal decision-making · 2022-02-28T12:02:00.351Z · EA · GW

Seems worth mentioning Russian nuclear policy. (I believe my interpretations/translators are fairly accurate but might miss diplomatic/legal nuance.)

Russian war doctrine (#27) of 2014 prohibits the first strike unless WMD are used against it or its allies or when aggression with conventional weapons greatly endangers Russia's existence:

Российская Федерация оставляет за собой право применить ядерное оружие в ответ на применение против нее и (или) ее союзников ядерного и других видов оружия массового поражения, а также в случае агрессии против Российской Федерации с применением обычного оружия, когда под угрозу поставлено само существование государства. Решение о применении ядерного оружия принимается Президентом Российской Федерации.

This was further clarified in section III of the 2020 nuclear weapons policy. Conditions are (a) launch of ballistic missiles against Russia or allies; (b) use of nuclear or other WMD against Russian or allies; (c) infiltrating with critical infrastructure, which might disrupt the second-strike capabilities; (d) aggression with conventional weapons greatly endangers Russia's existence.

  1. Условиями, определяющими возможность применения Российской Федерацией ядерного оружия, являются: а) поступление достоверной информации о старте баллистических ракет, атакующих территории Российской Федерации и (или) ее союзников; б) применение противником ядерного оружия или других видов оружия массового поражения по территориям Российской Федерации и (или) ее союзников; в) воздействие противника на критически важные государственные или военные объекты Российской Федерации, вывод из строя которых приведет к срыву ответных действий ядерных сил; г) агрессия против Российской Федерации с применением обычного оружия, когда под угрозу поставлено само существование государства.