What questions would you like to see forecasts on from the Metaculus community?

post by alexrjl · 2020-07-26T14:40:17.356Z · score: 25 (11 votes) · EA · GW · No comments

This is a question post.


    8 jacobpfau
    6 NunoSempere
    5 smclare
    3 Denkenberger
    2 NunoSempere
    1 D0TheMath
No comments

Metaculus is an online platform where users make and comment on forecasts, which has recently been particularly notable for its forecasting of various aspects of the pandemic, on a dedicated subdomain. As well as displaying summary statistics of the community prediction, Metaculus also uses a custom algorithm to produce an aggregated "Metaculus prediction". More information on forecasting can be found in this interview with Philip Tetlock on the 80,000 hours podcast.

Questions on Metaculus are submitted by users, and a thread exists on the platform where people can suggest questions they'd like to see but do not have the time/skill/inclination to construct themselves. Question construction is non-trivial, not least because for forecasting to work, clear criteria need to be set for what counts as positive resolution. A useful intuition pump here is "if two people made a bet on the outcome of this question, would everyone agree who had won?"

Although there is already a significant overlap between the EA community and the Metaculus userbase, I think it is likely that there exist many forecasting questions which would be very useful from an EA perspective, but that have not yet been written. As such, I've written this question as both a request and an offer.

The request:

Have a think about whether there are any forecasts you think have the potential to have a large impact on decision making within the EA community.

The offer:

If you do think of one, and post it below, and I'll write it up for you and submit it to the site. The closer it is to "fully formed", the more quickly this is likely to happen, but please don't feel the need to spend ages choosing resolution criteria, I'm happy to help with this. I intend to choose questions based on some combination of number of upvotes the suggestion has and how easy the question is to operationalise.

Examples of my question-writing on Metaculus are here, and I also recently become a moderator on the platform.

Some examples of EA-adjacent questions already on the platform:

How much will GiveWell guess it will cost to get an outcome as good as saving a life, at the end of 2021?

On December 1st, 2023 how many companies worldwide will pledge uphold GAP standards for broiler chickens raised for meat?

How many reviews will Toby Ord's book The Precipice have on Amazon on January 1st 2021?

How many infections of SARS-CoV-2 (novel coronavirus) will be estimated to have occurred worldwide, before 2021?

If you're interested in having someone make a forecast about a question that's more personal to you, and/or something that you wouldn't expect the Metaculus community as a whole to have the right combination of interest in and knowledge of, I'd recommend checking out this offer from amandango [EA · GW].


answer by jacobpfau · 2020-07-26T18:25:37.651Z · score: 8 (7 votes) · EA(p) · GW(p)

Thanks for doing this, great idea! I think Metaculus could provide some valuable insight into how society's/EA's/philosophy's values might drift or converge over the coming decades.

For instance, I'm curious about where population ethics will be in 10-25 years. Something like, 'In 2030 will the consensus within effective altruism be that "Total utilitarianism is closer to describing our best moral theories than average utilitarianism and person affecting views"?'

Having your insight on how to operationalize this would be useful, since I'm not very happy with my ideas: 1. Polling FHI and GW 2. A future PhilPapers Survey if there is one 3. Some sort of citation count/ number of papers on total/average/person utilitarianism. It would probably also be useful to get the opinion of a population ethicist.

Stepping back from that specific question, I think Metaculus could play a sort of sanity-checking, outside-view role for EA. Questions like 'Will EA see AI risk (climate change/bio-risk/etc.) as less pressing in 2030 than they do now?', or 'Will EA in 2030 believe that EA should've invested more and donated less over the 2020s?'

comment by alexrjl · 2020-07-27T06:01:58.660Z · score: 2 (2 votes) · EA(p) · GW(p)

The best operationalisation here I can see is asking that we are able to attach a few questions if this form to the 2030 EA survey, then asking users to predict what the results will be. If we can get some sort of pre-commitment from whoever runs the survey to include the answers, even better.

One thing to think about (and maybe for people to weigh in on here) is that as you get further out in time there's less and less evidence that forecasting performs well. It's worth considering a 2025 date for these sorts of questions too for that reason.

comment by Charlotte (CharlotteSiegmann) · 2020-07-28T14:14:07.712Z · score: 2 (2 votes) · EA(p) · GW(p)

Another operationalisation would be to ask to what extent the 80k top career recommendations have changed, e.g. what percentage of the current top recommendations wills till be in the top recommendations in 10 years.

comment by alexrjl · 2020-07-31T13:12:54.609Z · score: 3 (2 votes) · EA(p) · GW(p)

This question is now open for feedback, and will go live in a week's time.

How many of the "priority paths" identified by 80,000hours will still be priority paths in 2030?

comment by alexrjl · 2020-07-28T21:19:09.748Z · score: 2 (2 votes) · EA(p) · GW(p)

I really like this and will work something out to this effect

comment by alexrjl · 2020-07-28T13:16:12.941Z · score: 1 (1 votes) · EA(p) · GW(p)

Do you want to have a look at the 2019 EA survey and pick a few things it would be most useful to get predictions on? I'll then write a few up.

comment by jacobpfau · 2020-07-29T01:41:41.154Z · score: 1 (1 votes) · EA(p) · GW(p)

I think the 'Diets of EAs' question could be a decent proxy for the prominence of animal welfare within EA. I think there are similar questions on metaculus for the general US population https://www.metaculus.com/questions/?order_by=-activity&search=vegetarian

I don't see the ethics question as all that useful, since I think most of population ethics presupposes some form of consequentialism.

comment by alexrjl · 2020-07-31T12:49:24.785Z · score: 2 (2 votes) · EA(p) · GW(p)

It looks like a different part of the survey asked about cause prioritisation directly, which seems like it could be closer to what you wanted, my current plan (5 questions) for how to use the survey is here.

comment by jacobpfau · 2020-07-26T18:26:46.335Z · score: 1 (1 votes) · EA(p) · GW(p)

Somewhat unrelated, but I'll leave this thought here anyway: Maybe EA metaculus users could perhaps benefit from posting question drafts as short-form posts on the EA forum.

comment by alexrjl · 2020-07-26T22:31:10.618Z · score: 3 (3 votes) · EA(p) · GW(p)

I'm kind of hoping that this thread ends up serving that purpose. There's also a thread on metaculus where people can post ideas, the difference there is nobody's promising to write them up, and they aren't necessarily EA ideas, but I thought it was worth mentioning.

(I do have some thoughts on the top level answer here, but don't have time to write them now, will do soon)

answer by NunoSempere · 2020-08-06T20:27:44.951Z · score: 6 (2 votes) · EA(p) · GW(p)

A while ago, Leah Edgerton, of Animal Charity Evaluators, gave an AMA, and one of the questions I asked was What are some questions regarding EAA (effective animal advocacy) which are amenable to being forecasted?.

Her answer is in this video here. In short:

  • Will corporations stick to their animal welfare commitments?
  • When will specific animal free food technologies become cost-competitive with their traditional animal counterparts?
  • Timelines for cultured meat coming to market?
  • When will technology exist which allows the identification of the sex of a chicken before it hatches? When, if ever, will such a technology be adopted
  • When, if ever, will the global production and consumption of farmed animals stop growing? When will stop completely?
  • When will specific countries or states adopt legal protection for animals / farmed animals?
  • When will EAA organizations have a budget of more than $500 million? $1 billion?
  • Questions related to the pandemic.
  • Questions related to the budget of EAA organizations in the immediate future.

Operationalizing these questions, and finding out what the most useful things to forecast are may involve contacting ACE directly. For example, "corporations" is pretty general, so I imagine ACE has some particular ones in mind.

answer by smclare · 2020-07-31T12:17:45.987Z · score: 5 (4 votes) · EA(p) · GW(p)

Some fun, useful questions with shorter time horizons could be stuff like:

  • Will GiveWell add a new Top Charity to its list in 2020 (i.e. a Top Charity they haven't previously recommended)?
  • How much money will the EA Funds grant in 2020? (total or broken down by Fund)
  • How many new charities will Charity Entrepreneurship launch in 2020?
  • How many members will Giving What We Can have at the end of 2020?
  • How many articles in [The Economist/The New York Times/...?] will include the phrase "effective altruism" in 2020?

Stuff on global development and global poverty could also be useful. I don't know if we have data to resolve them, but questions like:

  • What will the global poverty rate be in 2021, as reported by the World Bank?
  • How many malaria deaths will there be in 2021?
  • How many countries will grow their GDP by more than 5% in 2021?
comment by alexrjl · 2020-08-03T08:24:57.903Z · score: 1 (1 votes) · EA(p) · GW(p)

Here are the three upcoming questions I've written in response to the ideas on this thread:

Who will win the 'worm wars'?

When will Charity Entrepreneurship help create a new GiveWell top charity?

How many charities will Charity Entrepreneurship help launch in 2021?

answer by Denkenberger · 2020-07-30T21:54:30.718Z · score: 3 (2 votes) · EA(p) · GW(p)

For EAs who are investing to give, more questions about the market would be great, e.g. my comment here [EA · GW].

comment by alexrjl · 2020-07-30T22:06:16.624Z · score: 3 (2 votes) · EA(p) · GW(p)

Metaculus did briefly experiment with a finance spinoff but I don't believe it was successful. I can definitely write a couple and I think they'd get a lot of interest, but I'd be surprised if making investment decisions based on metaculus was a winning strategy in the long-term. I'd be more optimistic, though still cautious, about political betting using metaculus predictions.

Here are some current questions which seem relevant, you can find more here.





comment by Denkenberger · 2020-07-31T05:09:05.350Z · score: 2 (1 votes) · EA(p) · GW(p)

Thanks - very helpful.

answer by NunoSempere · 2020-08-06T20:33:02.165Z · score: 2 (2 votes) · EA(p) · GW(p)

I'm also interested in questions around approval voting in general, and the Center for Election Science in particular.

Some stuff:

  • Conditional on less than 5 cities with >=50,000 people having implemented approval voting by Dec 31, 2022, what will the funding for the Center for Election Science be during 2023? Context: According to the CES's strategic plan converting 5 cities with >= 50,000 inhabitants is one of their main targets by 2022 (see p. 7). Conditional on them not achieving it, how will their funding look like? This can probably be operationalized with reference to IRS tax reports.
  • How many US cities with more than 50,000 people will have implemented approval voting by [date]?
  • What will CES funding look like in 2021, 2022, etc.
answer by D0TheMath · 2020-07-26T18:27:06.536Z · score: 1 (1 votes) · EA(p) · GW(p)

In The Precipice Toby Ord gives estimated chances of various existential risks happening within the next 100 years. It'd be cool if we could get estimates from Metaculus as well, although it may be impossible to implement, as Tachyons would only be awarded when the world doesn't blow up.

comment by niplav · 2020-07-26T20:42:37.470Z · score: 3 (3 votes) · EA(p) · GW(p)

Well, there's the Ragnarök question series, which seems to fit what you're looking for.

No comments

Comments sorted by top scores.