Cause-specific Effectiveness Prize (Project Plan)

post by GidonKadosh · 2020-05-02T16:25:30.445Z · EA · GW · 5 comments

Contents

  Context
  Definition
  Motivation
  Method
  Open Questions
  Possible Risks
  Plan
    Steps
    Steps
None
5 comments

Context

This post is a project plan for a Cause-specific Effectiveness Prize, intended to be launched locally in Israel, as a part of EA Israel’s activities.

The project is currently in the stage of market research, and we wanted to post this plan in order to receive your feedback before we continue.

We're continously updating this plan in this document.

Definition

This project’s aim is to establish a competition/prize for cost-effectiveness, between charities that work on the same cause.

Motivation

Our assumptions for potential value in this project:


If successful, this project could be launched:

Method

The general outline of this project consists of three main steps; Gathering resources for the project (especially funding), publishing the contest, and evaluating results. To simplify the evaluation process, we will pick a couple of different cause areas for the prize.

Below is a scale for the actions that can be undertaken by prize candidates. These actions are ordered by the amount of effort required by the charity.

The prize requirements should be adjusted so charities can undertake only level 1 actions, and still have a chance of winning (in order to minimize the effort required by charities for competing). Nevertheless, we should provide guidance (at least through online material) for all levels.

Open Questions

Possible Risks

Plan

Immediate Steps

Additional Steps

  1. Funding - starting with our current funding leads.
  2. Writing down precise definitions and requirements for the prize.
    • Definitions (and resolving the open questions).
    • Legal consulting.
    • Creating online materials.
  3. Advertisement and reaching out to charities.
  4. Follow-up on competing charities.
  5. Evaluations of charity filings.
  6. Optional: Final event for announcing the prize winner. Only if we can be certain enough about holding a big and fancy event. The main value of this is EA Israel’s status within Israeli charities, for other projects and future partnerships.
  7. Retrospective analysis of the project (conclusions for another round of this project in Israel, publicizing results in EA community, organizing project’s resources for future replication).

5 comments

Comments sorted by top scores.

comment by Ramiro · 2020-05-05T19:54:33.151Z · EA(p) · GW(p)

Thanks fo the post, I really enjoyed this idea; please, let us know about its progress. I'd like to see something analogous for academic research and published papers.

As for your open questions:

Personal counseling, or online comprehensive guides?

I guess comprehensive guidelines are better, but they’re not exclusive alternatives, right?

RCTs are the golden standard of measuring impact, but are expensive and complicated. What easier alternatives do we allow? What’s the level of evidence and reliability we accept?

Maybe you should let your contestants and committee have this discussion; of course you can't RCT everything - e.g., no one would do it for parachutes. Actually, it might even be an easier problem than finding a common metric to compare different interventions - e.g., take a look at GiveWell's blog.

Evaluation process: Are charities required to file documents, do we pay for third-party analysts? Should we have a committee? How transparent can/should we be about the results?

Yes, yes, a lot. Suggestion: evaluate contestants in a two, perhaps three-level process:

a) approval voting in an online platform: you can either have an open website where people can vote for free (after identifying themselves), or you can charge voters a fee (two pros: voters will have some skin in the game, and partially fund your costs);

b) individual reviewers rate / select the n most voted contestants;

c) your committee decides the winners.

Risks: I guess reputational risk might be your real issue. Some sort of worst-case scenario: you lose your money and resources to this awful winner... but the real problem is that it goes viral, and everyone starts associating "Effective Altruism" with something like "Canadian Satanists chanting religious poetry in rich schools". Even my illustrative example sucks. Is there any way of letting EA get the benefits of the exposure, but not its risks?

BTW, do you plan to do it just once, or do you intend people to expect it to become a periodic contest? Signalling it could be repeated could influence future projects (in case you end up being successful)

comment by meerpirat · 2020-05-05T08:04:05.805Z · EA(p) · GW(p)

Hey guys! I think it's a cool idea, and I think it's great form to share such a concise summary for feedback. Just some uncertain comments off the cuff, because nobody started the conversation yet:

1) It seems like a resource intense project, e.g. working out reasonable metrics, evaluating them, reaching out, get enough funding. I'd be worried that a bad execution might lead to bad press. For example, somewhere in the back of my mind I remember a discussion of charity representatives in Germany that were very dismissive of the idea that their impact could be measured.

2) "Contest" vs. "Prize": Maybe their is less risk at bad press when it is framed as a prize. Just a feeling that nobody ever forces you to compete for a prize, but it's sometimes mandatory to take part in a contest.

3) Maybe you could try to connect interested researchers with charities and let them work out a way to measure their impact. Then prizes go out to the best reports/papers (the money should probably go to the charity, to incentivize them). I think there is already an existing research field around impact measurement, so you could worry less about counseling the charities and let the researchers work this out with them.

comment by Aaron Gertler (aarongertler) · 2020-05-12T09:27:10.536Z · EA(p) · GW(p)

What are examples of times that it might be "mandatory to take part in a contest"? I'm curious what you're thinking of here, and whether my inability to think of examples might be based on a language barrier or something like that.

comment by meerpirat · 2020-05-12T15:48:37.308Z · EA(p) · GW(p)

I think I should have emphasized that this was just me non-native speakers gut reaction. Maybe a competition is more what I meant? I think about competitions in school where you had no choice but to take part (e.g. maths or reading or sports), or the competition from the movie Hunger Games. Somehow a prize sounds more like something I get for a bonus, while competing against others is to me more associated with "There is not enough for everyone, this is zero-sum, who is this evil person that would make charities compete against each other for their financial survival?".

comment by Aaron Gertler (aarongertler) · 2020-05-12T09:37:06.256Z · EA(p) · GW(p)

Here's where I imagine most of the value coming from for a project like this:

  1. EA Israel members learn about impact evaluation from working with charities.
  2. Other national groups learn from your project and figure out whether they should implement something similar themselves (including in countries like India, where I'd guess that more local charities have a reasonable chance of being high-impact choices on the level of a GiveWell-recommended charity).

Both of these could be very valuable, and I'm curious to see how the project goes!

A few caveats, though:

  1. I would guess that you won't learn about any charities that are really exciting in the sense of being competitive with GiveWell's recommendations. 
  2. If you get local publicity, you might be able to shift some donations toward higher-impact options within Israel, but I'd expect the overall effect to be pretty small. 
    1. Most people don't care that much about charity effectiveness and local giving at the same time, and I'm not sure how much impact differs between great and average Israeli charities (compared to the ten- or hundredfold increase in impact you can get by shifting people toward GiveWell-recommended orgs).

 

When you say that holding a nice event for the winner could help EA Israel's "status within Israeli charities, for other projects and future partnerships," are there any specific projects/partnerships you're thinking of? I can't immediately think of any examples of EA groups interfacing with non-EA-aligned charities for productive projects (though this doesn't mean it's never happened, or that it couldn't happen).