How frequently do ACE and Open Phil agree about animal charities?

post by Ben_West · 2019-12-17T23:56:09.987Z · score: 62 (28 votes) · EA · GW · 3 comments

Contents

    Should ACE and Open Phil agree?
  Raw data
    Data notes
  Animal Welfare Fund
    Credits
None
3 comments

I'm curious what the concordance between Animal Charity Evaluators (ACE) and The Open Philanthropy Project (Open Phil) is. Specifically: of the charities that ACE recommends, what fraction get grants from Open Phil? And conversely, of the charities that ACE considers but does not recommend, what fraction of those get grants from Open Phil?

It seems like there is pretty high concordance:

  1. All of the charities which have ever been recommended as a "Top Charity” have received a grant from Open Phil.
  2. About half of the charities ever described as a "Standout Charity" have received a grant from Open Phil. (Three of the four "general interest standout" charities have received a grant, although ACE only started separating general and special interest recently.)
  3. None of the charities ACE comprehensively reviewed but did not recommend have received a grant from Open Phil. [Edit: there actually is one discordant charity – see Kieran's comment below.]
  4. 5% of the charities ACE did an "exploratory" review of received a grant, as did 3% of the ones they "considered" but did not review.

Note the y-axis is logarithmic. This particularly affects the “Standout” column, which is about half concordant and half discordant. For standout and Top Charities, “concordance” means they got grants from Open Phil; for other recommendation types, “concordance” means that they did not get grants.

The biggest source of discordance seems to be with standout charities. This could be because some of their early standout charities were based on evidence that Open Phil disagreed with (possibly true of Vegan Outreach, maybe FARM) or because the criteria they use are different from Open Phil’s (maybe true of Faunalytics?).

Open Phil started their farmed animal welfare focus area about two years after ACE published their first recommendations, and Open Phil pretty quickly granted to all of ACE's Top Charities after starting. Since then, it seems about 50-50 whether Open Phil grants to something before or after ACE recommends it. So neither organization seems to be simply copying the other.

Should ACE and Open Phil agree?

My guess is that the criteria that ACE uses for their Top Charities is similar enough to what Open Phil looks at that all top charities “should” get grants.

I’m less clear about standout charities. For example, if a charity is highly effective but has limited room for more funding, ACE may choose it as a Standout and also Open Phil may choose not to grant to it. A more careful analysis of this would look at the reasons why charities were or were not selected for recommendations, and see how they line up with Open Phil’s criteria.

Raw data

Copy of the spreadsheet is here.

Data notes

  1. If an organization had multiple recommendation/review status levels at different times in history, I chose the “most recommended” level, using this ranking of recommendations from most to least: Top Charity, Standout Charity, comprehensive, exploratory, considered.
  2. I tried to match up organizations which have different names in ACE and Open Phil’s data set, but I probably missed some.
  3. I wrote this before ACE’s 2019 recommendations were published, but I think the results will be similar: the one new Top Charity (Anima International) received a grant from Open Phil, as did the new Standout Charity (FIAPO).

Animal Welfare Fund

Three charities which were named “Standout Charities” by ACE but did not receive Open Phil grants did receive grants from the Centre for Effective Altruism’s Animal Welfare Fund (Animal Ethics, Faunalytics, and Sociedade Vegetariana Brasileira). As Lewis Bollard is both the Chair of the Animal Welfare Fund and the Program Officer for Open Phil’s Farmed Animal Welfare focus area, it could be interesting to consider these as additional sources of concordance.

For example, maybe Lewis thought a given charity was high-quality, but that it would be better funded via the Fund instead of through Open Phil. None of the payout reports explicitly stated this (and the other fund managers could have been the ones who recommended these charities, instead of Lewis), but it seems reasonable to assume that Lewis at least did not think they were terrible grants.

Here is what concordance looks like if you include these three charities as concordant:

Credits

I would like to give a shout out to ACE for publishing such an easy to use list of their recommendations, as well as to Open Phil for providing a download of their grants. This post was inspired by conversations with JP Addison. Aaron Gertler and Gina Stuessy provided helpful comments.

3 comments

Comments sorted by top scores.

comment by kierangreig · 2019-12-18T16:26:39.028Z · score: 19 (7 votes) · EA(p) · GW(p)

[Fwiw: I previously worked at ACE and now work at Farmed Animal Funders. I'm also on the committee for EA Animal Welfare Fund]

Thanks for writing this, Ben! It is an interesting analysis. Here are some thoughts that I had while reading:

  • What could also be interesting would be looking at the concordance/ disconcordance between some of the regranting funds in this area (e.g., EAA fund, EA Animal Welfare Fund, and OWA.)
  • What could also be interesting for Open Phil and ACE is comparing and contrasting the funding allocations across the concordant groups.
  • Some of the discordance between Open Phil and ACE might also be explained by different views about the potential of cultivated meat, different overall capacities for funding, and different approaches on restricting funding.

Lastly, two quick notes:

Three charities which were named “Standout Charities” by ACE but did not receive Open Phil grants did receive grants from the Centre for Effective Altruism’s Animal Welfare Fund (Animal Ethics, Faunalytics, and Compassion in World Farming - USA).

I think Compassion in World Farming - USA has received three Open Phil grants.


None of the charities ACE comprehensively reviewed but did not recommend have received a grant from Open Phil.

I think Compassion in World Farming International has been comprehensively reviewed by ACE and then not recommended but have received several Open Phil grants.



comment by Ben_West · 2019-12-18T18:02:43.606Z · score: 3 (2 votes) · EA(p) · GW(p)

Thanks Kieran!

I think Compassion in World Farming - USA has received three Open Phil grants.


Good catch, SVB was the one who did not receive an OP grant (AFAICT) but did get one from the Fund. The text has been updated.

I think Compassion in World Farming International has been comprehensively reviewed by ACE and then not recommended but have received several Open Phil grants.


Interesting, thanks! You're right – I will look more into what happened here. It could be that my script got confused about the difference between the domestic and international version of CIWF.

comment by Henry_Stanley · 2019-12-18T16:36:05.205Z · score: 13 (3 votes) · EA(p) · GW(p)

This is a really neat analysis. Thanks for doing it!

Do you think there's some feedback in the system in which ACE/OpenPhil use the other's research to help make their decisions?