Cost-effectiveness analysis of a program promoting a vegan diet

post by nadavb, sella, GidonKadosh, MorHanany · 2020-11-12T17:24:28.721Z · EA · GW · 6 comments

Contents

  Background
  Why did we write this report?
    your convenience, we copy-pasted the executive summary of our report and present it at the remainder of this post.
    would appreciate any type of feedback!
  Introduction
  Cost-effectiveness estimates
  Limitations and uncertainties
  Conclusion
None
6 comments

Background

As part of the activities of EA Israel [EA · GW], we recently completed our first cost-effectiveness analysis of a program run by an Israeli charity. We analyzed Challenge 22, a program promoting a vegan diet that is carried out by Animals Now. The full report is available here.

 

Why did we write this report?

Here are some of our motivations:

  1. While we were uncertain about the competitiveness of this program compared to the world’s most effective charities helping to reduce animal suffering, we thought it was worth exploring. As a local EA group in Israel, we expected to have a comparative advantage in assessing the cost-effectiveness of the program (by avoiding the need to bridge over cultural differences).
  2. We hope this report will help Israeli donors who are interested in donating to effective Israeli charities to give more effectively. While we encourage donors to consider giving to global effective charities (such as those recommended by GiveWell or ACE), many Israelis feel strongly about giving locally. We believe that in some cases it is wiser to just give such donors what they want, helping them to find effective local charities rather than preaching to them about the merits of impartiality. In the long run we hope they will eventually come to absorb more of EA’s core values and someday consider giving to the world’s most effective charities, regardless of nationality.
  3. We anticipate that many of the Israeli donors who prefer giving locally also prefer cause areas that help people (rather than animals). We hope this report will strengthen our reputation among Israeli charities and philanthropists, allowing us to form more collaborations and write similar reports for charities in other cause areas as well.
  4. Our long-term goal is to promote standards of cost-effectiveness and impact evaluation in Israeli and Jewish philanthropy, a vision we promote through other projects as well (see for example our Cause-specific Effectiveness Prize [EA · GW]). The reasons we think this is a worthwhile goal to pursue are:
    • There are billions of dollars going to Israeli NGOs every year. Having this huge sum of money allocated even slightly better can have a very large direct impact.
    • If we want to promote the ideas of EA and make them more mainstream, we need to acknowledge how strange they sound to the average person. The idea that one should consider giving their money to distribute bed nets in Africa sounds very alien to the average Israeli. On the other hand, making the claim that one should consider giving to Israeli charities that are transparent and have strong evidence for high impact is much more intuitive to people. We think it is common-sense enough to have a real chance to take root and become mainstream in the coming years. As mentioned in point (2), we see the promotion of cost-effectiveness standards as a stepping stone on the way to accepting EA values more broadly.

 

 

For your convenience, we copy-pasted the executive summary of our report and present it at the remainder of this post.

We would appreciate any type of feedback!

 

Introduction

This report analyzes Challenge 22, an international program carried out by the Israeli charity Animals Now. The program promotes veganism and reduction in meat consumption by encouraging individuals to take a pledge to change their diet for 22 days. Throughout the challenge and afterwards, participants have access to social support through Facebook groups aimed at helping them to follow through with the pledge, and to sustain their dietary changes in the long run.

 

Cost-effectiveness estimates

The primary evidence we rely on for evaluating the cost-effectiveness of the program is a survey study published in 2019 (following participants who had signed up to the program during 2018). By integrating the results of this study with estimates on the scale and budget of the program (provided by Animals Now), we estimate that every 1 ILS of funding going into the program corresponds to a reduction in meat consumption of somewhere between 1 to 12 portions of meat for an average participant in the program over the course of the first 7 months after taking the pledge. In US dollars, we estimate that every $1 corresponds to somewhere between 3 to 40 portions of meat (based on the exchange rate of October 2020 - 1 ILS for 0.29 USD).

While we only have solid evidence for dietary changes of participants following the first 7 months after taking the pledge, there are positive indications that these changes are robust and could persist for substantially longer time periods. Specifically, there is no significant difference between the dietary change observed after 7 months to that observed after 1 month. This suggests that the reduction in meat consumption could actually be substantially larger than the aforementioned estimate. If we assume, for example, that dietary changes following the program last 5 years on average, we would get that every 1 ILS of funding going into the program corresponds to a reduction in meat consumption of 9 to 100 portions of meat. 

 

Limitations and uncertainties

These cost-effectiveness estimates are subject to some important limitations of our analysis, which should be kept in mind when judging the impact of the program:

  1. The numbers above describe the estimated change in meat consumption for participants who have joined the program, not direct causal effects. As the underlying study is observational, we don’t have sufficient evidence to prove a causal link. However, we believe that the results look encouraging, and causal effects seem plausible.
  2. The underlying study surveyed only Israeli participants, which comprise only about 10% of the program. As stated in the report, there are marked differences distinguishing the Israeli program. Our lower-bound estimate incorporates the assumption that the average magnitude of dietary change for non-Israelis is no less than 35% of the change for Israelis. Much of the wide uncertainty range in our final estimate (with a 12-fold factor between the lower to the upper bound) is due to this limitation.
  3. The report assesses the total magnitude of dietary changes per ILS based on data observed in the past. An important aspect of the program is that it is rapidly changing and evolving. While these changes are generally positive (reflecting attempts to scale the program and make it more cost-effective, with some indications of success), it also means that it is hard to extrapolate what the future cost-effectiveness of the program will be. Also, this report does not thoroughly tackle the question of how much room for more funding the program has.
  4. There are some important limitations of the survey study on which our estimates for reduction in meat consumption are based. Most substantial is its reliance on self-reports from participants, which might be prone to social desirability or memory biases. Other, more minor potential limitations are detailed in the report.

On top of these limitations, there are other uncertainties which we have already factored into our analysis, and which have led to the wide uncertainty range in our final estimate. A more exhaustive summary of the limitations of our analysis can be found at the 8th chapter of the report.

Another point that should be kept in mind is that Challenge 22 is only one of several other activities pursued by Animals Now. In terms of budget, it comprises about 20% of the charity. We haven’t analyzed the cost-effectiveness of the other programs.

As a result of all the many limitations of the analysis mentioned above, we encourage the reader not to take the numbers we have come up with too literally. Rather, this cost-effectiveness analysis should be interpreted as a rough estimate pointing at a general order of magnitude for the potential impact of the program.

 

Conclusion

Despite all of the mentioned limitations and uncertainties, in the context of Israeli philanthropy this program is quite exceptional in terms of measurement, transparency standards and evidence for positive impact. 

 

The full report is available here.

6 comments

Comments sorted by top scores.

comment by saulius · 2020-11-17T11:28:09.859Z · EA(p) · GW(p)

Hey, thanks for doing this! I think you did a good job at considering most of the uncertainties. My main disagreement would be that this is a moderate limitation: graph

I think that it is a major limitation. In general, since it seems that most of the work is done by volunteers, the situation reminds me of an example I gave in this article [EA · GW]:

Imagine many volunteers collaborating to do a lot of good, and having a small budget for snacks. Their cost-effectiveness estimate could be very high, but it would be a mistake to expect their impact to double if we double their funding for snacks.

You could imagine that program being run without any paid staff and any expenses, and having an infinite cost-effectiveness. But it wouldn’t follow that this is a good opportunity for donors. If volunteer involvement is the major reason for cost-effectiveness, I don’t see a reason to think why the cost-effectiveness of related activities like expanding advertising and accelerating the development of an app would be at all similar to the cost-effectiveness of the program so far. These seem to be totally different activities.

That said, the cost-effectiveness estimate does inform us that expanding the program into more geographic locations, cultures and languages could be promising. But if this is what you use the cost-effectiveness estimate for, maybe you shouldn’t adjust the estimates of future costs towards the lower present day costs, because in that case the set up costs are relevant. Also, in that case I wouldn’t feature the cost-effectiveness figures so prominently in this analysis if the target audience is Israelis wanting to donate to local charities.

Another thing is that if participants switched to a vegetarian diet and started eating more eggs to get enough protein instead of eating beef or lamb, the program might have caused more suffering than it prevented (see http://ethical.diet/). I imagine that they were encouraged to get their proteins in other ways though, but it is still something to consider.

comment by saulius · 2020-11-17T11:30:54.095Z · EA(p) · GW(p)

Some more thoughts: If someone were to look into the program deeper, maybe it is possible to run an RCT. You could randomly assign some participants who sign up to the program to the control group. You could just send a document to these people explaining how to be vegetarian or something, and not make them join any groups. And then you could send them the same questionnaire to them as to others. This has some flaws but would be better than nothing. But it’s pretty clear that the program should continue so I don’t think it’s worth the effort.

Another mildly useful thing to do would be to check if meat consumption in Israel has gone down in general, and use that as a control group. It could be mildly useful because it would make you dismiss hypotheses that they consumed less meat because of unrelated reasons that apply to all Israelis like increased meat prices, or increased availability of plant-based options, or meat-related health scare.

comment by nadavb · 2020-11-24T08:49:51.589Z · EA(p) · GW(p)

Thank you for your great feedback and suggestions! (and sorry for not responding sooner)

I guess that one’s meaning for a “major” or “moderate” limitation is, in the end, contingent on their aspirations. If we had the standards of an organization like GiveWell, this would most certainly be a very big limitation. But quite early on we understood that we did not have the data to be able to support as strong conclusions about cost-effectiveness as GiveWell’s recommendations. Rather, our approach was: let’s do the best we can with the data we have at hand, and simply make sure that we are very clear and transparent about the limitations of our analysis. The biggest limitation of this analysis is the lack of experimental data (with only observational data available). We wanted to make sure this got the most eye-catching label. In the end we believe that what’s important is that readers of the report (or just of the executive summary) get a good sense of what conclusions are justified given our analysis and which aren’t, and that they understand what the important limitations of the analysis are. We totally agree with your arguments and the fact that past cost-effectiveness is by no means proof of future cost-effectiveness given more funding (though we do think there are reasons for cautious optimism in the case of Animals Now). 

Also, thank you for the interesting suggestion for an RCT study design. This is something we have been considering in general, but haven’t thought of your exact idea. However, to approach anything like that that, we would first need the charity to have a strong motivation to get into that adventure.

comment by MichaelStJules · 2020-11-12T18:52:19.981Z · EA(p) · GW(p)

Looks promising! I was hoping for a breakdown by type of animal product and animals spared, but it looks like the data for this wasn't collected.

We could divide their animal product consumption reduction proportionally the same way as the their local population, or maybe flexitarians/reducetarians specifically, so we assume they reduced their animal product consumption uniformly and had similar proportions as their local populations. This might give us a rough ballpark estimate in terms of animals spared. (Also setting aside issues of supply and demand, international trade, etc..)

comment by nadavb · 2020-11-12T19:28:53.093Z · EA(p) · GW(p)

Thank you!

I agree it would be nicer to report actual spared animals, rather than generic “portions of meat”. We thought of using data about the average meat diet in the relevant countries, to be able to translate portions of meat into animal lives. But we eventually decided against it, because it would introduce even more assumptions and uncertainties into our analysis, which we felt had many uncertainties already. Given the amount of uncertainty that we already have (with over an order-of-magnitude between our lower and upper bounds), we felt that giving a too detailed breakdown might be inappropriate. In the end we decided to keep it simple and use the metric we had data on, hoping that “1 to 12 portions of meat per 1 ILS” would give readers a rough sense of the potential of this program to spare animal lives.