Posts
Comments
[This was originally posted as a response in the wrong thread - I've deleted the incorrectly placed response.]
Hi, David,
Thanks for tagging us in this suggestion! We're happy to see people talking about the creation of more compelling resources to correct misperceptions and get people thinking about the true cost of saving a life.
This doesn't seem exactly like what you have in mind, as it was more narrowly focused on GiveWell's recommended charities, but in the past we provided an impact calculator on our site. It allowed users to insert a donation amount and choose a GiveWell top charity to give to, and would return the number of outputs (e.g., nets or vitamin A supplements distributed) and outcomes (e.g., lives saved).
We stopped sharing the impact calculator in November 2021, because we didn't feel confident enough in our ability to produce a useful forward-looking estimate of an individual donation's impact. We now report on the impact of past grants directed by GiveWell (see this spreadsheet, for example, and our 2021 cost per life saved estimates for top charities). We feel that giving the estimated cost per life saved of a past grant to a program serves as a helpful proxy for the impact of a future donation to that same program, even if we can't count on the impact remaining the same.
We've written a bit more about why we focus on backwards-looking impact estimates here and here.
Best,
Miranda Kaplan
GiveWell Communications Associate
Hi, David,
Thanks for tagging us in this suggestion! We're happy to see people talking about the creation of more compelling resources to correct misperceptions and get people thinking about the true cost of saving a life.
This doesn't seem exactly like what you have in mind, as it was more narrowly focused on GiveWell's recommended charities, but in the past we provided an impact calculator on our site. It allowed users to insert a donation amount and choose a GiveWell top charity to give to, and would return the number of outputs (e.g., nets or vitamin A supplements distributed) and outcomes (e.g., lives saved).
We stopped sharing the impact calculator in November 2021, because we didn't feel confident enough in our ability to produce a useful forward-looking estimate of an individual donation's impact. We now report on the impact of past grants directed by GiveWell (see this spreadsheet, for example, and our 2021 cost per life saved estimates for top charities). We feel that giving the estimated cost per life saved of a past grant to a program serves as a helpful proxy for the impact of a future donation to that same program, even if we can't count on the impact remaining the same.
We've written a bit more about why we focus on backwards-looking impact estimates here and here.
Best,
Miranda Kaplan
GiveWell Communications Associate
Hey, David - yes, the section of the post you linked to contains our thinking on possible changes that could result from COM Contest critiques. We are still in the process of figuring out what specific changes we might make to our analyses based on these critiques.
Best, Miranda (GiveWell communications associate)
Hi, Thomas,
We agree that the language you highlight is misleading, and we're sorry that it was included in the ad. It wasn't a part of the copy we provided to advertisers, and we didn't notice it in our review of the ad before it went live.
We're now speaking to the agency that coordinates these ads about the possibility of updating the video, but we think this is unlikely to happen, as the ad has already been live as is for some time and can't easily be reuploaded.
Thank you for bringing this to our attention!
Best,
Miranda (GiveWell Communications Associate)
Hi, Bob,
We're very sorry to hear that you had a bad experience! We take feedback like this seriously and have passed it on to the senior staff in charge of research hiring.
Responding to all applicants in as timely a manner as we'd like has been challenging, in large part due to understaffing on the research team. But, we are working to improve on this. We invite others who may be considering applying for one of our research positions to review our FAQs for more information about the hiring process, including the typical timeline.
Thank you for sharing your concerns!
Best,
Miranda Kaplan, GiveWell Communications Associate
Hi, Tom,
Thanks for your question! There really isn't a tremendous difference between the two in terms of purpose—both are allocated to global health and development opportunities, either among GiveWell's top charities or to other programs that are high-expected-value. The EA Global Health and Development Fund preceded the All Grants Fund, which we launched in August of this year.
Best,
Miranda
Hi, kokotajlod,
Thank you for following our work; we're happy to have been a part of your journey! Apologies for the delay in responding here.
I'm not sure if by "your recommendations," you mean GiveWell's top charities, or our advice in the above post on giving to the All Grants Fund. It'd be pretty challenging to answer your question if it's about the All Grants Fund, since that money can go to any grant that meets our bar. We expect that a lot of All Grants Fund donations will support grants to top charities, but they could also support other programs, and there could be wide variation in how much of these programs' impact will occur in the next five years.
In our cost-effectiveness analysis for each of our top charities (AMF, Helen Keller's VAS program, Malaria Consortium's SMC program, and New Incentives), we count both deaths averted and future expected earnings from development effects as benefits. Our discount rate of 4% in our cost-effectiveness analyses applies to future expected earnings—if we raised the discount rate to 10%, it would reduce our estimate of these benefits across all four top charities. We don't expect that this would result in changes to our overall top charity recommendations, but it may have a greater impact on our estimated cost-effectiveness of some funding gaps than others, due to variance in the proportion of total benefits coming from development effects.
Steeply discounting future benefits would also make deworming look less cost-effective, since we expect the benefits of deworming to materialize only after treated children become adults.
I hope that's somewhat helpful!
Best,
Miranda
Hi, Peter! It's roughly the former - $600M at 10x cash (or higher) and an additional $300M in opportunities that are 6-9x cash.
Hi, Akash - all else equal, starting sooner is better, but the date is negotiable. We care more about finding someone who is an excellent fit for the role than when they can start, since we hope we’ll be working together over the long term. Thanks!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thanks for your entry!
Thank you for your entry!
Hi, brb243! Would you please submit your contest entry via the form here? You can include a link to this post in the form if you like, or submit a Google doc link or upload a Word doc. Full contest guidelines here. Thanks so much for participating!
Best,
Miranda
Thanks for your entry!
This is Miranda Kaplan, a communications associate at GiveWell. We want to thank everyone who submitted reviews of GiveWell's analyses as contest entries! It's extremely valuable to us when people outside our organization engage deeply and critically with our work. We will always do our best to consider and respond to such critiques, though, given other pressures on staff time, we may not be able to publish a full response.
Although CEA's done it in the post above, we'll take this opportunity to again plug our Change Our Mind Contest, which closes on October 31. Entries submitted to the Criticism and Red Teaming Contest are eligible, as long as they meet all other Change Our Mind Contest requirements (described here). We look forward to reading your work and thank you in advance for your participation!
Thanks for the flag, Jakob!
Thank you for your entry!
This is Adam Salisbury, a senior research associate at GiveWell, responding from GiveWell’s EA Forum account. Thank you for taking the time to engage with our work. Changes to how much we value the health benefits of lead reduction could change how much we could recommend funding to charities working in this area in the future, and the conceptual approach you lay out is both sensible and easy to follow.
I wanted to clarify how we came to the previous adjustment figure, in case it provides helpful context, and then raise a question about your methodology.
Our approach to estimating the health benefits of lead reduction
Our 120% adjustment is a best guess based on i) baseline mortality from lead exposure and ii) the hypothetical value derived from the Pure Earth intervention averting 5%, 15%, or 30% of these deaths. We thought we could get a more accurate estimate if we spent more time looking at the lead mortality/morbidity literature, but we ultimately decided not to do this, as we did not think this parameter would make a difference to this particular grant decision. This was because the economic benefits alone put Pure Earth above our ‘bar’ of cost-effectiveness.
The decision not to dive deeper was broadly reflective of GiveWell’s research approach: in general, the amount of time we spend researching a parameter is roughly proportional to how critical we think that parameter is in influencing our bottom-line estimate/overall decision. We think this helps us to prioritize our time effectively, by keeping us focused on the areas most relevant to our funding decisions.
Question on the approach you recommended
With this said, if we were to consider more lead grants in the future, it is possible that the economic/qualitative arguments would change, which might make us want to dig deeper on the health impacts of lead exposure. For this reason, feedback such as this is really useful to us. On this note, I have a question about your methodology:
I have tried to replicate your back-of the-envelope calculation here, and have a query related to your approach to discounting. In sum: I don’t think it’s necessary to discount the economic benefits calculated on line 17, because discount rates are already ‘baked-in’ to the 1.2% of GDP estimate this figure derives from.
As I understand it, the 1.2% estimate comes from Attina & Trasande (2013), who estimate the economic burden of lead exposure by: i) estimating baseline blood lead levels (BLLs) in LMICs; ii) estimating the relationship between BLLs and IQ; iii) estimating the relationship between IQ and lifetime economic productivity (LEP); iv) applying this to LEP estimates across LMICs, to ‘back out’ lost LEP due to lead exposure. Their LEP estimates are benchmarked against estimates from the US, which “assume annual growth in productivity of 1% and a 3% discount rate” (page 3).
In other words, I think the 1.2% figure comes from the following equation:
This is admittedly not easy to infer from the text, but I think is clearer in the Supplementary Appendix, where they report lost LEP per cohort for each LMIC (see Table S2). In any case, I think that because discount rates enter the numerator of this equation, we don’t want to further discount the 1.2% figure, since then we’d be ‘double-discounting’ future benefits.
If I don’t discount the economic benefits in line 17, then I get an implied adjustment for health benefits of 115% (line 24). This is broadly similar to the 120% adjustment we had before. Again, this could easily change if we decided to dip deeper into this parameter in the future.
I could easily be missing something, so please let me know if you disagree with my interpretation of Attina and Trasande (2013). And thanks again for your post – we really appreciate the feedback.
Hi Michael—this is Isabel Arjmand, Special Projects Officer at GiveWell. Thank you for the feedback and for HLI's critique of deworming, which played a role in inspiring this contest!
We designed this contest to incentivize critiques that are relatively straightforward for us to evaluate and particularly likely to change our mind about upcoming allocation decisions. This is our first time running a contest like this, so we wanted to keep the scope manageable. We may run future contests with different or broader prompts.
A bit more color on why we're keeping the contest's scope to our existing cost-effectiveness analyses:
- We believe critiques of our existing cost-effectiveness analyses will be relatively straightforward for us to review, as opposed to broader critiques, such as those that suggest we take an entirely different approach to recommending giving opportunities. We anticipate that having this well-defined scope will make it easier for us to compare and give due consideration to all entries with our current research capacity (which we are hoping to expand!).
- We're getting ready to make some large decisions about how to allocate funding across the programs we currently support at the end of the year. This contest is designed to solicit the feedback that we think has the greatest potential to improve those upcoming decisions; excellent entries could meaningfully change how we allocate funds, leading to more lives saved or improved. Broader critiques or proposals for wholly new approaches would be unlikely to influence this year’s decisions, given how much vetting we put into our allocations and how little time remains before they are finalized.
Outside of this contest, we welcome feedback on all aspects of our work, and we're glad to receive those at any time via email, as blog comments on our open threads, or here on the EA Forum.
We appreciate your continued engagement on subjective well-being, particularly the useful feedback you provided on our draft reports explaining why we're not as optimistic about subjective well-being measures and Interpersonal Psychotherapy Groups as you are. We're still planning to publish those reports, but we're behind the timeline we originally laid out. Thanks for your patience with this!
[This comment has been edited to remove the name of a specific organization we're looking into.]
This is Miranda Kaplan, a communications associate with GiveWell. Thanks for raising this topic! We do have ongoing investigations for several types of programs related to family planning and are currently prioritizing them among other opportunities.
Hi, Nuno,
To clarify, "Maximum Impact Fund" was our previous name for the fund that we allocated toward the highest-priority gaps within our top charities. The name of that fund is now "Top Charities Fund," and it'll continue to be used to fill top-charity funding gaps. The function hasn't changed, only the name (and now, given our recent update to our criteria for top charities, it will be distributed among a smaller set of programs).
We think the All Grants Fund and Top Charities Fund will likely be of similar cost-effectiveness, since we are using the same cost-effectiveness bar (currently 10x cash) for grants from both funds. But because the All Grants Fund may be used for higher-expected-value opportunities that don't necessarily meet the "confidence" criterion, it's possible that All Grants Fund will be higher-expected-value overall.
I'm not 100% sure I'm interpreting your query correctly, so please let me know if I can clarify anything further!
Best,
Miranda
Thanks, Karthik - good idea, and looks like someone has already done this on our behalf. We'll edit the post to ask people to use the tag.
This is Miranda Kaplan, a communications associate with GiveWell - apologies for the slight delay in responding. Thanks for your engagement with our work! We're looking into this and may post a detailed response here later depending on our capacity.
This is Miranda Kaplan, a communications associate at GiveWell. Thanks so much for publishing this; we really value critiques of our work, especially of our approach to moral weights, which is a particularly challenging feature of our analysis. I've shared this internally so the relevant people can take a thorough read through it, and we may share a response here, as we are able.
This is Miranda Kaplan, a communications associate with GiveWell. Thanks so much for engaging with our cost-effectiveness model, we really appreciate it! This is on our research team’s radar, and they might reach out to you to seek clarification for anything they don’t understand or want to dig into more deeply.
Hi, Andrew,
Yes, the cost-effectiveness estimates we discuss publicly, including the 35x cash (preliminary!) estimate for the maternal syphilis expansion grant, incorporate all "human adjustments" we make to raw expected value, which often appear as “supplemental adjustments” in our cost-effectiveness analyses. These include factors such as likelihood of leverage and funging; charity-level risks, like wastage or funds being diverted for some other purpose; or intervention-level adjustments, like reduction in nonfatal illness or spillover effects. We don’t explicitly model these factors, but incorporate rough best guesses of their effects, which can shift the final cost-effectiveness estimate.
Best,
Miranda
Hi, David,
To clarify our process here, which we haven't detailed except in emails to the donors to whom this applies:
We plan to stop accepting donations to the programs previously on our top charities list by December 31, 2022. In the few months before then, we are contacting donors with open recurring gifts to these programs several times, asking them if they'd like to reallocate their gifts elsewhere. If we don't get a response from these donors by the December 31 deadline, we are automatically cancelling any portions of recurring donations that are allocated to one of the five former top charities (except those set up via PayPal; see below). If I'm interpreting you correctly, you're asking why we're choosing to cancel those donations instead of automatically reallocating them to a different program or fund—is that right?
We made this decision because it was more practical for us administratively, and because we expect that we'll ultimately end up needing to cancel very few donations, if any. Most donors will be able to choose their reallocation rather than this happening by default. When we discontinued the standout charity designation, the vast majority of donors switched their designation to a different program based on their preference; we think that will most likely happen this time as well, so we don't expect to end up missing out on large amounts of funding.
The exception to the process above is recurring donations through PayPal—we can't cancel or change these ourselves, so, after December 31, any funds we get from donations to previous top charities will be automatically reallocated to the Maximum Impact Fund (which seemed like a simple, justifiable default).
I hope that's helpful, and thanks for your engagement!
Miranda
Hi, Andrew,
Apologies for the delay in responding!
It is a bit tricky to compare expected impact across the various funds. The tl;dr answer, without putting any real calculations into it, is that in practice, we don't expect there to be large differences. But theoretically there could be, and if you can give unrestricted or less restricted (assuming you trust GiveWell), that's probably better, as it allows us to deploy funding where it will be most impactful. Here are a few points to consider.
- Because a large portion of our funding is either technically unrestricted or flexible, we think that in practice, it's unlikely that our grantmaking to top charities or non–top charity programs will be constrained by the proportion of funding we receive for the Maximum Impact vs. All Grants fund. A lot of our funding comes from Open Philanthropy as flexible funding intended for grantmaking (so, very similar to funding from the All Grants Fund); this typically has gone to a mix of top charities and other programs. We also receive enough unrestricted funding nowadays that some of it ends up getting granted out, due to the excess assets policy you mention, as well as our single-donor cap, which prevents one donor from providing too much of our operating support (more here). In 2021, the vast majority of our grants from unrestricted funding went to top charities, and indeed the majority of our grant funding in general goes to top charities—as mentioned in the above post, we think that the ratio this year will be about 3:1 (based on the pipeline of opportunities we're looking at right now, not on any rules or proclivities).
- Though we will use the All Grants Fund to support some opportunities that are higher-expected-value than our top charities, we wouldn't predict that the All Grants Fund will be systematically higher in expected value than the Maximum Impact Fund. We are using the same cost-effectiveness bar for grants from both funds, and many of our grants to non–top charity programs are similar in cost-effectiveness to—not greatly more cost-effective than—our top charities.
- Three, with all of the above said, we agree with your suggestion that the most impactful way to give to GiveWell, assuming you trust our decision-making, is unrestricted. There could be a world in which we get way, way more Maximum Impact Fund donations than All Grants Fund donations, and because we're compelled to spend the former on top charities, we end up funding still-excellent-but-less-cost-effective opportunities from those charities (say, 8x cash rather than 10x) and have to raise our bar for granting to other programs to, say, 12x because we're flexible funding–constrained. We don't think that's going to happen because in reality, as noted above, a lot of our funding is flexible. But giving to us unrestricted (or restricted to grantmaking only, through the All Grants Fund) means that we can shift funding around as needed such that we're maximizing the overall impact of our portfolio. The Maximum Impact Fund, however, remains our top recommendation for donors who want to be assured that their donation goes toward high-impact/high-confidence opportunities, versus the riskier options that might be funded via the All Grants Fund.
I hope that's somewhat helpful!
Best,
Miranda
Hi, freedomandutility,
For us, it was less about the expense and more about the administrative burden. It required substantial time and effort to properly establish a new entity—e.g., get approval from the UK's charity commission, set up a UK bank account and donation platforms, and manage ongoing administration, etc. GiveWell has more of the operations staff needed to undertake such a project than we did, say, five years ago, but it was still a long process!
Best, Miranda
Hi, David,
Thank you for your comment! To clarify one point from what you wrote: the critique of our deworming analysis from Happier Lives Institute was not a factor in our decision to update our top charity criteria. We had been planning an update of this kind for about a year before Wednesday's announcement, and only began communicating with HLI about deworming a couple of months ago.
HLI's engagement has led us to begin considering changes to our cost-effectiveness analysis for deworming (and to how we present the decisions behind our models in general). But Wednesday's announcement does not represent a change in our analysis of deworming; it is about a change to our criteria for top charities. We expect to continue to recommend funding for cost-effective gaps we find in deworming—we'll just be recommending it from pots of money other than the Maximum Impact Fund.
I hope that's helpful!
Best, Miranda
Hi, David(s),
Thanks for your feedback! We considered renaming the Maximum Impact Fund in the lead-up to these changes, but decided not to in the end. The Maximum Impact Fund has been a popular giving option that's attracted a lot of new donors; we wanted to err on the side of not confusing these newer donors, who we believe associated this fund with high confidence rather than with high expected value, despite the name. For those like you who follow our work more closely, we expected the risk of confusion would be lower: we thought they'd be more likely to understand the differences in the funds and find it relatively straightforward to make the switch to supporting All Grants versus the Maximum Impact Fund, if their priority was high expected value.
All that said, while this was a considered decision, we recognize that it might not have been the right one. Part of our reason for introducing the All Grants Fund was to help clarify our giving options for donors, and it'd obviously be bad if keeping the Maximum Impact Fund's name were introducing more confusion than clarity. We appreciate your feedback about this, and we'll certainly think about whether this decision merits revisiting!
Best, Miranda
This is Alex Cohen, GiveWell senior researcher, responding from GiveWell's EA Forum account.
Joel, Samuel and Michael — Thank you for the deep engagement on our deworming cost-effectiveness analysis.
We really appreciate you prodding us to think more about how to deal with any decay in benefits in our model, since it has the potential to meaningfully impact our funding recommendations.
We agree with HLI that there is some evidence for benefits of deworming declining over time and that this is an issue we haven’t given enough weight to in our analysis.
We’re extremely grateful to HLI for bringing this to our attention and think it will allow us to make better decisions on recommending funding to deworming going forward.
We would like to encourage more of this type of engagement with our research. We’re planning to announce prizes for criticism of our work in the future. When we do, we plan to give a retroactive prize to HLI.
We’re planning to do additional work to incorporate this feedback into an updated deworming cost-effectiveness estimate. In the meantime, we wanted to share our initial thoughts. At a high level:
- We agree with HLI that there is some evidence for benefits of deworming declining over time and that this is an issue we haven’t given enough weight to in our analysis. We don’t totally agree with HLI on how to incorporate decay in our cost-effectiveness model and think HLI is making a mistake that leads it to overstate the decline in cost-effectiveness from incorporating decay. However, we still guess incorporating decay more in our model could meaningfully change our estimated cost-effectiveness of deworming. We plan to conduct additional research and publish updated estimates soon.
- Once we do this work, our best guess is that we will reduce our estimate of the cost-effectiveness of deworming by 10%-30%. Had we made this change in 2019 when KLPS-4 was released, we would have recommended $2-$8m less in grants to deworming (out of $55m total) since 2019.
- We also agree that we should do more to improve the transparency of our cost-effectiveness estimates. We plan to make key assumptions and judgment calls underlying our deworming cost-effectiveness estimate clearer on our website.
Should we adjust the effect of deworming down to account for decay in benefits?
We agree with HLI that there is some evidence in the Kenya Life Panel Survey (KLPS) data for benefits declining over time. We haven't explicitly made an adjustment in the cost-effectiveness analysis for the possibility that the effects decline over time.
Where I think we disagree is on how to incorporate that potential decay in benefits in our model. While incorporating decay into our model will most likely reduce cost-effectiveness overall, my guess is that HLI’s approach overstates the decline in cost-effectiveness from incorporating decay for three reasons. We plan to explore these further in our updated cost-effectiveness estimate for deworming, but I’ll summarize them quickly here.
First, if we were to model decay based on the KLPS data, we would likely use a higher starting point. We think HLI may have made an error in interpreting the data here.
- To estimate decay, HLI’s model begins with GiveWell’s current estimate of the benefits of deworming — which is based on the average of results from the 10-, 15-, and 20-year follow-ups (KLPS 2, KLPS 3, and KLPS 4, respectively) — then assumes effects decline from that value. However, if we believe that there are declining effects from KLPS 2 to KLPS 3 to KLPS 4, this should imply above average effects in initial years (as found in KLPS 2 and KLPS 3) and then below average effects in later years (as found in KLPS 4).
- Specifically, in its decay model, HLI uses a starting value of 0.006 units of ln(consumption). This is our current estimate of “Benefit of one year’s income (discounted back because of delay between deworming and working for income.” This is based on averaging across results from KLPS 2, KLPS 3, and KLPS 4. If we were to incorporate this decay model, we would most likely choose a value above 0.006 for initial years that declines below 0.006 in later years. For example, if we used the value effect size from KLPS 2 and applied the same adjustments we currently do, this value would be 0.012 in the first year of impacts. We expect this to substantially increase cost-effectiveness, compared to HLI’s model.
Second, we guess that we would not put full weight on the decay model, since it seems like there’s a decent chance the observed decline is due to chance or lack of robustness to different specifications of the effects over time.
-
The three rounds of KLPS data provide three estimates for the effect of deworming at 10-, 15- and 20-year follow-up, which seem to show a decline in effect (based on percentage increase in income and consumption) over time. We could interpret this as either a decline in effect over time, as recommended by HLI, or as three noisy estimates of a constant effect over time, which is the interpretation in our current cost-effectiveness analysis. We’re unsure how much credence to put on each of these interpretations going forward, but it’s unlikely that we will decide to put all of our credence on “decline in effect over time.”
-
When we look at evidence like this, we typically favor pooled results when there is no a priori reason to believe effects differ over time, across geography, etc. (e.g., a meta-analysis of RCTs for a malaria prevention program) because this increases the precision and robustness of the effect measurement. In cases where there’s more reason to believe the effects vary across time or geography, we’re more likely to focus on “sub-group” results, rather than pooled effects. We acknowledge this is often a subjective assessment.
-
In the deworming case, there are some reasons to put weight on the decay story. First, the point estimates we have from KLPS 2, KLPS 3, and KLPS 4, in terms of impact on ln income and consumption, tend toward a decline over time. Second, there are plausible stories for why effects would decline. For example, it’s possible individuals in the control group are catching up to individuals who were dewormed due to broader trends in the economy. This is speculative, however, and we haven’t looked into drivers of changes over time.
-
However, we also think there are reasons to put weight on the “noisy effects” story, which is why our current cost-effectiveness analysis uses a pooled estimate as our best guess of effects over time. First, the evidence for decline comes from three imprecise estimates of income and two imprecise estimates from consumption with overlapping confidence intervals. And comparing effect sizes across rounds and measures is not straightforward – for example, the small sample KLPS 3 consumption results implied at least a doubling of deworming's cost-effectiveness relative to GiveWell's historical model if taken literally (in part due to idiosyncrasies about effects being measured at the per-capita level in a household, rather than merely for a single individual who was dewormed), which is why we funded a larger consumption survey in KLPS-4 and expected to see a much smaller effect in a larger sample.[1] Factors like this give us reason to believe that some of the observed decline in particular measures is due to chance or measurement error. In this case, we would expect the average pooled effect factoring in multiple types of measures to be the best predictor of what we’ll find in future KLPS rounds. Second, it seems plausible that effects would be constant over time (or could even compound over time). For example, adults who were dewormed as children and see greater cognitive or educational gains may be less likely to enter sectors like agriculture, which may have flatter earnings trajectories, or be more likely to move to cities, where opportunities for wage growth are higher. However, these stories are also speculative.
-
As a result, even if we did incorporate the decay model, we would put less than 100% weight on it. We’d like to look further into studies of interventions where the mechanism (improving child development) is more plausibly similar to deworming to see if this provides additional information, as well as any evidence of mechanisms for deworming specifically that would point toward decline in effects. We'd also like to further explore how to interpret the round-by-round estimates, since many factors change between rounds (such as rates of labor force participation and methods of earnings measurement) and we would like to better understand how to predict future changes in control group standards of living when taking all of this into account.
Third, we would likely update our “replicability adjustment” for deworming, based on these results.
-
This is noted in the blog post. HLI notes, though, "The more puzzling concept is the idea that, if you realise you should change one bit of your analysis, you would be justified to arbitrarily alter another, unrelated and non-specified, part of it to ensure you retain a ‘plausible result’."
-
Our approach to the replicability adjustment is to do an informal Bayesian update. We have a prior that is more skeptical of the impact of deworming than seen in the data from KLPS. We incorporate that prior into our estimate of the effect size through the replicability adjustment. The larger the effect size estimated by the KLPS data, the greater the gap between that result and our prior, and the larger the adjustment needed to incorporate our prior.[2] As a result of this adjustment, we input a smaller effect of ln income in our cost-effectiveness analysis than we would if we took the data from KLPS at face value.
-
In our current cost-effectiveness analysis, we adjust the pooled effect of 0.109 in ln income downward by 87%. This reflects our prior belief that we should expect a much lower effect of deworming on later-life income. If we thought the pooled effect was lower than 0.109, we would likely apply a less strict adjustment. We would plan to specify how we update our adjustment in our follow-up work on this.
Our current best guess is that incorporating decay into our cost-effectiveness estimates would reduce the cost-effectiveness of deworming charities by 10%-30%. This adjustment would have led to $2-$8 million less out of $55 million total to deworming since late 2019 (when the most recent deworming study results were released).
We plan to do some additional research to refine our estimates and share an updated cost-effectiveness analysis soon.
Where we’d like to improve on reasoning transparency
We also agree with HLI that we have room for improvement on explaining our cost-effectiveness models. The decision about how to model whether benefits decline is an example of that—the reasoning I outlined above isn't on our website. We only wrote, "the KLPS 4 results are smaller in magnitude (on a percentage increase basis) and higher variance than earlier survey rounds."
We plan to update our website to make it clearer what key judgment calls are driving our cost-effectiveness estimates, why we’ve chosen specific parameters or made key assumptions, and how we’ve prioritized research questions that could potentially change our bottom line.
Encouraging more feedback on our research
We’re extremely grateful to HLI for taking the time to dig into our work and provide feedback. We think this type of engagement improves the quality of our research and our grant recommendations, which helps us allocate resources more cost-effectively, and we’d like to encourage more of it.
In the near future, we plan to announce prizes to individuals and organizations who identify issues in our cost-effectiveness analyses that are likely to lead to meaningful changes in our decisions.
As part of that contest, we also plan to retroactively recommend a prize to HLI (details TBD). We believe HLI’s feedback is likely to change some of our funding recommendations, at least marginally, and perhaps more importantly improve our decision-making across multiple interventions.
See the "internal forecasts" we published, which roughly predict a 25% chance of large consumption results (similar to KLPS-3) that would have doubled our estimate of deworming's cost-effectiveness. (Put differently, we predicted a 75% chance of not updating to the KLPS-3-like magnitude of results after seeing KLPS-4's larger survey.) ↩︎
To see how the math on this works, see this tool, which we used in generating our replicability adjustments. If you input a prior with mean = 0.5 and standard deviation = 0.25, and see evidence with mean = 10 and standard deviation = 1, the posterior effect estimate is ~1.1, for a "replicability adjustment" relative to the evidence of 1.1/10 = ~11%. However, if the evidence shows a smaller effect closer to the prior (mean = 5, sd = 1), the estimated posterior effect is ~0.8, with a replicability adjustment of 0.8/5 = ~16%. So, the overall estimated posterior effect falls when the evidence shows a lower effect estimate (from ~1.1 to ~0.8), but the skeptical Bayesian "replicability adjustment" is slightly less extreme in the second case (an 84% discount instead of an 89% discount). This is what we mean when we say that the replicability adjustment must be updated in conjunction with the estimated effect size, and this is what we have done historically. ↩︎
Thanks, Brennan - we may also post our open listings again in future "Who's hiring?" threads, and presumably there will be another one started up in October.
Hi, Joel, Sam, and Michael -
We really appreciate this type of thoughtful engagement. We find a lot of value in hearing well-reasoned critiques of our research and weaknesses in our communication: thank you for sharing this!
Facilitating feedback like this is a big part of why we hold transparency as one of our values. You're right that it shouldn't be as difficult as it is to understand why we made the decisions we did in our model and how our deworming estimates rely on priors and evidence. Our deworming cost-effectiveness analysis (and frankly other models as well) falls short of our transparency goals–this is a known shortfall in how we communicate about our research, and we are working on improving this.
We want to take the time to more deeply consider the points raised in this post, and do plan on sharing more thinking about our approach. Thanks again for your critical engagement with our research.
Nathaniel/Imma, we agree with Alexander that giving in December 2022 would not be significantly less impactful than giving now. We think the cost-effectiveness of opportunities we'll support in 2023 will probably be similar to those in 2022. If you expect to give in only one of those years, we'd suggest giving in 2022, so that donation can start having an impact earlier, but this decision would be largely up to your individual giving plans.
Thanks, Guy; we appreciate your positive feedback!