Posts

Against cash benchmarking for global development RCTs 2022-03-21T00:29:05.393Z

Comments

Comment by Rory Fenton on Direct effects of marine plastic pollution on some wild animals seem small · 2022-04-20T15:23:34.445Z · EA · GW

I really appreciated this short, clear post. Thank you!

Comment by Rory Fenton on Awards for the Future Fund’s Project Ideas Competition · 2022-03-24T20:47:51.795Z · EA · GW

Hey!

LEEP is indeed working on this -- I mentioned them in my original comment but I have no connection to them. I was thinking of a campaign on the $100M/year scale, comparable to Bloomberg's work on tobacco. That could definitely be  LEEP,  my sense (from quick Googling and based purely on the small size of their reported team) is they would have to grow a lot to take on that kind of funding, so there could also be a place for a large existing advocacy org pivoting to lead elimination. I have not at all thought through the implementation side of things here. 

Comment by Rory Fenton on Against cash benchmarking for global development RCTs · 2022-03-22T18:51:00.823Z · EA · GW

How does the time and monetary cost of buying these products compare to the time and monetary cost of giving cash?

The total value of the bundle ($120) includes all staffing (modelled at scale with 100k recipients), including procurement staff, shipping, etc. This trial was a part of a very large nonprofit, which has very accurate costs for those kinds of things.

But obviously the researchers didn't know beforehand that the programs would fail. So this isn't an argument against cash benchmarking.

That's true, I don't think I made my point well/clearly with that paragraph. I was trying to say something like, "The Vox article points to how useful the cash comparison study had been, but the usefulness (learning that USAID shouldn't fund the program) wasn't actually due to the cash arm".  That wasn't really an important point and didn't add much to the post. 

Comment by Rory Fenton on Against cash benchmarking for global development RCTs · 2022-03-22T02:22:52.022Z · EA · GW

I really like the idea of asking people what assets they would like. We did do a version of this to determine what products to offer, using qualitative interviews where people ranked ~30 products in order of preference. This caused us to add more chickens and only offer maize inputs to people who already grew maize. But participants had to choose from a narrow list of products (those with RCT evidence that we could procure), I'd love have given them freedom to suggest anything. 

We did also consider letting households determine which products they received within a fixed budget (rather than every household getting the same thing) but the logistics got too difficult. Interestingly, people had zero interest in deworming pills, oral hydration salts or Vitamin A supplements as they not were aware of needing them-- I could see tensions arising between households not valuing these kinds of products and donors wanting to give them based on cost-effectiveness models. This "what do you want" approach might work best with products that recipients already have reasonably accurate mental models of, or that can be easily and accurately explained.

At a very basic intuitive level, hearing "participants indicated strong preference for receiving our assets to receiving twice as much cash" feels more persuasive than comparing some measured outcome between the two groups (at least for this kind of asset transfer program where it seems reasonable to defer to participants about what they need/want)

Very interesting suggestion: we did try something like this but didn't consider it as an outcome measure and so didn't put proper thought/resources into it. We asked people, "How much would you be willing to pay for product X?", with the goal of saying something like "Participants valued our $120 bundle at $200" but unfortunately the question generally caused confusion: participants would think we were asking them to pay for the product they'd received for free and either understandably got upset or just tried lowballing us with their answer, expecting it to be a negotiation. 

If we had thought of it in advance, perhaps this would have worked as a way to generate real value estimates:

  • We randomise participants into groups
  • The first group is offered either our bundle (worth $120) or $120 cash
  • If >50% take the bundle, we then adjust our cash offer upwards and offer it to another group (or the opposite if <50% take the bundle)
  • We repeat this process of adjusting our price offer until we have ~50% of participants accepting our cash offer: that equilibrium price is then the "value" of the bundle
  • I'm not sure what the sample size implications would be but a big advantage would be the timeline: this could be done in a few weeks, not years
  • I bet proper economists have a way to do this but it's interesting to brainstorm on

I can see a few issues with this:

  • We still have to assume people "know what's best for them": that's a really patronising statement, but as above, people need reliable mental models to make these decisions, and won't have that for novel products
  • We need to have donors who will accept this measure: the data only really matters when it informs decisions

I'm very open to other thoughts here, I really like the concept of a cash benchmark and would love to find a way to resurrect the idea.

Comment by Rory Fenton on Against cash benchmarking for global development RCTs · 2022-03-22T01:40:37.562Z · EA · GW

Thanks for the interesting reflections. 

I agree that longer term data collection can help here in principle, if the initial differences in impact timing wash out over the years. One reason we didn't do that was statistical power: we expected our impact to decrease over time, so longer term surveys would require a larger sample to detect this smaller impact. I think we were powered to measure something like a $12/month difference in household consumption. I think I'd still call a program that cost $120 and increased consumption by, say, $3/month 10 years later a  "success", but cutting the detectable effect by 1/4 takes 16x the sample size. Throw in a cash arm, and that's a 32x bigger sample (64,000 households in our case). We could get a decent sense of whether our program had worked vs control over a shorter (smaller sample) timeline, and so we went with that. 

If the concern is about which measure of impact to use - you cite issues with people remembering their spending - then the (I think) obvious response is to measure individuals' subjective wellbeing, eg 0-10 "how satisfied are you with your life nowadays?" which allows them to integrate all the background information of their life when answering the question. 

The subjective wellbeing idea is interesting (and I will read your study, I only skimmed for now but I was impressed). It isn't obvious to me that subjective wellbeing isn't also just a snapshot of a person's welfare and so prone to similar issues to consumption e.g. you might see immediate subjective welfare gains in the cash arm but the program arm won't start feeling better until they harvest their crops. I'm not really familiar with the measure, I might be missing something there.

I agree with you that you don't need a cash arm to prove your alternative didn't work. But, if you already knew in advance your alternative would be worse, then it raises questions as to why you'd do it at all. 

Agreed-- I'm sure they expected their program to work, I just don't think adding a cash arm really helped them determine if it did or not.

Comment by Rory Fenton on The Future Fund’s Project Ideas Competition · 2022-03-07T17:50:26.482Z · EA · GW

Thanks for sharing! 

 

My initial sense is that China's method is focused on controlling rainfall, which might mitigate some of the effects of climate change (e.g. reduce drought in some areas, reduce hurricane strength) but not actually prevent it. The ideas I had in mind were more emergency approaches to actually stopping climate change either by rapidly removing carbon (e.g. algae in oceans) or reducing solar radiation absorbs on the Earth's surface (making clouds/oceans more reflective, space mirrors). 

Comment by Rory Fenton on Announcing the Future Fund · 2022-03-03T23:08:03.509Z · EA · GW

Will all funding applications be made public? If so, is it possible for ask for specific application not to be public?  No problem if actual funding will be publicized, I'm just wondering about the applications themselves. Thanks!

Comment by Rory Fenton on The Future Fund’s Project Ideas Competition · 2022-03-03T18:41:56.398Z · EA · GW

Eliminate all mosquito-borne viruses by permanently immunizing mosquitoes 

Biorisk and Recovery from Catastrophe

Billions of people are at risk from mosquito-borne viruses, including the threat of new viruses emerging. Over a century of large-scale attempts to eradicate mosquitoes as virus vectors has changed little: there could be significant value in demonstrating large-scale, permanent vector control for both general deployment and rapid response to novel viruses. Recent research has shown that infecting mosquitoes with Wolbachia, a bacterium, out-competes viruses (including dengue, yellow fever and Zika), preventing the virus from replicating within the insect, essentially immunizing it. The bacterium passes to future generations by infecting mosquito eggs, allowing a small release of immunized mosquitoes to gradually and permanently immunize an entire population of mosquitoes. We are interested in proposals for taking this technology to massive scale, with a particular focus on rapid deployment in the case of novel mosquito-borne viruses. 

Epistemic status: Wolbachia impact on dengue fever has been demonstrated in a large RCT and about 10 city-level pilots. Impact on other viruses only shown in labs. The approach is likely to protect against novel viruses but that has not been demonstrated. 

Conflict of interest: I work for a small, new nonprofit focused on $B giving. I have had conversations with potential Wolbachia implementers to understand their work but have no direct commercial interest.

Comment by Rory Fenton on The Future Fund’s Project Ideas Competition · 2022-03-03T18:37:57.890Z · EA · GW

Campaign to eliminate lead globally

Economic Growth

Lead exposure limits IQ, takes over 1M lives every year and costs Africa alone $130B annually, 4% of GDP: an extraordinary limit on human potential. Most lead exposure is through paint in buildings and toys. The US banned lead paint in 1978 but 60% of countries still permit it. We would like to see ideas for a global policy campaign, perhaps similar to Bloomberg’s $1B tobacco advocacy campaign (estimated to have saved ~30M lives), to push for regulations and industry monitoring.

Epistemic status: The “prize” feels very large but I am not aware of proven interventions for lead regulations. 30 minutes of Googling suggests the only existing implementer (www.leadelimination.org) might be too small for this level of funding so there may not be many applicants. 

Conflict of interest: I work for a small, new non-profit focused on $B giving. We are generally focused on projects with large, existing implementers so have not pursued lead elimination policy beyond initial light research

Comment by Rory Fenton on The Future Fund’s Project Ideas Competition · 2022-03-03T18:29:05.828Z · EA · GW

Pilot emergency geoengineering solutions for catastrophic climate change

Research That Can Help Us Improve

Toby Ord puts the risk of runaway climate change causing the extinction of humanity by 2100 at 1/1000, a staggering expected loss. Emergency solutions, such as seeding oceans with carbon-absorbing algae or creating more reflective clouds, may be our last chance to prevent catastrophic warming but are extraordinarily operationally complex and may have unforeseen negative side-effects. Governments are highly unlikely to invest in massive geoengineering solutions until the last minute, at which point they may be rushed in execution and cause significant collateral damage. We’d like to fund people who can:

  • Identify and pilot at large scale top geoengineering initiatives over the next 5-10 years to develop operational lessons. E.g. promote algae growth in a large, private lake, launch a small cluster of mirrors into space
  • Develop advanced supercomputer models, potentially with input from the above pilots, of the potential negative side-effects of geoengineering solutions
  • Identify and pilot harm-mitigation responses for geoengineering solutions

Epistemic status: there seems to be reasonable expert agreement on the kinds of geoengineering solutions that might work. I have no idea how much funding geoengineering pilots might need.

Conflict of interest: I work for a small, new nonprofit focused on $B giving. We are generally focused on projects that already have large implementers so have not pursued geoengineering beyond initial light research