Some benefits and risks of failure transparency

post by Vaidehi Agarwalla (vaidehi_agarwalla) · 2022-03-27T02:09:35.305Z · EA · GW · 3 comments

Contents

  Key takeaways
  Introduction
    Failure discourse in EA currently
  Benefits and risks of failure discourse
    What are the potential benefits?
      Building a more accurate map of the world
      A strong and trustworthy signal of our values
      A stronger community
    What are the potential risks and/or downsides?
      Opportunity Cost
      Reputational Costs
      Harming discourse
      Career risks
  When might we engage in discourse?
    Our own failures
    Others’ failures
  Further reading
None
3 comments

Key takeaways

Linguistic note: I refer to organisations and individuals collectively as “actors”. I also use the terms “failure” and “mistakes” somewhat interchangeably throughout this post. But I try to stick to “failure” for the most part.

Thanks to Arjun Khandelwal for extensive copyediting and review, and Adam Gleave for brainstorming on whiteboards. Thanks to Nathan Young, Abi Olvera, Ben Millwood, Arjun & Adam for many (many) helpful suggestions and comments. I’ve tried to cite in the footnotes where possible but I’m sure I’ve forgotten a bunch!

Introduction

The goal of this post is to discuss some heuristics on when it makes sense to discuss our own and other’s failures and mistakes in the EA community. I hope it provides a balanced and fair view of the considerations.

Failures are pretty complex and can be caused by a lot of different things. Some are caused by mistakes that were knowable in advance, others by things that could not have been predicted or were out of someone’s control. Some failures are related to not living up to one’s values and causing harm to other people, while other mistakes negatively impact one’s own productivity or growth without causing direct harm to anyone else.[1] It seems like all of the above are valuable, depending on who is reading. People external to EA might particularly care about value-related failures, while those working at EA organisations might benefit from knowing how other organisations have made mistakes that negatively harmed their productivity.

Failure discourse in EA currently

This section might be most useful to those new to the community or interested in a summary of failure discourse in the EA community.

The EA community can be keen to learn from mistakes and not punish people overly for making them in general, which can be rare.[2] A highlight is the presence of several public “Mistakes” pages where organisations discuss substantive ways they’ve made mistakes. This is partly through the presence and influence of GiveWell, which has been a champion of transparency in general, including failure transparency, in the nonprofit world and has maintained a prominent and substantive Mistakes page since its inception. Some other prominent EA orgs like CEA, 80,000 Hours, Giving What We Can, and Animal Charities Evaluators also maintain substantive Mistakes pages [3], which is likely very unusual in the broader nonprofit world. 

These pages are a costly signal of an intention to learn from mistakes and, to some extent, be publicly accountable for them. While some of the mistakes have long been common knowledge among parts of the EA community, putting them out in the open for anyone to see allows outsiders to form a much more accurate understanding of EA organisations without having to undergo a potentially costly process of becoming socially connected within the community. Having public documentation instead of relying on social connections and word of mouth is more robust for community insiders as well. 

Discussing mistakes at public events (e.g. EA conferences) is much less common. The only instance I’m aware of is the Celebrating Failed Projects Panel at EAG SF 2017.

In the past 6 months or so, there has been a lot of interest in having more critiques, and specifically red-teaming [? · GW]. Cremer and Kemp’s Democratising Risk [EA · GW] post sparked some conversation about how open EA is to critiques, and many [EA · GW] EA [EA · GW] funders [EA · GW] and leaders [EA · GW] commented in support of more criticism. A notable recent post from Jan Kulveit and Gavin Leech discusses the EA and rationalist communities’ failures in response to COVID-19 [EA · GW]. [4] Training for Good is running a red-teaming workshop [EA · GW], and there are multiple contests which support critiques - the Effective Ideas’ blog contest and just this week a contest for critiques and red-teaming was pre-announced [EA · GW].[5]

It seems like now might be a good time to reflect on some of the benefits and risks associated with failure discourse.

Benefits and risks of failure discourse

What are the potential benefits?

Building a more accurate map of the world

With more information we can help the community build a more accurate map of the world together [EA · GW] and update our decision-making accordingly. EA is young and new, it stands to reason that our map of the world has a lot of room for improvement.

Building a more accurate map may result in resources being better allocated, which seems generally net good for the world even though it can often be bad for individual actors, i.e. directed away from actors who are consistently underperforming or making mistakes. In some cases, it may even be a good sign to actors to try to find other ways to contribute, resulting in a better allocation of talent. [6]

A more accurate understanding of the world would enable us to identify and highlight trends in mistakes. Especially if done well/robustly, this should help us improve as a community. [7]

Sharing our mistakes can help the broader world outside of EA that is doing good improve as well (not just from a community building perspective). [8]

A strong and trustworthy signal of our values

When we talk about our mistakes publicly (which is not always advisable, see below), we are creating a very strong signal of our values. This benefits the community internally by encouraging and codifying our norms of transparency. Hopefully, this could create an environment that is more welcoming of criticism (both internally and externally) and makes us less susceptible to things like motivated reasoning. It also sends a strong signal to the outside world about our credibility. The higher the status of the person who talks about their own mistakes, the stronger the signal. Finally , it could nudge the outside world towards normalising this kind of transparency and discourse more. 

A stronger community

Demonstrating vulnerability and honesty can help communities grow stronger and closer together. Here are two examples from within the EA community:

The EA student group PISE has had some success [EA · GW] with being open and vulnerable in their community. 

What are the potential risks and/or downsides?

Opportunity Cost

If an actor chooses to engage (by sharing their own or others’ mistakes, or engaging in the discourse that follows), they are effectively taking time away from “direct” work they could be doing. [6]

It can also cost others resources to consume the information in a way that's useful to them, though this is more a reduction in benefit than a real cost since people can just not read it. [9]

Transparency in general can be very costly. Open Philanthropy changed their stance on transparency in 2016 after reflecting that it wasn’t directly aiding their goals (unlike GiveWell, whose goal is to make recommendations to the public). Charity Entrepreneurship decided to continue to publish their work in 2020 and accept the increased cost of spending more time polishing reports, but said they would evaluate in 2021. As of writing, their 2021 annual report has not yet been published.

Reputational Costs

Sometimes, writing about your mistakes could result in an actual loss of status or reputation. How much it could affect your reputation could be the function of how long ago it was, how understandable it was, how much you’ve changed since then and other factors, whether other people know about them.

If you demonstrate that you’ve learnt from your mistake, and have taken steps to improve, then sharing your mistakes could have a neutral or even positive effect. There are also instances where not sharing can be harmful to your reputation. If the mistake you are not sharing is something that might come up - say, if a potential employer was to speak to a reference of yours, then not owning your mistake may be perceived as either not being self-aware or reflective, or at worst, like you are trying to hide it.

Harming discourse

The way we react to actors sharing their own mistakes, or when we raise others’ actors’ mistakes can negatively harm discourse norms going forward. Jeff Kaufman [EA · GW] writes, “If people react critically and harshly to … failure, it makes organizations much less likely to be willing to be so transparent in the future. And not just this organization [or actor]: others will also see that sharing negative information doesn't go well.” This is less true “if the norm is very strong, then the pressure is not going to keep people from sharing similar things in the future, and it also means that seeing a failure from this organization but not from others is informative. On the other hand, if the norm is weaker we need to be careful to nourish it, not pushing it harder than it can stand.” It seems that EA does have somewhat strong norms around sharing failure, but it’s not clear to me how strong.

Another way that discourse norms can be harmed is that having very high standards for criticism, extending to things like tone can make it less likely that criticisms are raised. It’s possible that some people wouldn’t make the criticism rather than investing resources into trying to say it perfectly.[10] Some people I discussed Democratising Risk [EA · GW] with thought that it was tonally inappropriate - that the authors should have expressed their negative experience separately from announcing their paper and/or expressed the negative experience more objectively. I can understand where this is coming from, but personally I’d rather live in a world where something is said imperfectly than a world where it isn’t said at all. 

Career risks

If you’re in a competitive or reputation-heavy field such as politics, transparency could be used against you and affect your future prospects. 

When there is imbalance in transparency, organisations that are more transparent could be evaluated unfairly. If some actors in a space are much more transparent than others, and they seem to be doing less well than other, less transparent orgs, they would be evaluate against different standards. This could set up unhelpful incentive structures, which could encourage less transparency. [9]

When might we engage in discourse?

Although I personally think it’s usually quite valuable, and would want the community to discuss our failures less than we could, it may not always be worth it for individual actors to engage in failure discourse. The following are some decision-making heuristics that might help you make this decision. The examples are somewhat focused on whether you’d want to share your thoughts publicly, but I think they could also be helpful for deciding whether to engage in more private reflection too (e.g. sharing your thoughts with a small group of people).

Our own failures

Are you in a position where you can accept your mistakes?

How is it relevant to your ability to have impact or your work? How important is the mistake?

How much information do you stand to gain from investing resources into reflecting on this?

Do you think others could help you learn? Are you worried about motivated reasoning?

How many other people could find your insights valuable to learn about? 

What is your status within the EA community?

How competitive is your industry or profession - could this mistake be used against you?

Others’ failures

What is the appropriate fora of engagement?

What is the positive impact you think your feedback will have? What is your theory of impact or change?

What is the status or position of the actor you are criticising?

Do you have time to be careful when expressing your level of confidence and extrapolating?

Further reading

If you’re interested in actually doing this, some relevant reading might be a post I wrote on suggestions for online discourse norms [EA · GW]. There are also lots of interesting posts in the discussion norms tag [? · GW]. I also found Giving and receiving feedback [EA · GW] by Max Daniel very actionable - he suggests many specific questions and gives lots of examples, and other resources are linked in the comments of that post.


  1. ^

    See GiveWell’s current and past mistakes pages for examples of both.

  2. ^

    Here are some mostly non-EA examples. The EA-adjacent development charity Evidence Action shut down one of its programs a few years ago after publicly saying it wasn’t effective enough. In academia, there is a strong norm against transparency around failure - exemplified by economist Johannes Haushofer’s CV of failures which went viral in 2016 (and this earlier 2010 call in Nature for A CV of failures). One place where failures are discussed a lot, and sometimes embraced is in the startup world. Startup culture talking about failures - the leading thinkers in this space such as Eric Ries (The Lean Startup) and Paul Graham talk a lot about failures. VCs sometimes look favourably on founders who’ve got one failed startup behind them, if they can demonstrate they’ve learnt things. Ramit Sethi, a popular blogger who writes about careers, has a failure file where he aims to have 4 failures a month

  3. ^

    For those curious, the following organisations do not (to the best of my knowledge) have mistakes pages: FTX Future Fund, Open Philanthropy, One for the World, Founders’ Pledge, Rethink Priorities, Charity Entrepreneurship, Global Priorities Institute, Future of Humanity Institute. Some of these organisations do write about their mistakes in their annual reports. Other orgs that have mistakes pages: Sentience Institute

  4. ^

    Some other posts are the Good Technology Projects' postmortem [EA · GW], a postmortem of a mental health app [EA · GW] by Michael Plant, organisations discuss their learnings in retrospectives like Fish Welfare Initiative [EA · GW] or in posts announcing decisions to shut down like Students for High Impact Charities [EA · GW]. In the Rationalist community, there was the Arbital Postmortem [LW · GW]. You can see more examples on the Forum postmortems and retrospectives [? · GW] tag, and examples from the LessWrong community in their analogous [? · GW] tag.

  5. ^

    H/T Arjun Khandelwal for most of this section! 

  6. ^

    H/T Aaron Gertler & Adam Gleave

  7. ^

    H/T Abi Olvera

  8. ^

    H/T Arjun Khandelwal

  9. ^

    H/T Ben Millwood

  10. ^

    Chris Leong raised this concern on the pre-announcement for the contest for critiques and red teaming

3 comments

Comments sorted by top scores.

comment by Vaidehi Agarwalla (vaidehi_agarwalla) · 2022-03-27T02:12:19.451Z · EA(p) · GW(p)

Writing as a comment because it didn’t feel very central to the post. I want to share some thoughts on my motivation to write this post and how it evolved. Initially, I was going to write a slightly different post which emphasised the following points:

  1. We’re relatively overreporting successes and underreporting mistakes, failures, or lessons learned. (pretty strong claim)
    1. It’s difficult to quantify what is “enough” of the “right type” of discourse. This feels inherently fuzzy. It’s not just about the quantity of posts - I think the “right” level of discourse would make me feel like I’m getting exposed to lots of peoples’ internal models or hypotheses on how they expected things to go, and how they went wrong, and compare their models to mine to try and figure out if I could avoid making the mistakes they are making.
  2. Other community members could find it valuable to read about those experiences (medium strength claim)
    1. It would help them develop better models and thus make better decisions
    2. It could help them feel less alone
  3. It’s worth the cost for actors to invest more time into such reflection (somewhat confident claim)

I was motivated to write this for three reasons:

  • Despite having strong values of transparency in EA, there still feel like strong disincentives to write about our failures. The whole set-up of EA is kind of like “other people made mistakes, and we’re going to do better”. That sets a pretty high bar for trying new things, or sharing “dumb mistakes” (or "dumb questions"). Of course, if the bar is set too low, you might have much less impact than you could otherwise. But if it’s too high it could disincentivize people from trying things.
  • The recent influx of FTX funding, as well as the uptick in interest around EA entrepreneurship and people starting more, and more ambitious, projects means that we are likely to see more failed early-stage EA projects. I wasn’t sure how we’d deal with this
  • I felt there isn't enough transparency around failure in meta, despite the amount of uncertainty. It may be that we just don’t make that many mistakes - that seems very unlikely. When I reflect on my own time in community building, I’ve changed my mind many times (and continue to do so), most major meta organisations have as well.

While writing this post the feedback I got made me realise that rather than just advocating for "more of X" it would be more valuable to help people by providing heuristics for making decisions on how and when to engage in failure discourse.


 

Replies from: tamgent
comment by tamgent · 2022-03-27T09:33:23.000Z · EA(p) · GW(p)

Thanks for sharing your motivations! Personally, I would have liked to read your original post, even if it was more one-sided, and got the other side elsewhere. Being helped with heuristics for making decisions is not really what I was looking for in this post - it feels paternalistic and contrived in me, and I'd enjoy you advocating earnestly for more of something you think is good.

comment by tamgent · 2022-03-27T09:57:04.006Z · EA(p) · GW(p)

Some things I like about this post:
 - I like the topic, I am interested in failure and places where failure and mistake making is discussed openly feels more growthy.
- I liked that you gave lots of examples.

Some things I didn't like about this post:
- Sometimes I couldn't always see the full connections you were making, or I could but had to leap to them based on my own preconceptions, maybe they could be more explained? For example, a benefit was a stronger community, but you didn't explain the mechanism by which that leads to a stronger community. I don't think the Howie podcast supports the point, a lot of people liked the podcast, but how is that indicative of a stronger community exactly?

Things I disagree with in this post:
- I don't think the Opportunity Cost point was well argued. In particular, you discussed transparency in general, with examples of publishing annual reports and so on, which take a lot of time. However, this post is about being transparent about mistakes and failure, not transparency in general. I think the Opportunity Cost is much lower for just publishing big mistakes, even though it takes some time to word it properly, and then there is the stress of it. But you can choose simply not to look at reactions on social media. Same as people can choose not to engage in lengthy threads about it.
- I think your Reputational Cost point was better on the other side as some of the reasons would put it there. Also, I just think this is somewhat a normative cultural question rather than one about facts in the world. If my reputation will be destroyed in an area for publishing a mistake, either that is a good thing, or the person judging is undervaluing the growth/learning part and overvaluing a fixed view of people. I basically don't think someone who would incorrectly judge me negatively for publishing a mistake is worth me caring about the opinion of. Again, this is normative, not a fact about reality, it's about what kind of culture we want to create.
- Similar arguments to Reputational Cost apply to the Harming Discourse point - this is a normative culture question, we get to choose how we respond and whether we reward or disincentivise it! I would put it not as a risk/downside but in another category called cultural equilibrium or something, along with the reputation point. 
- I don't think the Career Risk point is different to the Reputational Cost point in any meaningful way. You can also take more ownership as an organisation rather than an individual, where appropriate. 

I recognise that the things I disagree with are all in the downsides/risks section, and that is because I am biased and uninterested in critiquing the other side. I feel somewhat entitled to do this because I'm under the impression that you added this section in after feedback to make it more balanced, so it's partially because I'm being mischievous and unfair (you made this easier), as well as not wanting to feel pressure myself to give a balanced comment and wanting to protest against feeling constrained in that way.