Do research organisations make theory of change diagrams? Should they?

post by MichaelA · 2020-07-22T04:58:41.263Z · score: 36 (13 votes) · EA · GW · 1 comment

This is a question post.

Contents

  My questions
  Context
None
  Answers
    28 Max_Daniel
    17 MichaelPlant
    15 Lukas_Gloor
    12 Joey
    7 MichaelA
    5 ishaan
    3 Matt_Lerner
    3 RayTaylor
    2 MichaelA
None
1 comment

My statements here do not necessarily reflect the views of any of my employers.

My questions

  1. Do many research organisations (within or outside of EA) make theory of change (ToC) diagrams? If not, why not?
  2. Do many research orgs make ToC diagrams, but not make them publicly accessible? If so, why?
  3. Should more research orgs make ToC diagrams? Why or why not?
  4. Should more research orgs make their ToC diagrams publicly accessible? Why or why not?

(You don’t need to answer all four questions; feel free to just address one of them, or just say what some org you know about does and why, or whatever.)

Context

According to Wikipedia:

Theory of Change (ToC) is a specific type of methodology for planning, participation, and evaluation that is used in companies, philanthropy, not-for-profit and government sectors to promote social change. Theory of Change defines long-term goals and then maps backward to identify necessary preconditions.[1]

Theory of Change explains the process of change by outlining causal linkages in an initiative, i.e., its shorter-term, intermediate, and longer-term outcomes.

I haven’t read the whole of that article, and I’m not an expert on ToC methodologies. When I say “theory of change diagrams”, I essentially just mean flowcharts/causal diagrams showing how an organisation thinks the concrete actions they plan to take might connect to the outcomes they ultimately care about causing. (Please let me know if that’s me stretching the jargon too far [EA · GW].)

Here’s an example of a ToC diagram, from the EA org Happier Lives Institute (source):

And here’s a ToC diagram I participated in creating, from the EA org Convergence Analysis (which I worked for until recently) (source):

(See the relevant section of Convergence’s strategic plan for more explanation and context. One relevant excerpt is: “We have refrained from showing arrows to connect each activity or outcome to those it helps bring about. This is partly because there are so many connections, and we wished to avoid cluttering the diagram.”)

It seems to me that it might be quite useful for research orgs to make ToC diagrams. But I haven’t actually seen any examples of ToC diagrams from research orgs except the two shown above. This has led me to wonder about the four questions stated above.

Answers

answer by Max_Daniel · 2020-07-22T07:18:38.417Z · score: 28 (14 votes) · EA(p) · GW(p)

I could go on a rant about this... :) Very briefly, my own impression is that:

  • ~All organizations should do explicit strategic planning, including research organizations.
  • Usually these plans should be public.
  • A theory of change diagram would be a good example of what I'd like to see when doing strategic planning, though not exhaustive / sufficient in itself. More broadly, I'm not too fuzzed about the specific methodology used, or way in which plans are communicated.
  • Perhaps surprisingly given these views, I mostly agree with the adage that "plans are useless, planning is everything". I expect that usually the main value of strategic planning will be due to improving the thinking of organizational leadership rather than for example due to enabling external feedback. (This can be different for some organizations, e.g. when buy-in by many external stakeholders is unusually important.) So for example the main reason I think plans should be public is that it incentivizes leadership to expend more effort and rigour when doing strategic planning. (I expect pushback along the lines of "but making plans public incentives plans that sound defensible rather than being actually useful". I don't have time to go into more detail but my response is: yes, these effects are real, but they can be mitigated to a large extent, and overall I think being public is still worth it on net).

But I've also found that I tend to disagree with staff at EA (research) organizations on this, including the people running the organizations or responsible for strategy. Sometimes the disagreement seemed fairly large, to the extent that it was a significant and sometimes decisive factor in my decisions between different jobs.

So after updating on the revealed and stated preferences of people with roughly similar amounts of or more experience in organizational leadership, my best guess is that the optimal policy is a bit closer to their impressions than mine.

(Of course, these disagreements are gradual rather than binary. It's not that people say "we should never think about our strategy". It's more that e.g. they think they have thought about strategy organically anyway, and that there's not sufficient value added by explicitly engaging in something labelled 'strategic planning' to justify the time cost.)

answer by MichaelPlant · 2020-07-29T13:44:14.848Z · score: 17 (5 votes) · EA(p) · GW(p)

At HLI, we've found creating a Theory of Change (TOC) very useful. It was (at least for me) quite a painful process of making explicit various assumptions and uncertainties and then talking through them. I think if we hadn't done it explicitly we would (a) have made a less thoughtful plan and (b) different members of the team would be carrying around their own plans in their heads.

Going through a ToC process has also helped us to focus on meeting the needs of our target audiences. After developing our TOC, we sent out surveys to some of our key stakeholders to identify their concerns about subjective well-being measures and what new information would make them more likely to use them. Their responses provided the basis for our research agenda and the questions we have chosen to investigate this year.

We have a slightly more detailed version of our ToC diagram on our blog. Thanks for pointing out that it’s hard to find; we’ll think about putting it on a main page.

comment by MichaelA · 2020-07-30T00:16:58.813Z · score: 4 (2 votes) · EA(p) · GW(p)

Very interesting, thanks for sharing!

And thanks for linking to your blog post with the ToC. I've now updated the post in light of that (by linking to that post rather than the video, and removing the sentence "I couldn’t quickly find Happier Lives Institute’s ToC diagram on their site, so it seems possible that they no longer consider it valuable, or that they primarily intended it just for internal use and then happened to use it in that one video").

comment by BarryGrimes · 2020-07-29T17:04:05.756Z · score: 3 (2 votes) · EA(p) · GW(p)

Having a well-developed Theory of Change has been essential for developing HLI’s communications strategy too. Without org-wide agreement on our route to impact, it would have been impossible to think about the changes in behaviour we are trying to bring about. It influences our audience personas, the communication channels we use, our writing style, and our branding.

Although much of our strategic planning remains internal, I think it’s important to be transparent about our high-level ToC so others can provide feedback, place our research in the correct context, and decide if they want to collaborate with us.

answer by Lukas_Gloor · 2020-07-22T09:42:03.114Z · score: 15 (7 votes) · EA(p) · GW(p)

Context: I'm drawing from experience with a small research organization in a young field where it used to be very hard to do good research without thoroughly understanding the causal paths to impact.

Strongly stated, weakly held, and definitely tainted by personal idiosyncrasies:

I often found myself suspicious about (too many) internal strategy documents because I think that in a well-functioning organization of the kind I described, the people who make prioritization decisions (researchers pursuing their interests autonomously or executive director/managers who define tasks and targets at the organization-level) should be hired, among other things, for their prioritization abilities.

My sense is that being good at prioritization is more about the mindset than following some plan, and it involves thinking through the paths to impact for every decision, every day. So when I'm asked to help write up a theory of change, my intuitive reaction is "Who is this for? This feels like tediously writing down things that are already second nature to many people, and so much goes into this that it's hard not to come away from this feeling like the document produced is too simplistic to be of any use."

So, I'm overall skeptical about the use of ToC documents for improving a small organization's focus, especially if the organization operates in a field/paradigm where staff have already been selected for their ability to prioritize well.

To be clear, this isn't measured against a comparison of not thinking about strategy at all. Instead, I favor leaner versions of strategy discussions. For instance, one person writes up their thoughts on what could be improved (this might sometimes look like an abbreviated version of a ToC document), then core staff use it as a basis for group discussions and try to identify the non-obvious questions that seem the most crucial to the organization's strategic direction. Then one discusses these questions from various angles, switches to a solution-oriented mode, and defines action points. The result of those discussions should be written down, but there's no need to start at "our mission is to reduce future suffering.")

Of course, there might be other reasons why internal ToC documents could be useful. For instance, not everyone's work involves making big-picture prioritization decisions, and it's helpful and motivating for all staff to have a good sense of what the organization concretely aims to accomplish. Still, if the reason for writing a ToC document is updating staff instead of actually improving overall prioritization and focus, then that calls for different ways of writing the document. And perhaps doing a (recorded) strategy Q&A with researchers and the executive director might be more efficient than a drily written document with rectangles and arrows.

Another instance where ToC documents might be (more) useful is for establishing consensus about an organization's aims. If it feels like the organization lacks a coherent framework for how to think about their mission, maybe the process of writing a ToC document could be helpful in getting staff to think along similar lines.

comment by Max_Daniel · 2020-07-23T10:00:07.139Z · score: 17 (5 votes) · EA(p) · GW(p)
My sense is that being good at prioritization is more about the mindset than following some plan, and it involves thinking through the paths to impact for every decision, every day.

I don't strongly disagree with this. (Though based on conversations we've had elsewhere I'd guess I'm still somewhat more positive about plans than you are.)

However, I actually think of day-to-day prioritization decisions on one hand and the kind of strategic planning I'd like to see more on the other hand as two quite different activities and to some extent skill sets. I think they are both important, and complement each other well.

I think one major difference is which decisions you're thinking about in the first place, and in particular how reactive versus proactive you are.

  • A lot of prioritization decisions are responding to a specific stimulus such as someone inviting you to give a talk or proposing a joint project. Others at least are very salient, e.g. because they concern the allocation of resources that are available by default - such as staff time. For these - and more broadly once you've identified a specific decision as important to make - I agree that following some explicit plan usually won't add much.
  • However, there are important prioritization decisions that will never, or only too late, become salient by following day-to-day incentives. Have you really considered the full option space of how to reach your goals? Have you thought about how to prevent low-probability risks to your organization? Have you developed metrics that tell you whether you're on track well before you can recognize obvious problems or achievements? Can you get more data to help you make decisions? I basically think that good strategic planning processes are a glorified checklist that direct leadership attention to issues that might otherwise be overlooked.

For example, I've seen all of the following happening (not at CLR):

  • Some years into a program, management wants to do an impact evaluation but hasn't done a baseline assessment. They can describe how their target audience is doing now, but when assessing change they have to rely on memory.
  • Some months into a program, management realizes that some aspects don't work well for some of their target audience. They say "it seems like we've designed a program for people who are like us, but actually some members of the target audience are quite different", and make changes.

Seeing what in my view are clear and basic mistakes like these on one hand and skepticism about strategic planning on the other hand reinforces my view. These are paradigm cases of mistakes that standard strategic planning processes have been designed to avoid. For example, any material I can think of will emphasize baseline assessments (and more broadly planning for the ability to evaluate impact from the very start) and the need to understand your target audience e.g. through interviews or other ways of data collection.

comment by MichaelA · 2020-07-22T23:32:51.632Z · score: 2 (1 votes) · EA(p) · GW(p)

Interesting perspective, thanks for sharing.

It sounds like part of your thinking is that ToC diagrams won't add much value when (1) the organisation already has consensus / "a coherent framework for how to think about their mission", and (2) all of its researchers are already very good at prioritization. I'd guess that both of those conditions will be harder to maintain as an organisation scales up. Would you guess that ToC diagrams tend to become more useful as organisations scale up?

Also, when you say "prioritization abilities", do you just mean ability to prioritise between research questions? Or also things like ability to generate new research questions, generate ideas of non-research activities to do (e.g., different ways of disseminating research findings to different audiences), and prioritise among those non-research activities?

I ask largely because one reason I suspect ToC diagrams may be helpful is to guide decisions about things like which forms of output to produce, who to share research findings with, and whether and how to disseminate particular findings broadly. It seems plausible to me that a researcher who's excellent at prioritizing among research questions might not be good at thinking about those matters, and a ToC diagram (or the process of making one) might speed up or clarify their thoughts on those matters.

But you might find that researchers with that discrepancy of skills are uncommon or selected against, or that strategy discussions without ToC diagrams can cover that, or that staff members other than researchers can cover that.

comment by Lukas_Gloor · 2020-07-23T07:15:36.078Z · score: 6 (3 votes) · EA(p) · GW(p)
I'd guess that both of those conditions will be harder to maintain as an organisation scales up. Would you guess that ToC diagrams tend to become more useful as organisations scale up?

I think so. I'm somewhat nervous about this because if the culture changes drastically, maybe that's generally bad and ToC documents just mitigate some of the badness, but can't quite get you back the culture at a smaller organization. Whether large scaling even makes sense might depend on the organization's mission, or the ability of the executive director (and hiring committee) to scale in a way that preserves the right culture.

Also, when you say "prioritization abilities", do you just mean ability to prioritise between research questions?

Also the other things you list.

I ask largely because one reason I suspect ToC diagrams may be helpful is to guide decisions about things like which forms of output to produce, who to share research findings with, and whether and how to disseminate particular findings broadly. It seems plausible to me that a researcher who's excellent at prioritizing among research questions might not be good at thinking about those matters, and a ToC diagram (or the process of making one) might speed up or clarify their thoughts on those matters.

That seems reasonable. My experience is that people often know the right answers in theory, but need a lot of nudging to choose mediums or venues different from the ones they find personally the most rewarding. I think there are also just large constraints by individual psychology that make things less flexible than one might think. So, to preserve intrinsic motivation for research, it's maybe not a good idea to push researchers too much. Still, I think it's crucial to have a culture where researchers think actively about which medium to pick, why they're doing it, and how the output will be shared. As long as this is being diligently considered and discussed, I think it's reasonable to defer to the judgment of individual researchers.

comment by MichaelA · 2020-07-23T00:13:48.356Z · score: 2 (1 votes) · EA(p) · GW(p)

Thanks for linking to that CLR post; that was an interesting snapshot into the process/output of an org's strategic thinking.

The post says:

In future docs inspired by this outline here, we are going to list the pros and cons for each of the above proposals in order to then assign rough weightings to them.

Do you know if those further docs ended up being written, and made public?

comment by Lukas_Gloor · 2020-07-23T07:04:28.517Z · score: 4 (2 votes) · EA(p) · GW(p)

I did write something that builds on it, yeah. It was about defining various proxies to optimize for (e.g., money, societal influence, connections to other EA organizations, followers of the organization's newsletter (with near-term EA as their main interest), value-aligned people with computer science expertise, etc.) and how well they do in futures where we decide different interventions are most important. I didn't want to make it public because it felt unpolished, and I was worried that some of the proxies could give outsiders the impression of instrumentalizing people.

Someone even helped me with excel to produce a heat map of the results weighted by probability we assign to various interventions mattering the most, and at the time this helped me clarify objections I had to EAF's 2015/2016 strategic direction (we interacted little with other EA orgs and tried to build up capacity with animal advocacy, but always promote cause neutrality with the intent of maybe pivoting to other causes later). It didn't lead to many important changes right away, but we made major changes in 2017 that strongly reflected the takeaways I had sketched in those documents.

answer by Joey · 2020-07-22T08:55:18.411Z · score: 12 (5 votes) · EA(p) · GW(p)

Sadly don’t have time to go into much depth on this, but we strongly recommend it to all charities that run through our CE program (including all the research orgs) and create a ToC for each idea we research.

comment by MichaelA · 2020-07-23T00:00:43.712Z · score: 2 (1 votes) · EA(p) · GW(p)

Thanks for that info. Do you think you'd do/recommend the same for charities or research focused on the long-term future? (Feel free to give an answer without explanation, given your time-constraints.)

answer by MichaelA · 2020-07-22T05:00:23.491Z · score: 7 (4 votes) · EA(p) · GW(p)

Here are my own current thoughts on those questions. These thoughts are of course tentative, and my main aim here is to learn more on this matter (otherwise I would’ve made a regular post rather than a question post):

On Q1: It seems like it’s rare for research orgs to make (publicly accessible) ToC diagrams? As I said, I only know of two examples of research orgs that have made such diagrams. But I haven’t done a systematic or extensive search.

If it’s indeed uncommon for research orgs to make ToC diagrams, that might be due to the “potential downsides” I list below.

On Q2: I have basically no evidence regarding whether many orgs make non-public ToC diagrams. I’ve never seen an example of that, but that’d likely be the case even if there are many such non-public diagrams, since I’ve only worked at one research org thus far (Convergence), and their ToC diagram was public anyway.

On Q3: My independent impression - i.e., what I’d believe if I wasn’t updating my beliefs based on what others seem to believe - is that, indeed, more research orgs should make ToC diagrams. But if so few research orgs do that, and most research orgs are run by smart people, maybe I should interpret that as evidence that there’s some good reason not to do this which I’m missing?

But I’m not sure how strong that update should be, because it might be more a matter of norms, inertia, and/or not having thought of doing this.

Some potential benefits of making ToC diagrams (even if they aren’t publicly accessible) that came to my mind are that doing so could help the org:

  • Make decisions about which research questions/directions to pursue
    • E.g., maybe you’d realise that a line of research that seems fascinating to your researchers actually doesn’t have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research. (But this consideration could be outweighed. See also Can we intentionally improve the world? Planners vs. Hayekians)
  • Make decisions about which audiences to target
    • E.g., you might realise that, in order to achieve your ultimate intended outcomes, you mainly need to influence decisions by [policymakers / AI researchers / EAs interested in global health / company leaders / whatever]. You can then have that in mind as you choose research questions, writing styles, types of output (e.g., EA Forum posts vs academic articles), etc.
  • Make decisions about what non-research activities to do, how to do them, and how much time to allocate to them
    • E.g., you might realise that most of the outcomes you want to bring about require specific people to read your work, such that it’s worth dedicating a decent amount of time to building relationships with those people, attending relevant conferences, etc., rather than just doing research and assuming it’ll have an impact somehow.
  • Work out how they can assess their progress and their impact
    • E.g., my personal opinion is that, once Convergence had made a ToC diagram, this helped us come up with metrics for assessing our progress and impact. It’s practically impossible to “directly” assess whether one has improved the expected value of the future (which was our ultimate aim), but easier to assess how well we’re achieving outcomes on the causal path from our actions to that outcome.
      • Note I said “easier” rather than “easy”, and that I’m comparing this to something that’s “practically impossible” - I definitely still find assessing the impact of longtermist research very difficult!

(Note that each of those benefits could likely be captured to at least some extent by an org just describing their theory of change fairly explicitly, without actually making a diagram about it. The same point applies to the remainder of this “answer”.)

And some potential downsides of making ToC diagrams that came to my mind are that doing so could:

  • Take up a lot of time
  • Not actually provide much clarity anyway
  • Lead to focusing too much on a particular set of paths to impact that one explicitly maps out, at the expense of doing things that seem worthwhile based on intuitions and heuristics
    • See again Can we intentionally improve the world? Planners vs. Hayekians, and perhaps also Goodhart’s law.
    • Of course, it’s hard to say when it’d be better to operate based on intuitions and heuristics rather than a ToC
    • It also seems possible that the ideal would be to make a ToC, and sometimes/often refer to it, but not always refer to it or always require that a project “fits” within that ToC.

On Q4: I’d guess that, if a research org has a ToC diagram, it’d typically make sense to make it publicly accessible. This could mean actively sharing it (e.g., in a post like this one [EA · GW] or this one [EA · GW]), making it prominently visible on the org’s website, or burying it in some obscure corner of the org’s website.

Some potential benefits of making ToC diagrams publicly accessible (rather than just making them in the first place) that came to my mind are that doing so could:

  • Help potential donors, potential employees, etc. understand why you’re doing what you’re doing, and thus make informed decisions about whether to donate to you, work for you, etc.
    • E.g., perhaps what you’re doing seems inefficient or unlikely to connect to important real-world outcomes unless people understand your ToC
    • Or perhaps what you’re doing is misguided, and it’d be good for the world for people to have more evidence of that if that’s true, so that they can then support something better instead
  • Help other orgs learn from and adopt clever aspects of your strategy and thinking
  • Help stakeholders coordinate with you or support you
    • I’m less sure what I could mean by this point
    • One vague and speculative example: If many orgs’ ToC diagrams suggest that a particular type of conference would be useful in order for those orgs to make their research impactful, and those ToC diagrams are publicly accessible, this could lead to someone setting up such a conference.

(It might be possible to capture some of these benefits by just sharing ToC diagrams with specific people, such as potential donors or employees, rather than making those diagrams totally publicly accessible.)

And some potential downsides of making ToC diagrams publicly accessible that came to my mind are that doing that could:

  • Lead to misperceptions about an org’s strategy
    • E.g., it could lead to stakeholders focusing too much on the set of pathways to impact highlighted in the ToC diagram (rather than other, vaguer reasons to believe impact might occur), or perceiving the org’s thinking as quite simplistic
  • Allow other actors to harm your org or harm things you value, as a result of them learning from your strategy and thinking
    • E.g., another research org might improve itself by adopting some of your ideas, and then get some funding you’d have otherwise gotten
      • But maybe there’s often little harm in that, as that other org could then do useful things as well
    • More speculatively, if there’s some actor that actively wants to prevent you accomplishing your mission, perhaps seeing your ToC would make that easier for them

But again, these are just my current, tentative thoughts, and I’m primarily interested in hearing what others have to say!

My thanks to Courtney Henry and David Kristoffersson for discussions that informed this answer.

comment by ddavies · 2020-07-22T16:38:48.992Z · score: 4 (3 votes) · EA(p) · GW(p)

Really interesting thoughts! Thanks for writing them up.

Disclaimer: The below is descriptive, not normative, and chiefly focuses on people and organisations outside of EA

This may be overly cynical, but I think some of the reasons you list in favour of ToCs also account for why they often either don't exist or aren't publicly available, i.e. there's a misalignment between what's good for society (what you're getting at with "whether they should exist") and what the organisation and/or researchers consider to be their self-interest. For example,

maybe you’d realise that a line of research that seems fascinating to your researchers actually doesn’t have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research

As you suggest, it's very possible that sometimes what's most interesting and what's most impactful aren't perfectly aligned. Sometimes, one's skills and one's interests don't align with what's needed in the field in which one wants to work. This can be a bitter pill to swallow, especially for people who have already invested heavily in their education and problem area and can't just pivot to a more impactful project. Highlighting the discrepancy privately could jeopardise one's role or standing within an organisation - including if one's coworkers feel that you're questioning their effectiveness/utility; highlighting it publicly could jeopardise funding and jobs. Most people aren't EAs, but even EAs are human too.

Help potential donors, potential employees, etc. understand why you’re doing what you’re doing, and thus make informed decisions about whether to donate to you, work for you, etc.

At risk of repeating myself, sometimes people/orgs would rather not help potential donors, potential employees, etc. make fully informed decisions about whether to fund or work for you. This is particularly the case with grifts. That said, it also applies to organisations that want to do something, act with the best of intentions, secure funding, hire a team, and then fear looking down lest they realise that, like a cartoon character, they aren't on solid footing. Sometimes it's easier to just keep going.

Again, I'm not saying this is right, just positing it as another explanation for why we don't see more ToCs.

comment by EricHerboso · 2020-09-03T00:45:27.829Z · score: 3 (2 votes) · EA(p) · GW(p)

On Q1: You mention only being aware of a few research orgs that have public ToC diagrams. I wanted to bring your attention to Animal Charity Evaluators, which uses ToC diagrams as a way of better communicating with the public how ACE thinks that a given recommended organization might be causing one of its animal advocacy outcomes.

ACE also uses a ToC diagram in its strategic plan, but this might not be easily searchable because it exists publicly only on pdf documents. (The webpage hosting the strategic plan doesn't use the phrase "theory of change" at all, even though a ToC diagram does exist within the linked pdf there.)

comment by MichaelA · 2020-09-03T06:31:27.312Z · score: 2 (1 votes) · EA(p) · GW(p)

Thanks for your comment. 

I was aware of some non-research orgs who have ToC diagrams, and some research orgs who make ToC diagrams about things other than their own org (e.g., Charity Entrepreneurship make ToC diagrams for the charity ideas they researched). That seems fairly different from a ToC diagram for a research org itself. This is because research seems unusually hard to imagine the intended path to impact of,  and research is where I most often get a sense people don't have a clear idea about the ToC. (Whereas with a charity like AMF, it's pretty easy for me to imagine roughly what the ToC would look like, even without seeing a diagram AMF have made.) 

So I was only a bit interested in ACE's ToC diagrams about the orgs it recommends. But ACE's ToC diagram for itself was indeed quite interesting for my purposes.

For the benefit of other readers, that diagram is on page 15 of the linked PDF.

answer by ishaan · 2020-07-29T19:34:23.851Z · score: 5 (3 votes) · EA(p) · GW(p)

A forum resource on ToC in research which I found insightful: Are you working on a research agenda? A guide to increasing the impact of your research by involving decision-makers [EA · GW]

Should they

Yes, but ToC don't improve impact in isolation (you can imagine a perfectly good ToC for an intervention which doesn't do much). Also, if you draw a nice diagram, but it doesn't actually inform any of your decisions or change your behavior in any way, then it hasn't really done anything. A ToC is ideally combined with cost-benefit analyses, the comparing of multiple avenues of action, etc and it should pay you back in the form of generating some concrete, informative actions e.g. consulting stakeholders to check your research questions, generally creating checkpoints at which you are trying to get measurements and indicators and opinions from relevant people.

For more foundational and theoretical questions where the direct impact isn't obvious, there may be a higher risk of drawing a diagram which doesn't do anything. I think there's ways to avoid this - understand the relevance of your research to other (ideally more practical) researchers who you've spoken to about it such as a peer review process, make a conceptual map of where your work fits in to other ideas which then lead to impact, try to get as close to the practical level as you realistically can. If it's really hard to tie it to the practical level it is sometimes a sign that you might need to re-evaluate the activity.

Do they

Back in academia, I didn't even know what a "theory of change" was, so I think not. But, one is frequently asked to state the practical and the theoretical value of your research, and the peer review and grant writing process implicitly incorporates elements of stakeholder relevance. However, as an academic, if you fail to make your own analyses, separately from this larger infrastructure, you may end up following institutional priorities (of grant makers, of academic journals, etc) which differ from "doing the most good" as you conceptualize it.

answer by Matt_Lerner · 2020-08-23T14:40:44.613Z · score: 3 (2 votes) · EA(p) · GW(p)

I've just finished a project working with a large American foundation (not sure I'm okay to say which, but it's in the top 10 largest). They use logic models / ToC diagrams internally as their lingua franca: everything is expressed as a diagram. I feel a little ambivalent.

On the one hand, they are a clear and expeditious way of expressing information that might otherwise be crammed into a memo no one will read. They also very clearly express causal flow in a way that other media might not, which can facilitate understanding. At the foundation I worked with, they seem to be used primarily as a way of rapidly communicating mechanisms of action (e.g. in proposed foundation grants or investments) to and between program officers, who seem extremely pressed for time.

On the other hand, I also saw reports and presentations crammed full of incredibly detailed logic models. I'm talking about pages and pages on which small-type boxes and arrows completely fill each page. I really don't think this is useful. These incredibly detailed models are not easy to understand at a glance, and they seem to sit in an unhappy middle ground: by being complicated, they challenge comprehension, but by being simplifications, they occlude important details relevant to the mechanism being described.

I got the impression that because the order had come down from on high to put everything in a logic model, it was being done even in contexts where these models made no sense. I worried that the focus on logic models encourages only a logic model - level understanding of the world, while simultaneously eating up huge amounts of foundation time creating diagrams that few will look at or understand.

However, I am still a convert. I think theory of change / logic models do have a lot of value, but I think they need to be used sparingly and kept small. I'd make some kind of a rule: no more than twenty boxes in a model, or something like that.

comment by MichaelA · 2020-08-23T15:01:06.222Z · score: 2 (1 votes) · EA(p) · GW(p)

Thanks for sharing that data and perspective. One aspect of that I hadn't had in mind before was that the value of explicit ToCs, ToC diagrams, and similar might depend in part on the extent to which they're: 

  1. mandated from above and then Goodharted on, or
  2. used by people who themselves believe these things are a good idea, and only when those people believe these things are a good idea.

Also, I was only really thinking about an organisation making one, organisation-wide ToC diagram; I hadn't considered the idea of making ToC diagrams for each particular project, program, etc.

Having read your comment, I thought it'd be interesting to consider what organisation leaders should do if it would be useful for employees to make ToC diagrams for a substantial portion of individual projects, programs, etc. 

I think that, if employees aren't already in the habit and it's not already common in the organisation, employees would by default hardly ever make ToC diagrams. If so, and if we assume that making ToC diagrams often would be good, then there should probably be some sort of push from the leadership. 

But maybe this push should take the form of explicitly highlighting the option of making ToC diagrams, providing some good examples, and encouraging people to try it a few times. And then hopefully, if the employees were chosen well, they'll naturally come to use them about as often as they should. 

Whereas perhaps it would be bad to have a strong push from above, e.g. mandating the use of ToC in every case, or being so persistent in encouraging their "optional" use that employees feel compelled. Maybe that would lead to employees making such diagrams even when it's not valuable, or making such diagrams in a crappy way when a good diagram would be valuable.

comment by Matt_Lerner (mattlerner) · 2020-08-23T15:10:55.295Z · score: 7 (2 votes) · EA(p) · GW(p)
But maybe this push should take the form of explicitly highlighting the option of making ToC diagrams, providing some good examples, and encouraging people to try it a few times. And then hopefully, if the employees were chosen well, they'll naturally come to use them about as often as they should. 

This is probably the right course of action. Before the project I just finished, it was never really clear to me the settings in which flow chart-type diagrams made sense. As a more or less mathy type, I think I didn't give them their due. Now that I've seen them in practice, I've started making them here and there.

I think just giving employees the allowance to make diagrams instead of slideshows or reports, and cluing them into best practices (see e.g. this guide from the CDC) can go a long way. It seems like lots of staffers go down the report/slideshow rabbit hole because they want to be seen to be doing something. This results in long, unread memos, etc.

There's another benefit, too: staffers have sometimes dramatically different writing and design skills, and simple diagrams can lower the barriers to communicating ideas for employees who may not be confident in these skills. If staff members are held to a strict standard for the clarity and coherence of logic models, they can be a way of rapidly iterating ideas that would otherwise remain unheard.

answer by RayTaylor · 2020-08-23T13:50:07.831Z · score: 3 (4 votes) · EA(p) · GW(p)

Brilliant to raise this topic, and I like what you wrote but both diagrams are weak. For me a good diagram shows very specifically how a single change will be achieved, and shows if there is too long a chain for success to be likely.

Regardless of diagrams, we all have conscious or unconscious theories of change, and many (especially in climate change) have been useless.

The classic unconscious theory of change is:

brainy guy does research > publishes > civil servants write a policy > wise politicians decide > funds are allocated > policy is implemented well

The main weakness here is that it's a very long chain, with many obstacles in each link.

Compare to coal industry's ToC, which they learned from Big Tobacco:

"create confusion about climate science" + "capture Congress" > block all carbon tax proposals nationally and internationally

Good ToC for EAs involves:

  • selecting good and astute targets of change (whether in real world, movement or metta)
  • smart routes to achieving the change
  • updating appropriately (at a Goldilocks frequently, not too rarely to stay current, not too often to frustrate the teams doing the work)

For "natural conservatives", this may sometimes involve finding ways of opposing harmful change, and proving that some policies are a bad idea, or need fine tuning.

answer by MichaelA · 2020-09-02T12:51:56.110Z · score: 2 (1 votes) · EA(p) · GW(p)

Here's a quote from a comment Owen Cotton-Barratt's made elsewhere [EA(p) · GW(p)](focusing on explicit theories of change in general, rather than specifically ToC diagrams):

Some general thoughts:

  • Advantages of having an explicit theory of change:
    • Makes it easier to sync up about direction/priorities/reasons for doing things
    • Makes it easier for people to engage critically, or otherwise to notice mistakes and course-correct
  • Disadvantages of having an explicit theory of change:
    • Easy to have the case where your best expression of something is dumber than your real internal sense of it
      • In this case it may be preferable to be guided by the internal sense rather than the explicit version
      • (this is at least some distant relative of Goodhart's law)
    • To the extent that you're going to be guided by your internal sense rather than an explicit version, sharing something as an explicit theory of change can be misleading
  • In general I think it's good to encourage lots of explicit discussion about theories of change
    • Ideally without committing to reaching an "answer", but having that as a goal may be helpful for prompting the discussion

I think that I find the disadvantages quite emotionally resonant, which may pull me to err too far in the direction of not being explicit. I have appreciated some cases where people have pushed me towards "let's have a discussion where we're pretty explicit about best guesses".

1 comment

Comments sorted by top scores.

comment by Moses · 2020-07-22T07:58:35.303Z · score: 9 (5 votes) · EA(p) · GW(p)

I haven’t actually seen any examples of ToC diagrams from research orgs except the two shown above.

A good example of a ToC diagram is this old Leverage Research plan.