The TUILS Framework for Improving Pro-Con Analysis

post by Harrison D · 2021-04-08T01:37:29.756Z · EA · GW · 1 comments

Contents

  Summary
  Roadmap/Outline
  Disclaimers, epistemic status, and related meta-notes.
  Describing the TUILS Framework
    overview, including the four components:
    implications/contentions of the TUILS framework
    TUILS framework vs. the INT framework
    of the TUILS framework being applied
  Reasons to learn/remember/use
    Justifications (e.g., reference class justifications)
    Justifications
    (generic[6]) downsides
  Conclusion
      That being said, I am still definitely open to suggestions for different component names as well as other suggestions or criticisms more generally (in fact, getting feedback is a major reason for this post).
  Notes
None
1 comment

Big thanks to Aaron Gertler for providing advice and reviewing/commenting on a draft of this article

Summary

Roadmap/Outline

  1. Disclaimers, epistemic status, etc.
  2. Explaining the framework
    1. Basic overview, including the four components
    2. Further theoretical contentions/characteristics
    3. Comparison with the INT framework
    4. Example usage
  3. Justifications for learning/using the framework
    1. Important characteristics
    2. General/indirect justifications
    3. Specific justifications
    4. Potential downsides
  4. Conclusion

Describing the TUILS Framework

As with things like Bayesian reasoning and the INT framework, there is a spectrum of informal and formal ways to define/use this framework. The following explanation gives a sort of middle-ground version, since it reflects what I tend to actually use in practice and because it is easier to both explain the concepts and illustrate the framework’s overall usefulness this way.

Basic overview, including the four components:

The framework consists of four collectively-exhaustive and (arguably) mutually-exclusive conceptual components: trajectory/uniqueness, implementation, linkage, and significance. Each component encapsulates a logical part of a claimed advantage or disadvantage, as will hopefully be made clearer as I go through each one (and bring them all together):

  1. Trajectory/Uniqueness. This component essentially asks “What will happen in the world without the proposed action/change (as it relates to the claimed advantage/disadvantage)?” As with the other components, answering this often involves asking common sub-questions, such as “what kinds of policies/projects are currently (or will soon/eventually be) in place that deal with this”; “what is the current trend with the problem in question”; “will this problem eventually resolve itself/what’s the expected timeline for this issue”; etc. For disadvantages in particular, a broad way of phrasing this question is “(to what extent) will this problem occur even if the plan is not put into place?” Especially when doing retrospective analysis on the pros and cons of an action, this component essentially refers to counterfactual analysis (i.e., “what would have happened if we did not take those actions?”).
  2. Implementation. When formulating and presenting a course of action, it’s common to use phrases like “we will do X” (perhaps with details like “using Y resources”, “following Z timeline”, and so on). This component, however, calls into question the underlying assumptions baked into the plan regarding implementation, essentially asking “how will the proposed action/change actually be implemented? Can we actually perform the specified task (within the given deadline, with the given budget, etc.)?” For disadvantages in particular, the questions tend to be more along the lines of “does the plan actually do what the disadvantage implies?” (e.g., “is there actually no grandfather clause?”). Ultimately, it’s important to understand implementation in concert with its close counterpart: linkage.
  3. Linkage. This essentially asks “what happens in the world where (a given implementation of) the plan takes place: does the suggested problem actually diminish (or, in the case of disadvantages, increase)?” Technically, this does not directly ask “what are the effects of the plan’s implementation,” or “to what extent does the plan fix/cause this problem,” but I find that it’s easier/more natural to at least word my questions that way so long as I remember to take into account trajectory/uniqueness. Regardless of how you phrase the questions, the analysis here ultimately evaluates "world with plan" so that one can compare with “world without plan” (assessed in the trajectory component): in theory (i.e., supposing you’ve controlled for other variables), any differences between these two worlds represents a causal effect of the plan. If the worlds are the same and the outcome supposed by the advantage/disadvantage does not materialize in either world, this means that the advantage/disadvantage lacks linkage to the plan. In contrast, if the worlds are the same and the supposed outcome does materialize in both worlds, it means that the advantage/disadvantage has problems in the trajectory segment of the analysis—or to put it more naturally, the advantage/disadvantage “lacks uniqueness” to the plan (hence why I sometimes use the term “uniqueness” instead of “trajectory”).
  4. Significance. This is where normative/moral analysis finally enters. Significance essentially asks “So what? What is the moral difference between ‘world without the action’ and ‘world with (a given implementation of) the action’?” Technically, part of this is where the debates over different moral frameworks come into play (e.g., deontology vs. utilitarianism, average utilitarianism vs. total utilitarianism), although like with other questions there is no “requirement” that you relitigate that specific issue every time. To be clear, though, I do think that this can be applied to a non-consequentialist moral framework like deontology or virtue ethics: a deontological disadvantage, for example, could take the form “inaction means we do not do X; this plan would be implemented in a way which involves doing X; doing X violates deontology which is bad.”

Putting all of these pieces together and summarizing, the framework breaks down advantages {disadvantages} as follows: Summary claim: This plan would cause X, which is good {bad}.

  1. Trajectory/uniqueness: X will not happen without this plan;
  2. Implementation: This plan involves a set of actions which can/would be implemented;
  3. Linkage: Implementation of this plan will result in X;
  4. Significance: X is a good {bad} thing.

Theoretical implications/contentions of the TUILS framework

In asserting that the four factors above are collectively exhaustive, this framework posits that every claimed advantage/disadvantage implicitly or explicitly relies on all four of these components, and any kind of challenge to a claimed advantage/disadvantage exclusively relates to one or more of these four concepts.[2] As a result, this analytical framework theoretically could be applied to any decision (i.e., not just government policy analysis), although it obviously is not always the best way to analyze choices (such as when trying to make split-second decisions based on instinct).
Further extending the points mentioned above: an important contention of this framework is that every component is necessary; similar to the idea of a zero in a multiplication equation, if any of the components is completely lacking (e.g., the implementation will completely fail), then every advantage/disadvantage that relies on that assumption will fall regardless of how accurate the other components are. In reality, the formal analysis is more complex than simplistic linear multiplication since, for example, achieving only 50% of the assumed degree of change (e.g., reduction of a pollutant) might not linearly translate to a 50% improvement (e.g., it might fall above or below a critical threshold for health effects). However, at the heuristic/informal level one can often make quick rough estimates as to the impact of, for example, a plan only being half as effective at reducing a problem as was originally assumed.

The TUILS framework vs. the INT framework

I think it is helpful to briefly compare and contrast TUILS with the INT framework:

Examples of the TUILS framework being applied

The following are some simplified examples of the framework’s usage—including the question/objection generation process—mainly for further illustration/clarification of how the framework works (rather than trying to primarily illustrate its value).

  1. Consider lobbying for some policy change in a developing country—for example, on tobacco policy. Suppose that the proposal is to fund an advocacy campaign that would push for tighter controls on cigarettes, with the primary claimed advantage being “it will (increase the likelihood of passing legislation that will) reduce the mortality caused by smoking.” To evaluate this advantage, you would likely face questions such as:

    1. Trajectory/Uniqueness: What would happen without this intervention? (Imagine for example that someone claims the campaign is likely to work because there is a “growing wave of support” for the reform: this might mean that the reform—or a slightly less strong version of the reform—already has a decent chance of passing. As part of this, it may be the case that the advocacy campaign will already receive sufficient funding.)
    2. Implementation: Do we actually have the necessary funding and can we actually meet the timeline outlined by the plan? (For example, are there any restrictions on foreign funding that have not been accounted for?)
    3. Linkage: Supposing that the plan is implemented (or, for a given implementation of the plan), what is the resulting likelihood that the desired reform will be signed into law—and subsequently, how effective will the desired reform be in reducing mortality caused by smoking (which introduces a recursion of this framework).
    4. Significance (assuming a utilitarian moral framework): How does “reducing mortality caused by smoking” translate to changes in wellbeing? If one considers the goal to simply be reducing mortality caused by smoking, that might be achieved, but it’s not guaranteed that achieving that goal will lead to an increase in wellbeing, such as is more-directly measured by a metric like QALYs. (For example, it’s possible that there are other widespread environmental problems that significantly reduce the effect of smoking mortality reduction on QALYs.)
  2. When choosing a research topic, one of the most prominent justifications is discovering and/or proliferating useful knowledge about the issue.[3] When evaluating this justification for a variety options, some of the major questions under each component would be as follows:

    1. Trajectory/Uniqueness: What aspect of this issue is currently unknown or misunderstood? If I don’t explain/discover this, will someone else do it eventually? When? How accurate do I expect their research to be?
    2. Implementation: Will I actually be able to devote sufficient time to this? Do I have the resources I need to do this research in the envisioned way?
    3. Linkage: (Under a given implementation) what is the likelihood that my research will be successful and/or to what extent?
    4. Significance: How valuable would it be to discover/explain the issue? Will the research still be relevant/actionable by the time I’m finished?
  3. When trying to evaluate the direct-impact benefits from working in a specific position (e.g., when deciding between multiple preliminary/contingent offers), some of the questions to consider would include:

    1. Trajectory/Uniqueness: If I don’t work in this field/position, what will happen? Who else will be working on this issue/in this position?[4]

    2. Implementation: What are the actual details of this position in question and can I actually get this position, or will I only end up as something lower than or adjacent to the position I have in mind?

    3. Linkage: What will happen if I receive this position? What will I be able to do within this position? Will I be an effective worker in this field?

    4. Significance: So what if I improve the work in this field by a given amount? Is it an important line of work?

Reasons to learn/remember/use

To be honest, I’ve come across plenty of frameworks or aids for decision-making and analysis (I even have a mini-book covering dozens of them): some are good, but some are sketchy at best, and more broadly there are just so many that I think it’s natural to be skeptical of “yet another framework/model.” Despite this, I think that at least two claimed characteristics seem to make it worth deeper scrutiny/attention in general:

For what it’s worth, I also personally think that the framework’s basic idea (e.g., the four overarching questions) is relatively simple/easy to learn, but I admit I am probably not the best judge of its complexity since I am already familiar with it. Still, I think it’s near the complexity level of the INT framework, which I consider to be fairly simple.

General/Indirect Justifications (e.g., reference class justifications)

Partially building on the observations above, the following points are some inferential or otherwise indirect arguments that I think are worth mentioning before I get into the more-focused/specific justifications. Generally speaking:

Specific/Narrower Justifications

Moving on from these general arguments and looking more narrowly at specific arguments (most of which are examples/instances of the above points), I would contend that this framework can help with[5]:

Potential (generic[6]) downsides

The following is a non-exhaustive list of potential downsides to learning/using this framework:

Conclusion

Ultimately, the TUILS framework only addresses part of the decision-making process and isn’t always optimal to use (especially when a basic pro-con analysis is already superfluous), but unlike many related frameworks (including the INT framework) it doesn’t have inbuilt imperfect assumptions or other oversimplifications: it simply breaks down the logical concept of pros and cons into their conceptual components.

I’ve analyzed and used the TUILS framework for years, and based on my personal experiences I would contend that even learning the basic skeleton (i.e., the main overarching question/idea for each component) can help with catching oversights and mitigating confirmation bias—even if only by helping you catch the mistakes faster than you would have otherwise. More generally though, I think that just as people in the EA community have found that the INT framework helps to normalize and semi-standardize language around the concepts of importance, neglectedness, and tractability, it seems that shared familiarity with the TUILS framework could similarly help decision-making discussions/debates.

That being said, I am still definitely open to suggestions for different component names as well as other suggestions or criticisms more generally (in fact, getting feedback is a major reason for this post).

Notes


  1. The name is just an acronym of the component names. One may notice that I could have chosen the acronym “UTILS” which may seem to fit better, but my two main concerns with this were: 1) I did not want this to be so specifically associated with utilitarianism since it (arguably) does not require using a utilitarian framework with it; 2) I thought it might come across as a bit corny or even “too convenient.” Ultimately, I am still very open to taking suggestions/feedback on both the individual components’ names as well as names for the overall framework. (If only there were some EA-aligned organization that helped people name things [EA · GW]) ↩︎

  2. It is possible to make responses such as “this disadvantage is wholly true, but we still outweigh with our two advantages”, but this is just an argument at the “impact calculus” level; it is not challenging the argument itself. Additionally, I believe that technically speaking, one could model this situation as the component of significance being expanded to weigh multiple arguments at the same time. ↩︎

  3. There may be other reasons for choosing certain research topics, such as signaling, generating interest in a topic, fulfilling the wishes of a benefactor, etc. Each of these justifications would be valid subjects of their own TUILS evaluation—although there likely will be important questions that cut across justifications (e.g., many justifications may rely on the likelihood that some information is actually discovered/proliferated). ↩︎

  4. Technically, the concept of replacement (i.e., “I would be taking the position that someone else would have taken” as opposed to “I would be an additional person working on this team”) is a separate disadvantage if the implication is that if the decision is implemented, the other person will not get the job. However, it might be easier/faster to just fold such a disadvantage into your analysis of the advantage. ↩︎

  5. In the past, when explaining this framework I have informally applied the framework to learning/using the framework—i.e., I loosely broke down the following justifications along the framework’s four components. In this post however, I decided not to do that in the main text since I figured it may be better to just explain the points in a standard/familiar way and note here that if someone wants I can go through those steps in a comment/followup. Still, I’ll preemptively say that a rough outline for the first justification for learning/using this framework (to mitigate biases/oversights) is “when and how often do I make these kinds of mistakes”, “would I be able to remember to apply this framework (and apply it correctly/not forget parts of it) at those times”, “would walking through the framework actually prompt me to recognize and correct my mistakes”, and “is the degree of mistake mitigation actually significant (e.g., how significant are the decisions in question)?” ↩︎

  6. One could say that these downsides are generically applicable to basically any decision-making framework, but I still felt I should note them if only to make it clear that I acknowledge them. ↩︎

1 comments

Comments sorted by top scores.

comment by Davidmanheim · 2021-04-08T07:42:02.016Z · EA(p) · GW(p)

Strongly agree that the focus on Implementation is critical, and can easily be missed by those only superficially acquainted with I/N/T analyses. It's also good to focus on linkage - see Pearl's amusing  / correct paper on why applying scientific knowledge to actual decisions is useless. Overall, 9/10 on content.

At the same time, I think this post would be greatly improved with editing and simplifying the arguments. (I tend to need help with the same things; structure, leaving things out, making a clear case in the introduction, etc. So I very, very often ask for editing help.) I would give the post itself an unfortunate 3/10 on clarity of presentation, which is unfortunate given what I think is the usefulness of the argument.

All that said, I upvoted this, but am unsurprised, and nonetheless disappointed, to see that someone / other people have downvoted this without saying why.