Posts

Informational Lobbying: Theory and Effectiveness 2020-07-30T22:02:15.200Z
Matt_Lerner's Shortform 2019-12-20T18:11:47.835Z
Kotlikoff et al., 'Making Carbon Taxation a Generational Win Win' 2019-11-24T16:27:50.351Z

Comments

Comment by Matt_Lerner (mattlerner) on EA cause areas are just areas where great interventions should be easier to find · 2021-07-17T20:01:31.593Z · EA · GW

While I’m skeptical about the idea that particular causes you’ve mentioned could truly end up being cost effective paths to reducing suffering, I’m sympathetic to the idea that improving the effectiveness of activity in putatively non-effective causes is potentially itself effective. What interventions do you have in mind to improve effectiveness within these domains?

Comment by Matt_Lerner (mattlerner) on EA cause areas are just areas where great interventions should be easier to find · 2021-07-17T16:53:25.352Z · EA · GW

Now that you’ve given examples, can you provide an account of how increased funding in these areas can lead to improved well-being / preserves lives or DALYs / etc in expectation? Do you expect that targeted funds could be cost-competitive with GW top charities or likewise?

Comment by Matt_Lerner (mattlerner) on Why EAs researching mainstream topics can be useful · 2021-06-14T18:40:58.695Z · EA · GW

To clarify, I'm not sure this is likely to be the best use of any individual EA's time, but I think it can still be true that it's potentially a good use of community resources, if intelligently directed.

I agree that perhaps "constitutionally" is too strong - what I mean is that EAs tend (generally) to have an interest in / awareness of these broadly meta-scientific topics.

In general, the argument I would make would be for greater attention to the possibility that mainstream causes deserve attention and more meta-level arguments for this case (like your post).

Comment by Matt_Lerner (mattlerner) on Intervention Report: Charter Cities · 2021-06-14T17:47:24.801Z · EA · GW

Thanks for this! It seems like much of the work that went into your CEA could be repurposed for explorations of other potentially growth- or governance-enhancing interventions. Since finding such an intervention would be quite high-value, and since the parameters in your CEA are quite uncertain, it seems like the value of information with respect to clarifying these parameters (and therefore the final ROI distribution) is probably very high.

Do you have a sense of what kind of research or data would help you narrow the uncertainty in the parameter inputs of your cost-effectiveness model?

Comment by Matt_Lerner (mattlerner) on Why EAs researching mainstream topics can be useful · 2021-06-14T15:17:33.384Z · EA · GW

On the face of it, it seems like researching and writing about "mainstream" topics is net positive value for EAs for the reasons you describe, although not obviously an optimal use of time relative to other competing opportunities for EAs. I've tried to work out in broad strokes how effective it might be to move money within putatively less-effective causes, and it seems to me like (for instance) the right research, done by the right person or group, really could make a meaningful difference in one of these areas.

Items 2.2 and 2.3 (in your summary) are, to me, simultaneously the riskiest and most compelling propositions to me.  Could EAs really do a better job finding the "right answers" than there are to be found in existing work? I take "neglectedness" in the ITN framework to be a heuristic that serves mainly to forestall hubris in this regard: we should think twice before assuming we know better than the experts, as we're quite likely to be wrong.

But I think there is still reason to suspect that there is value to be captured in mainstream causes. Here are a few reasons I think this might be the case.

  • "Outcome orientation" and a cost-benefit mindset are surprisingly rare, even in fields that are nominally outcomes-focused. This horse has already been beaten to death, but the mistakes, groupthink, and general confusion in many corners of epidemiology and public health during the pandemic suggests that consequences are less salient in these fields than I would have expected beforehand. Alex Tabarrok, a non-epidemiologist, seems to have gotten most things right well before the relevant domain experts simply by thinking in consequentialist terms. Zeynep Tufekci, Nate Silver, and Emily Oster are in similar positions.
     
  • Fields have their own idiosyncratic concerns and debates that eat up a lot of time and energy, IMO to the detriment of overall effectiveness. My (limited) experience in education research and tech in the developed world led me to conclude that the goals of the field are unclear and ill-defined (Are we maximizing graduation rates? College matriculation? Test scores? Are we maximizing anything at all?). Significant amounts of energy are taken up by debates and concerns about data privacy, teacher well-being and satisfaction, and other issues that are extremely important but which, ultimately, are not directly related to the (broadly defined) goals of the field. The drivers behind philanthropic funding seem, to me, to be highly undertheorized.

    I think philanthropic money in the education sector should probably go to the developing world, but it's not obvious to me that developed-world experts are squeezing out all the potential value that they could. Whether the scale of that potential value is large enough to justify improving the sector, or whether such improvements are tractable, are different questions.
     
  • There are systematic biases within disciplines, even when those fields or disciplines are full of smart, even outcomes-focused people. Though not really a cause area, David Shor has persuasively argued that Democratic political operatives are ideological at the cost of being effective. My sense is that this is also true to some degree in education.
     
  • There are fields where the research quality is just really low. The historical punching bag for this is obviously social psychology, which has been in the process of attempting to improve for a decade now. I think the experience of the replication crisis—which is ongoing—should cause us to update away from thinking that just because lots of people are working on a topic, that means that there is no marginal value to additional research. I think the marginal value can be high, especially for EAs, who are constitutionally hyper-aware of the pitfalls of bad research, have high standards of rigor, and are often quantitatively sophisticated. EAs are also relatively insistent on clarity, the lack of which seems to be a main obstacle to identifying bad research.
     
Comment by Matt_Lerner (mattlerner) on Exporting EA discussion norms · 2021-06-01T13:58:09.606Z · EA · GW

I think about this all the time. It seems like a really high-value thing to do not just for the sake of other communities but even from a strictly EA perspective— discourse norms seem to have a real impact on the outcome of decision-relevant conversations, and I have an (as-yet unjustified) sense that EA-style norms lead to better normative outcomes. I haven't tried it, but I do have a few isolated, perhaps obvious observations.

  • For me at least, it is easier to hew to EA discussion norms when they are, in fact, accepted norms. That is, assuming the best intentions of an interlocutor, explaining instead of persuading, steelmanning, etc— I find it easier to do these things when I know they're expected of me. This suggests to me that it might hard to institute such norms unilaterally.
  • EA norms don't obviously all go together. You can imagine a culture where civility is a dominant norm but where views are still expressed and argued for in a tendentious way. This would suck in a community where the shared goal is some truth-seeking enterprise, but I imagine that the more substantive EA norms around debate and discussion would actually impose a significant cost on communities where truth-seeking isn't the main goal!
  • Per the work of Robert Frank, it seems like there are probably institutional design decisions that can increase the likelihood of observing these norms. I'm not sure how much the EA Forum's designers intended this, but it seems to me like hiding low-scoring answers, allowing real names, and the existence of strong upvotes/downvotes all play a role in culture on the forum in particular.
Comment by Matt_Lerner (mattlerner) on Matt_Lerner's Shortform · 2021-05-13T16:33:48.894Z · EA · GW

I guess a more useful way to think about this for prospective funders is to move things about again. Given that you can exert c/x leverage over funds within Cause Y, then you're justified in spending c to do so provided you can find some Cause Y such that the distribution of DALYs per dollar meets the condition...

...which makes for a potentially nice rule of thumb. When assessing some Cause Y, you need only ("only") identify a plausibly best or close-to-best opportunity, as well as the median one, and work from there.

Obviously this condition holds for any distribution and any set of quintiles, but the worked example above only indicates to me that it's a plausible condition for the log-normal.

Comment by Matt_Lerner (mattlerner) on Matt_Lerner's Shortform · 2021-05-11T19:51:27.731Z · EA · GW

Under what circumstances is it potentially cost-effective to move money within low-impact causes?

This is preliminary and most likely somehow wrong.  I'd love for someone to have a look at my math and tell me if (how?) I'm on the absolute wrong track here.

Start from the assumption that there is some amount of charitable funding that is resolutely non-cause-neutral. It is dedicated to some cause area Y and cannot be budged. I'll assume for these purposes that DALYs saved per dollar is distributed log-normally within Cause Y:

I want to know how impactful it might, in general terms, be to shift money from the median funding opportunity in Cause Y to the 90th percentile opportunity. So I want the difference between the value of spending a dollar at those two points on the impact distribution.

The log-normal distribution has the following quantile function:

So the value to be gained by moving from p = 0.5 to p = 0.9 is given by

This simplifies down to

Or

Not a pretty formula, but it's easy enough to see two things which were pretty intuitive before this exercise. First, you can squeeze out more DALYs from moving money in causes where the  mean DALYs per dollar across all funding opportunities is higher, and, for a given average, moving money is higher-value where there's more variation across funding opportunities (roughly, since variance is proportional to but not precisely given by sigma). Pretty obvious so far.

Okay, what about making this money-moving exercise cost-competitive with a direct investment in an effective cause, with a benchmark of $100/DALY? For that, and for a given investment amount $x, and a value c such that an expenditure of $c causes the money in cause Y to shift from the median opportunity to the 90th-percentile one, we'd need to satisfy the following condition:

Moving things around a bit...

Which, given reasonable assumptions about the values of c and x, holds true trivially for larger means and variances across cause Y.  The catch, of course, is that means and variances of DALYs per dollar in a cause area are practically never large, let alone in a low-impact cause area. Still, the implication is that (a) if you can exert inexpensive enough leverage over the funding flows within some cause Y and/or (b) if funding opportunities within cause Y are sufficiently variable, cost-effectiveness is at least theoretically possible.

So just taking an example: Our benchmark is $100 per DALY, or 0.01 DALYs per dollar, so let's just suppose we have a low-impact Cause Y that is between three and six orders of magnitude less effective than that, with a 95% CI of [0.00000001,0.00001], or one for which you can preserve a DALY for between $100,000 and $100 million, depending on the opportunity. That gives mu = -14.97 and sigma = 1.76. Plugging those numbers into the above, we get approximately...

...suggesting, I think, that if you can get roughly 4000:1 leverage when it comes to spending money to move money, it can be cost-effective to influence funding patterns within this low-impact cause area.

There are obviously a lot of caveats here (does a true 90th percentile opportunity exist for any Cause Y?), but this is where my thinking is at right now, which is why this is in my shortform and not anywhere else.

Comment by Matt_Lerner (mattlerner) on AMA: Tom Chivers, science writer, science editor at UnHerd · 2021-03-10T20:34:38.529Z · EA · GW

What do you see as the consequentialist value of doing journalism? What are the ways in which journalists can improve the world? And do you believe these potential improvements are measurable?

Comment by Matt_Lerner (mattlerner) on Do power laws drive politics? · 2021-02-10T21:36:43.570Z · EA · GW

One thing to note here is that lots of commonly-used power law distributions have positive support. Political choices can and sometimes do have dramatically negative effects, and many of the catastrophes that EAs are concerned with are plausibly the result of those choices (like nuclear catastrophe, for instance). 

So a distribution that describes the outcomes of political choices should probably have support on the whole real line, and you wouldn't want to model choices with most simple power-law distributions.  But you might be on to something-- you might think of a hierarchical model in which there's some probability that decisions are either good or bad, and that the degree to which they are good or bad is governed by a power law distribution. That's the model I've been working with, but it seems incomplete to me.

Comment by Matt_Lerner (mattlerner) on Good altruistic decision-making as a deep basin of attraction in meme-space · 2021-01-02T21:37:18.781Z · EA · GW

I read this post with a lot of interest; it has started to seem more likely to me lately that spreading productive, resilient norms about decision-making and altruism is a more effective means of improving decisions in the long run than any set of particular institutional structures. The knock-on effects of such a phenomenon would, on a long time scale, seem to dwarf the effects of many other ostensibly effective interventions.

So I get excited about this idea. It seems promising.

But some reflection about what is commonly considered precedent for something like this makes me a little bit more skeptical.

I think we see another kind of self-correction mechanism in the belief system of science. It provides tools for recognising truth and discarding falsehood, as well as cultural impetus to do so; this leads not just to the propagation of existing scientific beliefs, but to the systematic upgrading of those beliefs; this isn't drift, but going deeper into the well of truth.

I have a sense that a large part of the success of scientific norms comes down to their utility being immediately visible. Children can conduct and repeat simple experiments (e.g. baking soda volcano); undergraduates can repeat famous projects with the same results (e.g. the double slit experiment), and even non-experimentalists can see the logic at the core of contemporary theory (e.g. in middle school geometry, or at the upper level in real analysis). What's more, the norms seem to be cemented most effectively by precisely this kind of training, and not to spread freely without direct inculcation: scientific thinking is widespread among the trained, and (anecdotally) not so common among the untrained. For many Western non-scientists, science is just another source of formal authority, not a process that derives legitimacy from its robust efficacy.

I can see a way clear to a broadening of scientific norms to include what you've characterized as "truth-seeking self-aware altruistic decision-making." But I'm having trouble imaging how it could be self-propagating. It would seem, at the very least, to require active cultivation in exactly the way that scientific norms do-- in other words, that it would require a lot of infrastructure and investment so that proto-truth-seeking-altruists can see the value of the norms. Or perhaps I am having a semantic confusion: is science self-propagating in that scientists, once cultivated, go on to cultivate others? 

Comment by Matt_Lerner (mattlerner) on Big List of Cause Candidates · 2020-12-25T22:35:05.151Z · EA · GW

I very strongly upvoted this because I think it's highly likely to produce efficiencies in conversation on the Forum, to serve as a valuable reference for newcomers to EA, and to act as a catalyst for ongoing conversation.

I would be keen to see this list take on life outside the forum as a standalone website or heavily moderated wiki, or as a page under CEA or somesuch, or at QURI.

Comment by mattlerner on [deleted post] 2020-12-16T21:47:29.837Z

I'm not sure why this is being downvoted. I don't really have an opinion on this, but it seems at least worth discussing. OP, I think this is an interesting idea.

Comment by Matt_Lerner (mattlerner) on Books / book reviews on nuclear risk, WMDs, great power war? · 2020-12-15T16:57:31.678Z · EA · GW

John Lewis Gaddis' The Cold War: A New History contains a number of useful segments about the nuclear tensions between the U.S. and the U.S.S.R., insightful descriptions of policymakers' thinking during these moments, and a consideration of counterfactual histories in which nuclear weapons might have been deployed. I found it pretty useful in terms of helping me get a picture of what decision-making looks like when the wrong decision means (potentially) the end of civilization.

Comment by Matt_Lerner (mattlerner) on Careers Questions Open Thread · 2020-12-04T18:39:38.713Z · EA · GW

How harmful is a fragmented resume? People seem to believe this isn't much of a problem for early-career professionals, but I'm 30, and my longest tenure was for two and a half years (recently shorter). I like to leave for new and interesting opportunities when I find them, but I'm starting to wonder whether I should avoid good opportunities for the sake of appearing more reliable as a potential employee.

Comment by Matt_Lerner (mattlerner) on A beginner data scientist tries her hand at biosecurity · 2020-10-24T02:55:18.699Z · EA · GW

First, congratulations. This is impressive, you should be very proud of yourself, and I hope this is the beginning of a long and fruitful data science career (or avocation) for you.
 

What is going on here?


I think the simplest explanation is that your model fit better because you trained on more data. You write that your best score was obtained by applying XGBoost to the entire feature matrix, without splitting it into train/test sets. So assuming the other teams did things the standard way, you were working with 25%-40% more data to fit the model. In a lot of settings, particularly in the case of tree-based methods (as I think XGBoost usually is), this is a recipe for overfitting. In this setting, however, it seems like the structure of the public test data was probably really close to the structure of the private test data, so the lack of validation on the public dataset paid off for you.

I think one interpretation of this is that you got lucky in that way. But I don't think that's the right takeaway. I think the right takeaway is that you kept your eye on the ball and chose the strategy that worked based on your understanding of the data structure and the available methods and you should be very satisfied.

 

Comment by Matt_Lerner (mattlerner) on Effective donation for Moria / Lesbos · 2020-10-20T18:05:32.167Z · EA · GW

I wonder if the forum  shouldn't encourage a class of post (basically like this one) that's something like "are there effective giving opportunities in X context?" Although EA is cause-neutral, there's no reason why members shouldn't take the opportunity provided by serendipity to investigate highly specific scenarios and model "virtuous EA behavior." This could be a way of making the forum friendlier to visitors like the OP, and a way for comments to introduce visitors to EA concepts in a way that's emotionally relevant.

Comment by Matt_Lerner (mattlerner) on EA's abstract moral epistemology · 2020-10-20T15:00:42.382Z · EA · GW

I also found this (ironically) abstract. There are more than enough philosophers on this board to translate this for us, but I think it might be useful to give it a shot and let somebody smarter correct the misinterpretations.

The author suggests that the "radical" part of EA is the idea that we are just as obligated to help a child drowning in a faraway pond as in a nearby one:

The morally radical suggestion is that our ability to act so as to produce value anywhere places the same moral demands on us as does our ability to produce value in our immediate practical circumstances

She notes that what she sees as the EA moral view excludes "virtue-oriented" or subjective moral positions, and lists several views (e.g. "Kantian constructivist") that are restricted if one takes what she sees as the EA moral view. She maintains that such views, which (apparently) have a long history at Oxford, have a lot to offer in the way of critique of EA.

Institutional critique

In a nutshell, EA focuses too much on what it can measure, and what it can measure are incrementalist approaches that ignores the "structural, political roots of global misery." The author says that the EA responses to this criticism (that even efforts at systemic change can be evaluated and judged effective) are fair. She says that these responses constitute a claim that the institutional critique is a criticism of how closely EA hews to its tenets, rather than of the tenets themselves. She disagrees with this claim.

Philosophical critique

This critique holds that EAs basically misunderstand what morality is-- that the point of view of the universe is not really possible. The author argues that attempting to take this perspective actively "deprives us of the very resources we need to recognise what matters morally"-- in other words, taking the abstract view eliminates moral information from our reasoning.

The author lists some of the features of the worldview underpinning the philosophical critique. Acting rightly includes:

acting in ways that are reflective of virtues such as benevolence, which aims at the well-being of others

 

acting, when appropriate, in ways reflective of the broad virtue of justice, which aims at an end—giving people what they are owed—that can conflict with the end of benevolence

She concludes:

In a case in which it is not right to improve others’ well-being, it makes no sense to say that we produce a worse result. To say this would be to pervert our grasp of the matter by importing into it an alien conception of morality ... There is here simply no room for EA-style talk of “most good.”

So in this view there are situations in which morality is more expansive than the improvement of others' well-being, and taking the abstract view eliminates these possibilities.

The philosophical-institutional critique

The author combines the philosophical and institutional critiques. The crux of this view seems to be that large-scale social problems have an ethical valence, and that it's basically impossible to understand or begin to rectify them if you take the abstract (god's eye) view, which eliminates some of this useful information:

Social phenomena are taken to be irreducibly ethical and such that we require particular modes of affective response to see them clearly ... Against this backdrop, EA’s abstract epistemological stance seems to veer toward removing entirely it from the business of social understanding.

This critique maintains that it's the methodological tools of EA ("economic modes of reasoning") that block understanding, and articulates part of the worldview behind this critique:

Underlying this charge is a very particular diagnosis of our social condition. The thought is that the great social malaise of our time is the circumstance, sometimes taken as the mark of neoliberalism, that economic modes of reasoning have overreached so that things once rightly valued in a manner immune to the logic of exchange have been instrumentalised.

In other words, the overreach of economic thinking into moral philosophy is a kind of contamination that blinds EA to important moral concerns.

Conclusion

Finally, the author contends that EA's framework constrains "available moral and political outlooks," and ties this to the lack of diversity within the movement. By excluding more subjective strains of moral theory, EA excludes the individuals who "find in these traditions the things they most need to say." In order for EA to make room for these individuals, it would need to expand its view of morality.

Comment by Matt_Lerner (mattlerner) on The Risk of Concentrating Wealth in a Single Asset · 2020-10-18T21:46:09.702Z · EA · GW

I'm curious to hear Michael's response, but also interested to hear more about why you think this. I have the opposite intuition- presumably 1910 had its fair share of moonshots which seemed crazy at the time and which turned out, in fact, to be basically crazy, which is why we haven't heard about them.

A portfolio which included Ford and Edison would have performed extremely well, but I don't know how many possible 1910 moonshot portfolios would have included them or would have weighted them significantly enough to outperform the many failed other moonshots.

Comment by Matt_Lerner (mattlerner) on Introducing LEEP: Lead Exposure Elimination Project · 2020-10-06T19:35:01.313Z · EA · GW

I'm really excited to see this!

I understand that, lead abatement itself aside, the alkalinity of the water supply seems to have an impact on lead absorption in the human body and its attendant health effects. I'm curious whether (1) this impact is significant (2) whether interventions to change the pH of water are competitive in terms of cost-effectiveness with other types of interventions and (3) whether this has been tried.

Comment by Matt_Lerner (mattlerner) on No More Pandemics: a grassroots group? · 2020-10-04T16:02:47.784Z · EA · GW

The venue of advocacy here will depend at least in part on the policies you decide are worth advocating. Even with hundreds of grassroots volunteers, it will be hard to ensure the fidelity of the message you are trying to communicate. It is hard at first blush to imagine how greater attention to pandemic preparedness could do harm, but it is not difficult that simply exhorting government to "do something" could have bad consequences.

Given the situation, it seems likely that governments preparing for future pandemics without clear guidance will prepare for a repeat of the pandemic that is already happening, rather than a different and worse one in future.

Once you select certain highly effective policy worth advocating (for example, an outbreak contingency fund), that's the stage at which to determine the venue and the tactic. I'm not a bio expert, but it's not difficult to imagine that once you identify a roster of potential policies, the most effective in expectation may involve, for example, lobbying Heathrow Airport Holdings or the Greater London Authority rather than Parliament.

Comment by Matt_Lerner (mattlerner) on Some learnings I had from forecasting in 2020 · 2020-10-04T15:34:54.809Z · EA · GW
The EA community overrates the predictive validity and epistemic superiority of forecasters/forecasting.

This seems to be true and also to be an emerging consensus (at least here on the forum).

I've only been forecasting for a few months, but it's starting to seem to me like forecasting does have quite a lot of value—as valuable training in reasoning, and as a way of enforcing a common language around discussion of possible futures. The accuracy of the predictions themselves seems secondary to the way that forecasting serves as a calibration exercise. I'd really like to see empirical work on this, but anecdotally it does feel like it has improved my own reasoning somewhat. Curious to hear your thoughts.

Comment by Matt_Lerner (mattlerner) on [Linkpost] Some Thoughts on Effective Altruism · 2020-09-20T22:38:02.820Z · EA · GW

I think scale/scope is a pretty intuitive way of thinking about problems, which is I imagine why it's part of the ITN framework. To my eye, the framework is successful because it reflects intuitive concepts like scale, so I don't see too much of a coincidence here.

If Importance is all that matters, then I would expect these critics to be very interested in existential risks, but my impression is they are not. Similarly, I would be very surprised if they were dismissive of e.g. residential recycling, or US criminal justice, as being too small a scale an issue to warrant much concern.

This is a good point. I don't see any dissonance with respect to recycling and criminal justice—recycling is (nominally) about climate change, and climate change is a big deal, so recycling is important when you ignore the degree to which it can address the problem; likewise with criminal justice. Still, you're right that my "straw activist" would probably scoff at AI risk, for example.

I guess I'd say that the way of thinking I've described doesn't imply an accurate assessment of problem scale, and since skepticism about the (relatively formal) arguments on which concerns about AI risk are based is core to the worldview, there'd be no reason for someone like this to accept that some of the more "out there" GCRs are GCRs at all.

Quite separately, there is a tendency among all activists (EAs included) to see convergence where there is none, and I think this goes a long way toward neutralizing legitimate but (to the activist) novel concerns. Anecdotally, I see this a lot—the proposition, for instance, that international development will come "along for the ride" when the U.S. gets its own racial justice house in order, or that the end of capitalism necessarily implies more effective global cooperation.

Comment by Matt_Lerner (mattlerner) on [Linkpost] Some Thoughts on Effective Altruism · 2020-09-19T04:47:33.347Z · EA · GW

This is certainly a charitable reading of the article, and you are doing the right thing by trying to read it as generously as possible. I think they are indeed making this point:

the technocratic nature of the approach itself will only very rarely result in more funds going to the type of social justice philanthropy that we support with the Guerrilla Foundation – simply because the effects of such work are less easy to measure and they are less prominent among the Western, educated elites that make up the majority of the EA movement

This criticism is more than fair. I have to agree with it and simultaneously point out that of course this is a problem that many are aware of and are actively working to change. I don't think that they're explicitly arguing for the worldview I was outlining above. This is my own perception of the motivating worldview, and I find support in the authors' explicit rejection of science and objectivity.

Comment by Matt_Lerner (mattlerner) on [Linkpost] Some Thoughts on Effective Altruism · 2020-09-18T22:58:56.069Z · EA · GW

I can get behind your initial framing, actually. It's not explicit—I don't think the authors would define themselves as people who don't believe decision under uncertainty is possible—but I think it's a core element of the view of social good professed in this article and others like it.

A huge portion of the variation in worldview between EAs and people who think somewhat differently about doing good seems to be accounted for by a different optimization strategy. EAs, of course, tend to use expected value, and prioritize causes based on probability-weighted value. But it seems like most other organizations optimize based on value conditional on success.

These people and groups select causes based only on perceived scale. They don't necessarily think that malaria and AI risk aren't important, they just make a calculation that allots equal probabilities to their chances of averting, say, 100 malarial infections and their chances of overthrowing the global capitalist system.

To me, this is not necessarily reflective of innumeracy or a lack of comfort with probability. It seems more like a really radical second- and third-order uncertainty about the value of certain kinds of reasoning— a deep-seated mistrust of numbers, science, experts, data, etc. I think the authors of the posted article lay their cards on the table in this regard:

the values of the old system: efficiency and cost-effectiveness, growth/scale, linearity, science and objectivity, individualism, and decision-making by experts/elites

These are people who associate the conventions and methods of science and rationality with their instrumental use in a system that they see as inherently unjust. As a result of that association, they're hugely skeptical about the methods themselves, and aren't able or willing to use them in decision-making.

I don't think this is logical, but I do think it is understandable. Many students, in particular American ones (though I recognize that Guerrilla is a European group) have been told repeatedly, for many years, that the central value of learning science and math lies in getting a good job in industry. I think it can be hard to escape this habituation and see scientific thinking as a tool for civilization instead of as some kind of neoliberal astrology.

Comment by Matt_Lerner (mattlerner) on evelynciara's Shortform · 2020-09-04T23:34:21.237Z · EA · GW

I think the instrumental benefits of greater equality (racial, gender, economic, etc.) are hugely undersold, particularly by those of us who like to imagine that we're somehow "above" traditional social justice concerns (including myself in this group, reluctantly and somewhat shamefully).

In this case, I think your thought is spot on and deserves a lot more exploration. I immediately thought of the claim (e.g. 1, 2) that teams with more women make better collective decisions. I haven't inspected this evidence in detail, but on an anecdotal level I am ready to believe it.

Comment by Matt_Lerner (mattlerner) on More empirical data on 'value drift' · 2020-09-03T20:55:25.728Z · EA · GW

The former! This is pretty sensitive to modeling choices-- tried a different way, I get an engagement effect of 31 percentage points (38% vs. 7% dropout).

The modeling assumption made here is that engagement level shifts the whole distribution of dropout rates, which otherwise looks the same; not sure if that's justifiable (seems like not?), but the size of the data is constraining. I'd be curious to hear what someone with more meta-analysis experience has to say about this, but one way to approximate value drift via a diversity of measurements might be to pile more proxy measurements into the model—dropout rates, engagement reductions, and whatever else you can come up with—on the basis that they are all noisy measurements of value drift.

I'd be super curious to know if the mean/median age of EA right now is a function of the people who got into it as undergrads or grad students several years ago and who have continued to be highly engaged over time. Not having been involved for that long, I have no idea whether that idea has anecdotal resonance.

Comment by Matt_Lerner (mattlerner) on More empirical data on 'value drift' · 2020-09-03T16:12:29.118Z · EA · GW
they're for really different groups at very different levels of engagement (which leads to predictably very different drop out rates).

This is the reason for doing a random effects meta-analysis in the first place: the motivating assumption is that the populations across studies are very different and so are the underlying dropout rates (e.g. differing estimates are due not just to within-study variation but also to cross-study variation of the kind you describe).

Still, it was sloppy of me to describe 23% as the true estimate above- in RE, there is no true estimate. A better takeaway is that, within the scope of the kind of variation we see across these survey populations, we'd almost certainly expect to see dropout of less than 40%, regardless of engagement level. Perhaps straining the possibilities of the sample size, I ran the analysis again with an intercept for engagement-- high engagement seems to be worth about 21 percentage points' worth of reduced dropout likelihood on the 5-year frame.

>60% persistence in the community at large seems pretty remarkable to me. I understand that you haven't been able to benchmark against similar communities, but my prior on youth movements (as I think EA qualifies) would be considerably higher. Do you have a reference class for the EA community in mind? If so, what's in it?

Comment by Matt_Lerner (mattlerner) on More empirical data on 'value drift' · 2020-09-02T22:32:10.873Z · EA · GW

FWIW, I did a quick meta-analysis in Stan of the adjusted 5-year dropout rates in your first table (for those surveys where the sample size is known). The punchline is an estimated true mean cross-study dropout rate of ~23%, with a 90% CI of roughly [5%, 41%]. For good measure, I also fit the data to a beta distribution and came up with a similar result.

I struggle with how to interpret these numbers. It's not clear to me that the community dropout rate is a good proxy for value drift (however it's defined), as in some sense it is a central hope of the community that the values will become detached from the movement -- I think we want more and more people to feel "EA-like", regardless of whether they're involved with the community. It's easy for me to imagine that people who drift out of the movement (and stop answering the survey) maintain broad alignment with EA's core values. In this sense, the "core EA community" around the Forum, CEA, 80k, etc is less of a static glob and more of a mechanism for producing people who ask certain questions about the world.

Conversely, value drift within members who are persistently engaged in the community seems to be of real import, and presumably the kind of thing that can only be tracked longitudinally, by matching EA Survey respondents across years.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-25T19:34:08.478Z · EA · GW

Though I didn't read Godwin (now on my to-do list), I encountered some useful research that seemed to point toward the idea that regulatory lobbying could be a lot more efficient than legislative lobbying. By the end of my review, I had started to think that it would have more productive to do that instead.

Since I finished, though, I've been thinking about one of the main concerns I have about regulatory lobbying. The fact that it's probably (comparatively) easy to influence regulatory agencies means that it's pretty easy to walk back any positive rule changes. This seems to happen fairly frequently, e.g. with EPA regulations.

From that standpoint, the stickiness of the status quo in the legislative context is also an advantage: when policy change succeeds legislatively, the new policy becomes part of the difficult-to-change status quo. For longermist-oriented policies, it seems like this is a major advantage over regulatory changes.

Curious to hear your thoughts.

Comment by Matt_Lerner (mattlerner) on Do research organisations make theory of change diagrams? Should they? · 2020-08-23T15:10:55.295Z · EA · GW
But maybe this push should take the form of explicitly highlighting the option of making ToC diagrams, providing some good examples, and encouraging people to try it a few times. And then hopefully, if the employees were chosen well, they'll naturally come to use them about as often as they should. 

This is probably the right course of action. Before the project I just finished, it was never really clear to me the settings in which flow chart-type diagrams made sense. As a more or less mathy type, I think I didn't give them their due. Now that I've seen them in practice, I've started making them here and there.

I think just giving employees the allowance to make diagrams instead of slideshows or reports, and cluing them into best practices (see e.g. this guide from the CDC) can go a long way. It seems like lots of staffers go down the report/slideshow rabbit hole because they want to be seen to be doing something. This results in long, unread memos, etc.

There's another benefit, too: staffers have sometimes dramatically different writing and design skills, and simple diagrams can lower the barriers to communicating ideas for employees who may not be confident in these skills. If staff members are held to a strict standard for the clarity and coherence of logic models, they can be a way of rapidly iterating ideas that would otherwise remain unheard.

Comment by Matt_Lerner (mattlerner) on Do research organisations make theory of change diagrams? Should they? · 2020-08-23T14:40:44.613Z · EA · GW

I've just finished a project working with a large American foundation (not sure I'm okay to say which, but it's in the top 10 largest). They use logic models / ToC diagrams internally as their lingua franca: everything is expressed as a diagram. I feel a little ambivalent.

On the one hand, they are a clear and expeditious way of expressing information that might otherwise be crammed into a memo no one will read. They also very clearly express causal flow in a way that other media might not, which can facilitate understanding. At the foundation I worked with, they seem to be used primarily as a way of rapidly communicating mechanisms of action (e.g. in proposed foundation grants or investments) to and between program officers, who seem extremely pressed for time.

On the other hand, I also saw reports and presentations crammed full of incredibly detailed logic models. I'm talking about pages and pages on which small-type boxes and arrows completely fill each page. I really don't think this is useful. These incredibly detailed models are not easy to understand at a glance, and they seem to sit in an unhappy middle ground: by being complicated, they challenge comprehension, but by being simplifications, they occlude important details relevant to the mechanism being described.

I got the impression that because the order had come down from on high to put everything in a logic model, it was being done even in contexts where these models made no sense. I worried that the focus on logic models encourages only a logic model - level understanding of the world, while simultaneously eating up huge amounts of foundation time creating diagrams that few will look at or understand.

However, I am still a convert. I think theory of change / logic models do have a lot of value, but I think they need to be used sparingly and kept small. I'd make some kind of a rule: no more than twenty boxes in a model, or something like that.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-18T22:32:32.659Z · EA · GW

1) Sounds good to me! We can connect about it over DM.

2) Your reading is right. A priori, a positive correlation means lower cost-effectiveness in expectation. However, I'm not sure if it means anything generally for the median cost-effectiveness (which I tried to work with in my existing CEA), irrespective of the other model parameters. And in my existing setup, if worlds of high spending and high success are more likely co-occur, and worlds with low spending and low success are more likely to co-occur, then I believe the distribution of their product would have been more dispersed, since there would be more values at the extremes (high/high and low/low) then there would be if they were independent. But I'm pretty convinced now that a better approach would have been, as you've suggested, to do separate CEAs conditional on various assumed interventions. Rather than change the parameters of independent distributions as I did in the posted analysis, the true next step is probably to re-model under varying assumptions about the covariance of the different variables.

3) I have a different sense of this, but not an overwhelmingly different sense, and I'm going to think about it some more.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-17T14:50:34.270Z · EA · GW

After your comments and @jackva's, I actually struck this conclusion. I was trying to make a more modest statement that upon reflection (thanks to you) is (1) not such a valuable claim and (2) not well-supported enough to have >50% confidence in. It's true Baumgartner don't find that money doesn't matter; my initial (now disavowed) read was that if resources mattered independent of deployment strategy, then we'd expect to see a much stronger correlation even in the observational context. I sort of think that this observation holds true even given the passage you've cited, but it's definitely not a top-level extract from the lit review and definitely needs a considerably more robust defense than I am prepared to muster.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-12T20:22:06.910Z · EA · GW

Points all well-taken. I'd love to share with FP's journal club, though I hasten to add that I'm still making edits and modifications based on your feedback, @smclare's, and others.

With respect to uncertainty in the CE calculation, my thinking was (am I making a dumb mistake here?) that because

and , then . So if covariance is nonzero, then (I think?) the variance of the product of two correlated random variables should be bigger than in the uncorrelated counterfactual.

To me, the main value of the CE model was in the sensitivity analysis - working through it really helped me think about what "effective lobbying" would have to be able to do, and where the utility would lie in doing so. I think if it doesn't serve this purpose for the reader, then I agree this document would have been better off without the model altogether.

Thanks for your thoughts on money in politics. Vis (1) I have to think more about this, but I do definitely view the topic a little differently. For instance, it's not obvious to me that economic arguments and political representation do the necessary work of regulatory capture. Boeing is in Washington and Northrop Grumman is in Virginia. It seems clear that the representatives of the relevant districts are prepared to argue for earmarks that will benefit their constituents... but these companies are still in direct competition, and it seems like there's still strategic benefit to each in getting the rest of Congress on their side. I might misunderstand- maybe we're reaching the limits of asynchronous discussion on this topic.

Vis (2), the "inside view" I was talking about was actually yours, as someone who thinks about this professionally- so thank you for your thoughts!

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-11T16:54:38.889Z · EA · GW

I'm replying again here to note that I've struck the salience point from my conclusions. I've noted why up top. I now have a lot of uncertainty about whether this is the case or not, and don't stand by my suggestion that salience is a good guide to resource allocation.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-09T18:06:44.756Z · EA · GW

Thanks for your response!

With respect to your first point, I'm considering striking this conclusion upon reflection - see my discussion with @jackva elsewhere in this thread. In any case, my confidence level here is certainly too high given the evidence, and I really appreciate your close attention to this.

With respect to your second point, I don't mean to imply that the lack of organized opposition is the only thing that justifies lobbying expenditure, and think my wording is sloppy here as well. I used "lack of an organized opposition" to refer broadly to oppositions that are simply doing less of the (ostensibly) effective things — lower "organizational strength" as in Caldeira and Wright (1998), number of groups, as in Wright (1990), or simply lower relative expenditure, as in Ludema, Mayda, and Mishra (2018).

The evidence in Baumgartner et al that you reference about the apparent association between lack of countermobilization and success is also related to @jackva's concern about my underemphasis on potential lobbying equilibria here. On the one hand, I think this is clearly evidence in favor of the hypothesis that there is some efficiency in the market for lobbying- perhaps most lobbyists have a good idea of which efforts succeed, and don't bother to countermobilize against less sophisticated opposition. On the other hand, lobbying is a sequential game, and, since the base rate for policy enactment is so low to start with, it makes sense that opposition wouldn't appear until there's a more significant threat.

EDIT: I've actually struck the first bit, with a note. I wanted to add one more thing, which is that I don't know how much you've adjusted your prior on lobbying, but I wouldn't say this has made me "optimistic" about lobbying. The core thing I've come away with is that lobbying for policy change is extraordinarily unlikely to succeed, but that marginal changes to increase the probability of success are (1) plausible, based on the research and (2) potentially cost-effective, based on the high value of some policies.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-08T04:00:06.705Z · EA · GW

I like this spreadsheet idea and think I may kick it off (if you haven't already done so!)

I took the project on because I got interested in this topic, went looking for this, couldn't find it, and decided to make it so that it might be useful to others. I wasn't feeling very useful in my day job, so it was easy to stay motivated to spend time on this for a while. I tend to be most interested in generalizable or flexible approaches to improving welfare across different domains, and this seemed like it might be one of those.

Some areas I'm thinking about exploring. These are pretty rough thoughts:

  • Some more exploration of strategies for ameliorating child abuse in light of the well-known ACES Study. GiveWell and RandomEA have both explored Nurse-Family Partnerships. This problem is just so huge in terms of people affected (and in terms of second-order effects) that I think it's worth exploring a lot more. I'm particularly interested in focusing on child sexual abuse in particular.
  • Aggregating potentially cost-effective avenues to improve institutional performance. I'm curious about thinking at a higher level of abstraction than institutional decision-making. It seems worthwhile to put together the existing cross-disciplinary evidence on the question: what steps outside of those explicitly focusing on rationality and decision-making can companies/nonprofits/government agencies take to increase the probability that they make good decisions? A good example of one such step is in the apparent evidence that intellectually diverse teams make better decisions.
  • Long-term cost-effectiveness of stress reduction for pregnant women (with potential effects of infant mortality, maternal health, and long-term outcomes like brain development and violence).
  • Review of recent innovations that seem to like they might have potential for expediting scientific progress (like grant lotteries)
Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-08-08T02:59:51.920Z · EA · GW

Hello and thank you for your response!

Your criticism of the cost-effectiveness model is fair. Thematically, I guess it does contradict the spirit of my prior analysis in that it avoids the concerns of strategic choice. I was actively trying to be as general as possible, and actively trying to err on the side of greater uncertainty by not including any assumptions about correlatedness, though it occurs to me now that making such an assumption (e.g. a correlation between expenditure and likelihood of success) would actually have increased the variance of the final estimate, which would have been more in line with my goals. When I have time, I may comment here with an updated CEA.

I also agree that the only useful way to do this analysis is, as you've described, with a suite of models for different scenarios. I don't have a defense for not having done this beyond my own capacity constraints, though I hope it's more useful to have included the flawed model than not to have one at all (what do you think?).

I also think that the conclusion which, I believe, mostly draws from Baumgaertner " (80%) Well-resourced interest groups are no more or less likely to achieve policy success, in general, than their less well-resourced opponents." is quite surprising and I would be curious to find out why you think that / in how far you trust that conclusion.


Thanks for this, in particular. I think your surprise stems from a lack of clarity on my part. The reason I have high confidence in this conclusion is that it's a much weaker claim than it might seem. It does stem primarily from Baumgartner et al and from Burstein and Linton (2002). The claim here is that resource-rich groups are no more or less likely to get what they want--holding all else equal, including absolute expenditure and the spending differential between groups and their opponents.

There are three types of claim that are closely related:
1) Groups that spend more relative to their opposition on a given policy are likelier to win
2) Groups that spend more in absolute terms are likelier to win
3) Groups that have more money to spend are likelier to win

So I found fairly consistent evidence for (1), some evidence for (2), and no real evidence for (3). It's not obvious to me that (3) should be the case irrespective of (1): why would resource-rich groups succeed in lobbying if they deploy those resources poorly? It seems like the success of resource-rich groups is dependent upon (1), and that (3) should not be true when in isolation, unmediated by (1). Although Baumgartner et al conduct an observational study, the size of their (to me, convincingly representative) sample to me suggests that if such an effect exists, it should be observable as a correlation in their analysis. The association they observe is pretty small.

I have to say, though, that in writing this comment, my confidence in this conclusion has eased up a bit, so I'm curious to hear your response. I also think that since Baumgartner et al do find a small effect, I probably overstate the case here.

Baumgartner et al offer a theoretical take on this: "...organizations rarely lobby alone. Citizen groups, like others, typically participate in policy debates alongside other actors of many types who share the same goals. For every citizen group opposing an action by a given industrial group, for example, there may also be an ally coming from a competing industry with which the group can join forces" (p.12). So it's important to recognize that the finding here is about individual parties, not "sides" or coalitions advocating a given policy.

Finally, I'm curious to hear your take on the two potential money-in-politics explanations you mentioned. I've never found (1) particularly convincing—it's not clear to me that firms and their employees have the same interests, or that (if they do) the marginal value of regulatory capture isn't still high. But I agree that I underemphasized (2) and think it would be useful to have in this thread the "inside view" on lobbying equilibria from someone who works in the field.

Comment by Matt_Lerner (mattlerner) on Informational Lobbying: Theory and Effectiveness · 2020-07-31T19:27:54.309Z · EA · GW

Thanks for your response!

(1) I spent something like 100 hours on this over the course of several months. I think I could have cut this by something like 30-40% if I'd been a little bit more attentive to the scope of the research. I decided on the scope (assessing the effectiveness of national-level legislative lobbying in the U.S.) at the beginning of the project, but I repeatedly wound up off track, pursuing lines of research outside of what I'd decided to focus on. I also spent a good chunk of time on the GitHub repo with the setup for analyzing lobbying data, which wasn't directly related to the lit review but which I felt served the goal of presenting this as a foundation for further research.

If I had 40 more hours, I'd intentionally pursue an expanded scope. In particular, I'd want to fully review the research on lobbying of (a) regulatory agencies and (b) state and local governments. I explicitly excluded studies along those lines, some of which were very interesting.

(2) Thanks for asking for clarification on this. Baumgartner et al mean that it takes a long time for policy change to be observed on any given issue. After starting to pursue a policy goal, lobbyists are more likely to see success after four years than after two.

Baumgartner et al include a chapter that is mostly critical of the incrementalist idea of policy change, which they trace to Charles Lindblom's 1959 article The Science of "Muddling Through". Incrementalism is tied to Herbert Simon's idea of "bounded rationality." Broadly, the incrementalist idea is that policymakers face a broad universe of possible policy options, and in order to reduce the landscape to a manageable set, they choose from only the most available options, e.g. those closest to the status quo: "incremental" changes.

Frank Baumgartner, with Bryan Jones, is now well-known for their theory of "punctuated equilibrium." This is a partial alternative to incrementalism which uses the analogy of friction to understand policy change. Basically: the pressure builds on an issue over a period of time, during which no change occurs. After the pressure is overwhelming, policy shifts in a major way.

I say that punctuated equilibrium is a "partial" alternative because Baumgartner and Jones actually collected data that seems to demonstrate that policy change follows a steeply peeked, fat-tailed distribution. Their overall takeaway is that very small changes are overwhelmingly common, but moderate changes are relatively uncommon, and very large changes are surprisingly common. To come back to your question, Baumgartner et al might say that although most policy change is incremental—like year-to-year changes in agency budgets—meaningful policy change happens in a big way, all of a sudden.

(3) I agree with you. I think some of my suggested policies are not likely to be those most effectively advocated for, and I included them just to give a flavor of the types of things we might care about lobbying for. Coming up with more practicable ideas is, I think, a much bigger, much longer-term project.

I also think that although lobbying for the status quo is more effective all other things being equal, it may not be the best use of EA resources to focus exclusively on that side of things. That's because (per the counteractive lobbying theory) on many issues there is are latent interests that will arise to lobby against harmful proposals. It's hard to identify beforehand which proposals will stimulate this opposition, so there's a lot of prior uncertainty as to whether funding opposition to policy change is marginally useful in expectation.

(4) There are a lot of takes on the Tullock paradox, but I'll present two broad possible explanations.

  • Explanation A: Lobbying is basically ineffective, and the reason we don't see more lobbying is that most organizations recognize its ineffectiveness.
  • Explanation B: Lobbying is highly effective, and the reason we don't see more lobbying is that relatively small expenditures can exert enormous amounts of leverage.

Given the evidence here, I'm starting to be a lot more inclined toward Explanation B. I think it's demonstrably not the case, as you have noted with respect to the Clean Air Task Force, that organizations that lobby are wasting their money. For both altruistic and self-interested interest groups, the rewards to be captured are very large, and they make it worth the risk of wasting money. Alexander, Scholz, and Mazza (2009), for example, find a 22,000% return on investment.

If Explanation B holds, then the question is really just why the market for policy isn't efficient. Why hasn't the price of lobbying been bid up to the value of the rewards to be captured? I think it seems likely that this is down to multiple layers of information asymmetry (between legislators and their staffs, between these staffers and lobbyists, between lobbyists and their clients, etc.), which create multiple layers of uncertainty and drive the expected value of lobbying down from the standpoint of those in a position to purchase it.

I agree with you that a normal distribution is probably not the best choice to model the expected incremental change in probability. I felt like, given my CI for this figure and my sense that values closer to 0% and values closer to 5% were each less likely than values in the middle of that range, this served my purposes here - but please take my code and modify as you see fit!

Perhaps we want to start with a low prior chance of policy success, and then update way up or down based on which policy we're working on. Do you think we'd be able to identify highly-likely policies in practice?

I don't know. I think it's worth investigating. It seems like, given an already-existing basket of policies we'd be interested in advocating for, we can make lobbying more cost-effective just by allocating more resources to (e.g.) issues that are less salient to the public.

I have a sense that lobbyists, do, in fact, do something like what you're describing, and that this is part of the resolution to the Tullock paradox. Money spent on lobbying is not spent all at once: lobbyists can make an effort, check their results, report to their clients, and identify whether or not they're likely to meet with success in continued expenditure. If lobbying expenditure on a given topic seems unlikely to make a difference, then it can just stop. I wasn't able to find anything on how this process actually works, so the next step in this research is to actually talk to some lobbyists.

(5)

I think perhaps something that's missing here is a discussion of incentives within the civil service or bureaucracy

I agree with this too. I'd love for an EA with a public choice background to tackle this topic. I didn't consider it as part of my scope, but I do want to note something:

A policy proposal like taking ICBMs off hair-trigger alert just seems so obvious, so good, and so easy that I think there must be some illegible institutional factors within the decision-making structure stopping it from happening.

I think this is probably true in many if not most cases of yet-to-be-implemented policy changes that are obvious, good, and easy. It is probably true in this case. But I want to warn against concluding that, because some obvious, good, and easy policy change has not been implemented, that means that there is some illegible institutional factor that is stopping it from happening. It could just be that no one has been pushing for it. In EA terms, it's an important and tractable policy change that's neglected by the policy community. Given what I know about the policy community, it's not at all difficult for me to imagine that such policies exist.

Comment by Matt_Lerner (mattlerner) on Sample size and clustering advice needed · 2020-07-30T17:56:45.691Z · EA · GW

I refer you to Sindy's comment (she is actually an expert) but I want to note and verify that it sounds as if you may not actually be thinking of collecting individual-level data, and that you're thinking of making observations at the village level (e.g. what % of people in this village wear masks?). So it's not just the case that you wouldn't have enough clusters to make a statistical claim, but you may actually be talking about doing an experiment in which the units are villages... so n = 6 to 12. Then of course you'd have considerable error in the village-level estimate, and uncertainty about the representativeness about the sample within each village. I agree with Sindy that you probably don't want an RCT here.

Comment by Matt_Lerner (mattlerner) on Sample size and clustering advice needed · 2020-07-29T16:14:24.547Z · EA · GW

If you don't already have it, I would strongly recommend getting a copy of Gerber & Green's Field Experiments. I would also very strongly recommend that you (or EA Cameroon) engage an experimental methodology expert for this project, rather than pose the question on the forum (I am not such an expert).

It is very difficult to address all of these questions in a broad way, since the answers depend on:

  • The smallest effect size you would hope to observe
  • Your available resources
  • The population within each cluster
  • The total population
  • Your analysis methodology

I'm a little confused about the setup. You say that there are 6 groups— so how would it be possible to have "6 intervention + 3 non-intervention?" Sorry if I'm misunderstanding.

In general, and particularly in this context, it makes sense to split your clusters evenly between treatment and control. This is the setup that minimizes the standard error of the difference between groups. When the variance is larger, smaller effect sizes are difficult to detect. The smaller the number of clusters in your control group, for example, the larger the effect size that you would have to detect in order to make a statistically defensible claim.

With such a small number of clusters, effect sizes would have to be very large in order to be statistically distinguishable from zero. If indeed 50% of the population in these groups is already masked, 6 clusters may not be enough to see an effect.

Can we get some clarification on some of your questions? Particularly:

How important, in terms of statistical power is to include all clusters

If you have only 6 to choose from, then the answer is very important. But I'm not sure this is the sense in which you mean this.

How many persons should be observed at each place?

My inclination here is to say "as many as possible." But this is constrained by your resources and your method of observation. Can you say more about the data collection plan?

Comment by Matt_Lerner (mattlerner) on Nathan Young's Shortform · 2020-07-23T14:42:06.207Z · EA · GW

I also thought this when I first read that sentence on the site, but I find it difficult (as I'm sure its original author does) to communicate its meaning in a subtler way. I like your proposed changes, but to me the contrast presented in that sentence is the most salient part of EA. To me, the thought is something like this:

"Doing good feels good, and for that reason, when we think about doing charity, we tend to use good feeling as a guide for judging how good our act is. That's pretty normal, but have you considered that we can use evidence and analysis to make judgments about charity?"

The problem IMHO is that without the contrast, the sentiment doesn't land. No one, in general, disagrees in principle with the use of evidence and careful analysis: it's only in contrast with the way things are typically done that the EA argument is convincing.

Comment by Matt_Lerner (mattlerner) on The EA movement is neglecting physical goods · 2020-06-18T20:43:35.346Z · EA · GW

I don't work in physical goods (I'm a data scientist) but I am definitely interested in leveling up my skillset in this way. I'm probably only available for 3 to 4 hours a week to start, but that will probably change soon.

Thanks for making this post! This is an interesting observation.

Comment by Matt_Lerner (mattlerner) on HLI’s Mental Health Programme Evaluation Project - Update on the First Round of Evaluation · 2020-06-12T06:54:34.343Z · EA · GW

Thank you for doing this work! I really admire the rigor of this process. I'm really curious to hear how this work is received by (1) other evaluation orgs and (2) mental health experts. Have you received any such feedback so far? Has it been easy to explain? Have you had to defend any particular aspect of it in conversations with outsiders?

I do have one piece of feedback. You have included a data visualization here that, if you'll forgive me for saying so, is trying to tell a story without seeming to care about the listener. There is simply too much going on in the viz for it to be useful.

I think a visualization can be extremely useful here in communicating various aspects of your process and its results, but cramming all of this information into a single pane makes the chart essentially unreadable; there are too many axes that the viewer needs to understand simultaneously.

I'm not sure exactly what you wanted to highlight in the visualization, but if you want to demonstrate the simple correlation between mechanical and intuitive estimates, a simple scatterplot will do, without the extra colors and shapes. On the other hand, if that extra information is substantive, it should really be in separate panes for the sake of comprehensibility. Here's a quick example with your data (direct link to a larger version here):

I don't think this is the best possible version of this chart (I'd guess it's too wide, and opinions differ as to whether all axes should start at 0), but it's an example of how you might tell multiple stories in a slightly more readable way. The linear trend is visible in each plot, it's easier to make out the screening sizes, and I've outlined the axes delineating the four quadrants of each pane in order to highlight the fact that mostly top-scoring programmes on both measures were included in Round 2.

Feel free to take this with as much salt as necessary. I'm working from my own experience, which is that communicating data has tended to take just as much work on the communication as it does on the data.

Comment by Matt_Lerner (mattlerner) on EA Forum Prize: Winners for April 2020 · 2020-06-08T23:09:27.587Z · EA · GW
while some users reported finding the Prize valuable or motivating, that number wasn’t quite as high as I had been hoping for

It seems like the instrumental thing here is whether users who won prizes found them motivating. Most users will not write prize-winning posts, but if the users who did were at least partially motivated by the prospect of winning one, then the world with the prize is almost certainly better than the counterfactual. More generally, if users who wrote original posts were likelier to endorse the prize than users in general, that is some indication that the prize is somewhat effective. Did you have enough data to determine whether either of these situations obtain?

Comment by Matt_Lerner (mattlerner) on I Want To Do Good - an EA puppet mini-musical! · 2020-05-21T16:25:35.642Z · EA · GW

I don't have anything to say except that I loved this, and I'm really happy somebody is starting to present a warmer and fuzzier side of EA.

Comment by Matt_Lerner (mattlerner) on Matt_Lerner's Shortform · 2020-05-01T17:13:31.724Z · EA · GW

In general, I'm skeptical about software solutionism, but I wonder if there's a need/appetite for group decision-making tools. While it's unclear exactly what works for helping groups make decisions, it does seem like a structured format could provide value to lots of organizations. Moreover, tools like this could provide valuable information about what works (and doesn't).

Comment by Matt_Lerner (mattlerner) on Matt_Lerner's Shortform · 2020-04-22T18:21:37.428Z · EA · GW

Proportional representation

Comment by Matt_Lerner (mattlerner) on Matt_Lerner's Shortform · 2020-04-13T21:16:51.339Z · EA · GW

School closures

Workplace closures


The usual caveats apply here: cross-country comparisons are often BS, correlation is not causation, I'm presenting smoothed densities instead of (jagged) histograms, etc, etc...

I've combined data on electoral system design and covid response to start thinking about the possible relationships between electoral system and crisis response. Here's some initial stuff: the gap, in days, between first confirmed cases and first school and workplace closures. Note that n= ~80 for these two datasets, pending some cleaning and hopefully a fuller merge between the different datasets.

To me, the potentially interesting thing here is the apparently lower variability of PR government responses. But I think there's a 75% chance that this is an illusion... there are many more PR governments than others in the dataset, and this may just be an instance of variability decreasing with sample size.

If there's an appetite here for more like this, I'll try and flesh out the analysis with some more instructive stuff, with the predictable criticisms either dismissed or validated.