Posts

Morality vs related concepts 2020-02-10T08:02:10.570Z · score: 12 (8 votes)
Differential progress / intellectual progress / technological development 2020-02-07T15:38:13.544Z · score: 26 (15 votes)
What are information hazards? 2020-02-05T20:50:25.882Z · score: 9 (9 votes)
Four components of strategy research 2020-01-30T19:08:37.244Z · score: 16 (12 votes)
When to post here, vs to LessWrong, vs to both? 2020-01-27T09:31:37.099Z · score: 12 (6 votes)
Potential downsides of using explicit probabilities 2020-01-20T02:14:22.150Z · score: 23 (13 votes)
[Link] Charity Election 2020-01-19T08:02:09.114Z · score: 8 (5 votes)
Making decisions when both morally and empirically uncertain 2020-01-02T07:08:26.681Z · score: 11 (5 votes)
Making decisions under moral uncertainty 2020-01-01T13:02:19.511Z · score: 33 (12 votes)
MichaelA's Shortform 2019-12-22T05:35:17.473Z · score: 2 (1 votes)
Are there other events in the UK before/after EAG London? 2019-08-11T06:38:12.163Z · score: 9 (7 votes)

Comments

Comment by michaela on MichaelA's Shortform · 2020-02-20T19:14:19.153Z · score: 1 (1 votes) · EA · GW

Some concepts/posts/papers I find myself often wanting to direct people to

https://forum.effectivealtruism.org/posts/omoZDu8ScNbot6kXS/beware-surprising-and-suspicious-convergence

https://www.lesswrong.com/posts/oMYeJrQmCeoY5sEzg/hedge-drift-and-advanced-motte-and-bailey

https://forum.effectivealtruism.org/posts/voDm6e6y4KHAPJeJX/act-utilitarianism-criterion-of-rightness-vs-decision

http://gcrinstitute.org/papers/trajectories.pdf

(Will likely be expanded as I find and remember more)

Comment by michaela on The Web of Prevention · 2020-02-19T17:02:20.423Z · score: 2 (2 votes) · EA · GW

Also related is the recent (very interesting) paper using that same term (linkpost).

(Interestingly, I don't recall the paper mentioning getting the term from computer security, and, skimming it again now, I indeed can't see them mention that. In fact, they only seem to say "defence in depth" once in the paper.

I wonder if they got the term from computer security and forgot they'd done so, if they got it from computer security but thought it wasn't worth mentioning, or if the term has now become fairly common outside of computer security, but with the same basic meaning, rather than the somewhat different military meaning. Not really an important question, though.)

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-19T14:17:17.430Z · score: 2 (2 votes) · EA · GW

I've gone with adding a footnote that links to this comment thread. Probably would've baked this explanation in if I'd had it initially, but I now couldn't quickly find a neat, concise way to add it.

Thanks again for prompting the thinking, though!

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-19T14:07:33.071Z · score: 1 (1 votes) · EA · GW

I've updated this post since initial publication, particularly to further discuss the terms "progress" vs "development" (and their connotations) and to add "Relationship to all "good actions"". This was mostly a result of useful comments on the post that prompted me to think some things through further, and some offline discussions.

In the interests of over-the-top transparency, the original version of the post can be found here.

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-19T08:51:16.542Z · score: 1 (1 votes) · EA · GW

On a different post of mine on LW, which quoted this one, Pattern commented:

"What we do have the power to affect (to what extent depends on how we define “we”) is the rate of development of various technologies and potentially the sequence in which feasible technologies are developed and implemented. Our focus should be on what I want to call differential technological development: trying to retard the implementation of dangerous technologies and accelerate implementation of beneficial technologies, especially those that ameliorate the hazards posed by other technologies."
An idea that seems as good and obvious as utilitarianism.

I agree that the idea seems good and obvious, in a sense. But beware hindsight bias and the difficulty of locating the hypothesis. I.e., something that seems obvious (once you hear it) can be very well worth saying. I think it's apparent that most of the people making decisions about technological development (funders, universities, scientists, politicians, etc.) are not thinking in terms of the principle of differential technological development.

Sometimes they seem to sort-of approximate the principle, in effect, but on closer inspection the principle would still offer them value (in my view).

E.g., concerns are raised about certain biological research with "dual use" potential, such as gain of function research, and people do call for some of that to be avoided, or done carefully, or the results released less widely. But even then, the conversation seems to focus almost entirely on whether this research is net beneficial, even slightly, rather than simultaneously also asking "Hey, what if we didn't just try to avoid increasing risks, but also tried to direct more resources to decreasing risks?" Rob Wiblin made a relevant point (and then immediate self-counterpoint, as is his wont) on the latest 80k episode:

If you really can’t tell the sign, if you’re just super unconfident about it, then it doesn’t seem like it’s probably a top priority project. If you’re just unsure whether this is good or bad for the world, I don’t know, why don’t you find something that’s good? That you’re confident is good. I suppose you’d be like, “Well, it’s a 55-45 scenario, but the 55 would be so valuable.” I don’t know.

Having said that, I feel I should also re-emphasise that the principle is not a blanket argument against technological development; it's more like highlighting and questioning a blanket assumption often implicitly made the other direction. As Bostrom writes in a later paper:

Technology policy should not unquestioningly assume that all technological progress is beneficial, or that complete scientific openness is always best, or that the world has the capacity to manage any potential downside of a technology after it is invented. [emphasis added]

Pattern's comment goes on to say:

But what if these things come in cycles? Technology A may be both positive and negative, but technology B which negates its harms is based on A. Slowing down tech development seems good before A arrives, but bad after. (This scenario implicitly requires that the poison has to be invented before the cure.)

I think this is true. I think it could be fit into the differential technological development framework, as we could say that Technology A, which appears "in itself" risk-increasing, is at least less so than we thought, and is perhaps risk-reducing on net, if we also consider how it facilitates the development of Technology B. But that's not obvious or highlighted in the original formulation of the differential technological development principle.

Justin, also of Convergence, recently wrote a post something very relevant to this point, which you may be interested in.

Comment by michaela on Thoughts on The Weapon of Openness · 2020-02-15T10:37:46.000Z · score: 2 (2 votes) · EA · GW

My own past experience as a teacher suggests (weakly and somewhat tangentially) that there's truth to both sides of this debate.

Remarkably, and I think quite appallingly, I and most other teachers at my school could mostly operate in secret, in the sense that we were hardly ever observed by anyone except our students (who probably wouldn't tell our superiors anything less than extreme misconduct). I do think that this allowed increased laziness, distorting results to look good (e.g., "teaching to the test" in bad ways, or marking far too leniently[1]), and semi-misconduct (e.g., large scary male teachers standing over and yelling angrily at 13 year olds). This seems to tangentially support the idea that "secrecy" increases "corruption".

On the other hand, the school, and curriculum more broadly, also had some quite pointless or counterproductive policies. Being able to "operate in secret" meant that I could ditch the policies that were stupid, not waste time "ticking boxes", and instead do what was "really right".

But again, the caveat should be added that it's quite possible that the school/curriculum was right and I was wrong, and thus I would've been better off being put under the floodlights and forced to conform. I tried to bear this sort of epistemic humility in mind, and therefore "go my own way" only relatively rarely, when I thought I had particularly good reasons for doing so.

This all also makes me think that the pros and cons of secrecy will probably vary between individuals and organisations, and in part based on something like how "conscientious" or "value aligned" the individual is. In the extreme, a highly conscientious and altruistic person with excellent morals and epistemics to begin with might thrive if able to operate somewhat "secretly", as they are then freed to optimise for what really matters. (Though other problems could of course occur.) Conversely, for someone with a more normal level of conscientiousness, self-interest, flawed beliefs about what's right, and flawed beliefs about the world, openness may free them instead to act self-interestedly, corruptly, or based on what they think is right but is actually worse than what others would've told them to do.

Comment by michaela on Thoughts on The Weapon of Openness · 2020-02-15T10:29:23.557Z · score: 2 (2 votes) · EA · GW

Thanks for this post.

By screening not only the decisions but the decision-making progress from outside scrutiny, secrecy greatly reduces the incentive for decision-makers to make decisions that could be justified to outside scrutinisers. Given the well-known general tendency of humans to respond to selfish incentives, the result is unsurprising: greatly increased toleration of waste, delay and other inefficiencies, up to and including outright corruption in the narrow sense, when these inefficiencies make the lives of decision-makers or those they favour easier, or increase their status (e.g. by increasing their budget)

Relatedly, in a reply to Gregory Lewis you write:

All else equal, I would expect a secret organisation to have worse epistemics and be more prone to corruption than an open one, both of which would impair its ability to pursue its goals. Do you disagree?

I don't think I know enough about this to clearly disagree or agree, but I've seen some arguments that I think would push against your claims, if the arguments are sound. (I'm not sure the arguments are sound, but I'll describe them anyway.)

As you say, "secrecy greatly reduces the incentive for decision-makers to make decisions that could be justified to outside scrutinisers." The arguments I have in mind could see that as a good thing. It could be argued that this frees organisations/individuals to optimise for what really matters, rather than for acting in ways they could easily defend after the fact. (See here for Habryka's discussion of similar arguments, focusing on the idea of "legibility" - though I should note that he seems to not necessarily or wholeheartedly endorse these arguments.)

For example, it is often claimed that confidentiality/secrecy in cabinet or executive meetings is very important so that people will actually share their thoughts openly, rather than worrying about how their statements might be interpreted, taken out of context, used against them, etc. after the fact.

For another example, I somewhere saw it claimed that things like credentials, prestige of one's university, etc. are overly emphasised in hiring processes partly because people in charge of hiring aren't just optimising for the best candidate, but the candidate they can best justify hiring if things do turn out badly. There may be many "bets" they think would be better in expectation, but if they turned out poorly the hirer would struggle to justify their decision to their superiors, whereas if the person from Harvard turned out badly the hirer can claim all the evidence looked good before the fact. (I have no idea if this is true; it's just a claim I've seen.)

In other words, in response to "Given the well-known general tendency of humans to respond to selfish incentives", these arguments might highlight that there's also a tendency for at least some people to truly want to do what they believe is "right", but to be restrained by incentives of a "justifying" or "bureaucratic" type. So "secrecy" (of a sort) could, perhaps, sometimes allow individuals/organisations to behave more effectively and have better epistemics, rather than optimising for the wrong things, hiding their real views, etc.

But again, these are just possible arguments. I don't know if I actually agree with them, and I think more empirical evidence would be great. I think there's also two particular reasons for tentative suspicion of such arguments:

  • They could be rationalisations of secrecy that is actually in the organisation/individual's interest for other reasons
  • There are often reasons why people should "tick the boxes" and optimise what's "justifiable"/legible/demanded rather than "what they believe is right". This can happen when individuals are wrong about what's right, and their organisations or superiors do know better, and did put those tick-boxes there for a good reason.
Comment by michaela on Thoughts on The Weapon of Openness · 2020-02-15T09:18:19.556Z · score: 5 (4 votes) · EA · GW
"I'm not persuaded, although this is mainly owed to the common challenge that noting considerations 'for' or 'against' in principle does not give a lot of evidence of what balance to strike in practice."
I basically agree with this, with the proviso that I'm currently trying to work out what the considerations to be weighed even are in the first place. I currently feel like I have a worse explicit handle on the considerations mitigating in favour of openness than those mitigating in favour of secrecy. I do think these higher-level issues (around incentives, institutional quality, etc) are likely to be important, but I don't yet know enough to put a number on that.

I think the points each of you make there are true and important.

As a further indication of the value of Will's point, I think a big part of the reason we're having this discussion at all is probably Bostrom's paper on information hazards, which is itself much more a list of considerations than an attempt to weigh them up. Bostrom makes this explicit:

The aim of this paper is to catalogue some of the various possible ways in which information can cause harm. We will not here seek to determine how common and serious these harms are or how they stack up against the many benefits of information—questions that would need to be engaged before one could reach a considered position about potential policy implications.

(We could describe efforts such as Bostrom's as "mapping the space" of consequences worth thinking about further, without yet engaging in that further thought.)

It seems possible to me that we've had more cataloguing of the considerations against openness than those for it, and thus that posts like this one can contribute usefully to the necessary step that comes before weighing up all the considerations in order to arrive at a well-informed decision. (For the same reason, it could also help slightly-inform all the other decisions we unfortunately have to make in the meantime.)

One caveat to that is that a post that mostly covers just the considerations that point in one direction could be counterproductive for those readers who haven't seen the other posts that provide the counterbalance, or who saw them a long time ago. But that issue is hard to avoid, as you can't cover everything in full detail in one place, and it also applies to Bostrom's paper and to a post I'll be making on this topic soon.

Another caveat in this particular case is that there are two related reasons why decisions on whether to develop/share (potentially hazardous) information may demand somewhat more caution than the average decision: the unilateralist's curse, and the fact that hard-to-reverse decisions destroy option value.

I personally think that it's still a good idea to openly discuss the reasons for openness, even if a post has to be somewhat lopsided in that direction for brevity and given that other posts were lopsided in the other direction. But I also personally think it might be good to explicitly note those extra reasons for caution somewhere within the "mostly-pro" post, for readers who may come to conclusions on the basis of that one post by itself.

(Just to be clear, I don't see this as disagreeing with Greg or Will's comments.)

Comment by michaela on Thoughts on The Weapon of Openness · 2020-02-15T08:36:08.282Z · score: 3 (3 votes) · EA · GW

Regarding the footnote issue, it sounds like maybe you had the same issue I had, so I'll share the fix I found in case that helps.

Standard footnotes on EAF and LessWrong only work for one paragraph. If you have a second paragraph (or dot points) that you want included, it just disappears.

To fix that, use "bignotes", as described here and here. (And it still works with dot points; just put 4 spaces before the dot point, as you would before any other paragraph.)

Also, as a general point, it seems there's little info on how writing posts/comments works on EAF, but there is for LessWrong, and a lot of that applies. So when I get stuck, I search on LessWrong (e.g., here). (And if that fails too, I google my problem + "markdown", as that's the syntax used).

It might be that you knew all that and your issue was different, but just thought I'd share in case it helps.

Comment by michaela on Thoughts on The Weapon of Openness · 2020-02-15T08:28:33.479Z · score: 1 (1 votes) · EA · GW

Interesting points, thanks for sharing!

One minor thought, in response to:

It might also be easier to bring external experts into secret private projects, through NDAs and the like, than it is to get them clearance to consult on secret state ones.

It seems to me that it's also possible that that's more of a symptom than a cause of the secrecy being less corrosive/corrupting in private than state projects (if indeed that is the case). That is, perhaps there's some other reason why secrecy leads to less corruption and distortion of incentives in businesses than in governments, and then because of that, those in-the-know in business are more willing to let external experts get NDAs-or-similar and look at what's going on than are those in-the-know in government.

Comment by michaela on Morality vs related concepts · 2020-02-12T17:45:09.199Z · score: 1 (1 votes) · EA · GW

Very interesting. I hadn't come across that way of using those three terms. Thanks for sharing!

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-10T07:34:31.758Z · score: 2 (2 votes) · EA · GW

Great!

And thanks for the suggestion to make this idea/criterion into its own post. I'll think about whether to do that, just adjust this post's main text to reflect that idea, or just add a footnote in this post.

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-09T20:41:13.981Z · score: 3 (2 votes) · EA · GW

And then we could perhaps further say that differential technological development is when a change in technology was a necessary step for the effect to occur. Again, it's not just an inevitable consequence of the chain of events, but rather something on the causal pathway between our action and the outcome we care about.

I think it's possible that this framing might make the relationship between all three clearer than I did in this post. (I think in the post, I more just pointed to a general idea and assumed readers would have roughly the same intuitions as me - and the authors I cite, I think.)

(Also, my phrasing about "causal pathways" and such is influenced by Judea Pearl's The Book of Why, which I think is a great book. I think the phrasing is fairly understandable without that context, but just thought I'd add that in case it's not.)

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-09T20:37:41.176Z · score: 3 (2 votes) · EA · GW

I think the first of those is close, but not quite there. Maybe this is how I'd put it (though I'd hadn't tried to specify it to this extent before seeing your comment):

Differential progress is about actions that advance risk-reducing lasting changes relative to risk-increasing progress, regardless of how these actions achieve that objective.

Differential intellectual progress is a subset of differential progress where an increase in knowledge by some participant is a necessary step between the action and the outcomes. It's not just that someone does learn something, or even that it would necessarily be true that someone would end up having learned something (e.g., as an inevitable outcome of the effect we care about). It's instead that someone had to learn something in order for the outcome to occur.

In the democracy example, if I teach someone about democracy, then the way in which that may cause risk-reducing lasting changes is via changes in that person's knowledge. So that's differential intellectual progress (and thus also differential progress, since that's the broader category).

If instead I just persuade someone to be in favour of democracy by inspiring them or making liking democracy look cool, then that may not require them to have changes in knowledge. In reality, they're likely to also "learn something new" along the lines of "this guy gave a great speech", or "this really cool guy likes democracy". But that new knowledge isn't why they now want democracy; the causal pathway went via their emotions directly, with the change in their knowledge being an additional consequence that isn't on the main path.

(Similar scenarios could also occur where a change in knowledge was necessary, such as if they choose to support democracy based on now explicitly thinking doing so will win them approval from me or from their friends. I'm talking about cases that aren't like that; cases where it's more automatic and emotion-driven.)

Does that seem clearer to you? (I'm still writing these a bit quickly, and I still think this isn't perfectly precise, but it seems fairly intuitive to me.)

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-08T18:11:05.306Z · score: 2 (2 votes) · EA · GW

Thanks for that feedback!

I get why you'd feel unsure about those points you mention. That's one of the things I let stay a bit implicit/fuzzy in this post, because (a) I wanted to keep things relatively brief and simple, and (b) I wanted to mostly just summarise existing ideas/usages, and that's one point where I think existing usages left things a bit implicit/fuzzy.

But here's roughly how I see things, personally: The way Tomasik uses the term "differential intellectual progress" suggests to me that he considers "intellectual progress" to also include just spreading awareness of existing ideas (e.g., via education or Wikipedia), rather than conceiving of new ideas. It seems to me like it could make sense to limit "intellectual progress" to just cases where someone "sees further than any who've come before them", and not include times when individuals climb further towards the forefront of humanity's knowledge, without actually advancing that knowledge. But it also seems reasonable to include both, and I'm happy to stick with doing so, unless anyone suggests a strong reason not to.

So I'd say "differential intellectual progress" both includes adding to the pool of things that some human knows, and just meaning that some individual knows/understands more, and thus humanity's average knowledge has grown.

None of this directly addresses your questions, but I hope it helps set the scene.

What you ask about is instead differential intellectual progress vs differential progress, and where "implementation" or "application" of ideas fits. I'd agree that fundamental vs applied research can be blurry, and that "even the applications of ideas and solutions requires a whole lot of intellectual work." E.g., if we're "merely implementing" the idea of democracy in a new country, that will almost certainly require at least some development of new knowledge and ideas (such as how to best divide the country into constituencies, or whether a parliamentary or presidential system is more appropriate for this particular country). These may be "superficial" or apply only in that location, and thus not count as "breakthroughs". But they are still, in some sense, advancements in humanity's knowledge.

I'd see it as reasonable to classify such things as either "differential intellectual progress" or "differential progress". I think they're sort-of edge cases, and that the terms are a bit fuzzy, so it doesn't matter too much where to put them.

But there are also things you'd have to do to implement democracy that aren't really best classified as developing new ideas or having individuals learn new things. Even if you knew exactly what form of democracy to use, and exactly who to talk to to implement this, and exactly what messages to use, you'd still need to actually do these things. And the shift in their political attitudes, values, ideologies, opinions, etc., seem best thought of as influenced by ideas/knowledge, but not themselves ideas/knowledge. So these are the sorts of things that I consider clear cases of "progress" (in the neutral, descriptive sense of "lasting changes", without necessarily having positive connotations).

Does that clarify things a bit? Basically, I think you're right that a lot of it is fuzzy/blurry, but I do think there are some parts of application/implementation that are more like just actually doing stuff, or people's opinions changing, and aren't in themselves matters of intellectual changes.

Comment by michaela on What are information hazards? · 2020-02-08T17:22:06.667Z · score: 2 (2 votes) · EA · GW

Thanks! That's great to hear.

And yes, I think that section you point at was important, and I David Kristoffersson for pushing me to attend to that distinction between actual harms, "information hazards" (where it's just a risk), and "potential information hazards" (where we don't have a specific way it would be harmful in mind, or something like that). (He didn't formulate things in that way, but highlighted the general issue as one worth thinking about more.)

Comment by michaela on Differential progress / intellectual progress / technological development · 2020-02-07T17:04:33.953Z · score: 3 (3 votes) · EA · GW

Thanks!

And yes, I think that's a fair point. (David also said the same when giving feedback.) This is why I write:

Also note that we’re using “progress” here as a neutral descriptor, referring to something that could be good or could be bad; the positive connotations that the term often has should be set aside.

I think that if I could unilaterally and definitively decide on the terms, I'd go with "differential technological development" (so keep that one the same), "differential intellectual development", and "differential development". I.e., I'd skip the word "progress", because we're really talking about something more like "lasting changes", without the positive connotations.

But I arrived a little late to set the terms myself :D

I.e., the term "differential intellectual progress" is already somewhat established, as is "differential progress", though to a slightly lesser extent. It's not absolutely too late to change them, but I'd worry about creating confusion by doing so, especially in a summary-style post like this.

However, it is true that my specific examples include some things that sound particularly not like progress at all, such as spread of nationalism and egoism. In contrast, Muehlhauser and Salamon's examples of "risk-increasing progress" sound more like the sort of thing people might regularly call "progress", but that they're highlighting could increase risks.

My current thinking is that how I've sliced things up (into something like lasting changes in techs, then lasting changes in ideas, then lasting changes of any sort) does feel most natural, and that the way that this makes the usage of the term "progress" confusing is a price worth paying.

But I'm not certain of that, and would be interested in others' thoughts on that too.

Comment by michaela on Four components of strategy research · 2020-02-03T14:56:52.586Z · score: 1 (1 votes) · EA · GW

Oh, good catch, sorry about that! I've now updated the link in the post.

Comment by michaela on Four components of strategy research · 2020-02-01T06:08:01.905Z · score: 4 (3 votes) · EA · GW

Thanks!

(My views, not Convergence's, as with most of my comments)

I think I'd personally also suspect that modelling causality might be the hardest step. But I'd also suspect mapping the space to perhaps seem easier than it really is, in a sense, because it's quite easy to map part of the space, and perhaps quite easy to then overestimate how much of the space you mapped, and underestimate the odds that you've failed to even notice some very important (clusters of) possible interventions or consequences. This is related to ideas like unknown unknown and Bostrom's crucial considerations.

I think some of the biggest wins of EA have probably been from mapping (maybe particularly of consequences, rather than interventions), more than from modelling. In particular, I have in mind identifying that it may be worth considering things like the wellbeing of people millions of years from now, astronomical waste, AI safety, global famine in a nuclear or impact winter (I'm thinking of ALLFED's work), and the welfare of wild animals. I think all of these things were noticed by some people before EAs got to them, but that they were mostly ignored by the vast majority of people.

In fact, I think one of the main reasons why "modelling" in the more standard sense is often so hard to do well is that there are many possible unknown unknowns like those things I listed, which could totally change the results of one’s model but which one hasn’t even thought to account for. So with this framework separating out mapping the space and modelling causality, the modelling causality step itself may be less difficult than one would normally expect "modelling" to be (though definitely still difficult).

(Prioritizing between strategies seems to me the simplest step. And constructing strategies seems like a step where it might be hard to get the optimal results, but where the gap between optimal and ok results won't be huge in the same way it might be for the mapping and modelling stages.)

I'm less sure what specifically you're interested in with your two questions. Also, I didn't come up with the framework in this post (my role was more to refine and communicate it), so my own knowledge and views weren't pivotal in forming it. But I can say that it seems to me that this framework doesn't clash with, and can be complementary too, various ideas, prioritisation processes, and modelling processes from groups like 80k, GPI, GiveWell, and Charity Entrepreneurship. And I believe that Convergence will soon be releasing various other work related to things like frameworks and tools for modelling causality and (I think) prioritisation research.

Comment by michaela on Four components of strategy research · 2020-01-31T11:28:08.293Z · score: 13 (9 votes) · EA · GW

(My view, not Convergence's)

I think it's very often the sign of a good model or framework that it seems fairly intuitive. Our intuitions often track the truth, but somewhat messily and implicitly. So it's useful to crystallise them into something more explicit and structured, with labels for the different pieces of it, to facilitate clearer thinking and discussion. It would be odd if the result of that crystallisation seemed totally counterintuitive - that might be a sign that a mistake has been made in the process.

Consider, for example, that many of the most valuable economic models/theories seem to some extent very obvious (e.g., the ideas that buyers buy more when things are cheaper, while sellers sell more when things are more expensive). But making them as explicit as economics does aids in thinking, and can sometimes then uncover quite counterintuitive ideas or findings.

Now, that obviously doesn't mean that all intuitive-seeming models are worth writing up or sharing. If it's not just that the model is "obvious", but in fact that people already explicitly recognised all the individual pieces and interactions in a clear way, then the model may not add value. But personally I think this is one useful way of collecting, making explicit, and structuring various intuitive ideas that often aren't made fully explicit. (Though they sometimes are: as we note, "We’re sure other models could be generated, and could likely also be helpful.")

Comment by michaela on Concerning the Recent 2019-Novel Coronavirus Outbreak · 2020-01-31T09:04:56.186Z · score: 30 (12 votes) · EA · GW

(Kind of just a nitpick)

I think I strongly agree with you on the value of being open to using betting in cases like these (at least in private, probably in public). And if you mean something like "Just in case anyone were to interpret Chi a certain way, I'd like to say that I strongly object to...", then I just fully agree with your comment.

But I think it's worth pointing out that no one said "we're not allowed to" do these bets - Chi's comment was just their personal view and recommendation, and had various hedges. At most it was saying "we shouldn't", which feels quite different from "we're not allowed to".

(Compare thinking that what someone is saying is racist and they really shouldn't have said it, vs actually taking away their platforms or preventing their speech - a much higher bar is needed for the latter.)

Comment by michaela on Concerning the Recent 2019-Novel Coronavirus Outbreak · 2020-01-31T08:17:42.108Z · score: 14 (8 votes) · EA · GW

Personally, I don't see the bet itself as something that shouldn't have happened. I acknowledge that others could have the perspective Chi had, and can see why they would. But didn't feel that way myself, and I personally think that downside is outweighed by the upside of it being good for the community's epistemics - and this is not just for Justin and Sean, but also for people reading the comments, so that they can come to more informed views based on the views the betters' take and how strongly they hold them. (Therefore, there's value in it being public, I think - I also therefore would personally suggest the comments shouldn't be deleted, but it's up to Sean.)

But I did feel really weird reading "Pleasure doing business Justin!". I didn't really feel uncomfortable with the rest of the upbeat tone Sean notes, but perhaps that should've been toned down too. That tone isn't necessary for the benefits of the bet - it could be civil and polite but also neutral or sombre - and could create reputational issues for EA. (Plus it's probably just good to have more respectful/taking-things-seriously norms in cases like these, without having to always calculate the consequences of such norms.)

Also, I feel uncomfortable with someone having downvoted Chi's comment, given that it seemed to have a quite reasonable tone and to be sharing input/a suggestion/a recommendation. It wasn't cutting or personal or damning. It seemed to me more like explaining Chi's view than persuading, so I think we should be somewhat wary of downvoting such things, even when we disagree, so we don't fall into something like groupthink. (I've strong upvoted for reasons of balance, even though I feel unsure about Chi's actual recommendations.)

Comment by michaela on Four components of strategy research · 2020-01-30T20:26:05.016Z · score: 1 (1 votes) · EA · GW

(I've managed to make it merely too large, rather than massive, by adding large white borders to the image itself, so that the image can't expand too far. But I'd still be interested in hearing of a proper fix.)

Comment by michaela on Four components of strategy research · 2020-01-30T19:12:11.897Z · score: 1 (1 votes) · EA · GW

(By the way, the "prioritizing" image isn't meant to be massive - if anyone knows how to choose the size of images, please let me know!

If I use the standard editor, many of the images are tiny. If I use the markdown editor, the rest are a good size, but that one is huge.)

Comment by michaela on When to post here, vs to LessWrong, vs to both? · 2020-01-27T09:33:06.276Z · score: 1 (1 votes) · EA · GW

(In my specific case, I'm currently mostly writing about stuff to do with good decision-making practices - which seems more LessWrong-y - but in the context of moral uncertainty, or with lots of altruism-related examples - which seems more EA Forum-y. But I'm also curious about these where-to-post questions more generally.)

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-24T23:50:52.240Z · score: 1 (1 votes) · EA · GW
I subtracted the reference to martingales from my previous comment because: a) not my expertise, b) this discussion doesn’t need additional complexity.
I'm sorry for having raised issues about paradoxes (perhaps there should be a Godwin's Law about them); I don’t think we should mix edge cases like St. Petersburg (and problems with unbounded utility in general) with the optimizer’s curse – it’s already hard to analyze them separately.

In line with the spirit of your comment, I believe, I think that it's useful to recognise that not all discussions related to pros and cons of probabilities or how to use them or that sort of thing can or should address all potential issues. And I think that it's good to recognise/acknowledge when a certain issue or edge case actually applies more broadly than just to the particular matter at hand (e.g., how St Petersburg is relevant even aside from the optimizer's curse). An example of roughly the sort of reasoning I mean with that second sentence, from Tarsney writing on moral uncertainty:

The third worry suggests a broader objection, that content-based normalization approach in general is vulnerable to fanaticism. Suppose we conclude that a pluralistic hybrid of Kantianism and contractarianism would give lexical priority to Kantianism, and on this basis conclude that an agent who has positive credence in Kantianism, contractarianism, and this pluralistic hybrid ought to give lexical priority to Kantianism as well. [...]
I am willing to bite the bullet on this objection, up to a point: Some value claims may simply be more intrinsically weighty than others, and in some cases absolutely so. In cases where the agent’s credence in the lexically prioritized value claim approaches zero, however, the situation begins to resemble Pascal’s Wager (Pascal, 1669), the St. Petersburg Lottery (Bernoulli, 1738), and similar cases of extreme probabilities and magnitudes that bedevil decision theory in the context of merely empirical uncertainty. It is reasonable to hope, then, that the correct decision-theoretic solution to these problems (e.g. a dismissal of “rationally negligible probabilities” (Smith, 2014, 2016) or general rational permission for non-neutral risk attitudes (Buchak, 2013)) will blunt the force of the fanaticism objection.

But I certainly don't think you need to apologise for raising those issues! They are relevant and very worthy of discussion - I just don't know if they're in the top 7 issues I'd discuss in this particular post, given its intended aims and my current knowledge base.

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-24T02:20:32.313Z · score: 1 (1 votes) · EA · GW

Thanks for these links. I know a little about the satisficer's curse, and share the view that "This generalization doesn't obviously seem to differentially affect explicit probabilities though." Hopefully I'll have time to look into the other two things you mention at some point.

(My kneejerk reaction to ""fast and frugal" heuristics often just perform better than more formal, quantitative models" is that if it's predictable that a heuristic would result in more accurate answers, even if we imagine we could have unlimited time for computations or whatever, then that fact, and ideally whatever causes it, can just be incorporated into the explicit model. But that's just a kneejerk reaction. And in any case, if he's just saying that in practice heuristics are often better, then I totally agree.)

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-24T00:16:35.199Z · score: 1 (1 votes) · EA · GW

Yeah, I've seen mentions of Buchak's work and one talk from her, but didn't really get it, and currently (with maybe medium confidence?) still think that, when talking about utility itself, and thus having accounted for diminishing returns and all that, one should be risk-neutral.

I hadn't heard of martingales, and have relatively limited knowledge of the St Petersburg paradox. It seems to me (low confidence) that:

  • things like the St Petersburg paradox and Pascal's mugging are plausible candidates for reasons to reject standard expected utility maximisation, at least in certain edge cases, and maybe also expected value reasoning
  • Recognising that there are diminishing returns to many (most?) things at least somewhat blunts the force of those weird cases
  • Things like accepting risk aversion or rounding infinitemal probabilities to 0 may solve the problems without us having to get rid of expected value reasoning or entirely get rid of expected utility maximisation (just augment it substantially)
  • There are some arguments for just accepting as rational what expected utility maximisation says in these edge cases - it's not totally clear that our aversion to the "naive probabilistic" answer here is valid; maybe that aversion just reflects scope neglect, or the fact that, in the St Petersburg case, there's the overlooked cost of it potentially taking months of continual play to earn substantial sums
  • I don't think these reveal problems with using EPs specifically. It seems like the same problems could occur if you talked in qualitative terms about probabilities (e.g., "at least possible", "fairly good odds"), and in either case the "fix" might look the same (e.g., rounding down either a quantitative or qualitative probability to 0 or to impossibility).
    • But it does seem that, in practice, people not using EPs are more likely to round down low probabilities to 0. This could be seen as good, for avoiding Pascal's mugging, and/or as bad, for a whole host of other reasons (e.g., ignoring many x-risks).

Maybe a fuller version of this post would include edge cases like that, but I know less about them, and I think they could create "issues" (arguably) even when one isn't using explicit probabilities anyway.

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-23T08:51:51.569Z · score: 1 (1 votes) · EA · GW

I now believe the statement of mine you quote was incorrect, and I've updated the optimizer's curse section, primarily to remove the sentence you quoted (as I think it's unnecessary in any case) and to alter an earlier part where I made a very similar claim so that it now says:

As best I can tell:
*The optimizer’s curse is likely to be a pervasive problem and is worth taking seriously.
*In many situations, the curse will just indicate that we're probably overestimating how much better (compared to the alternatives) the option we estimate is best is - it won't indicate that we should actually change what option we pick.
*But the curse can indicate that we should pick an option other than that which we estimate is best, if we have reason to believe that our estimate of the value of the best option is especially uncertain, and we don't model that information.

(I think I already knew this but just previously didn't explain it properly, leaving the conditions I had in mind as assumed, even though they often won't hold in practice.)

But I think this updated version doesn't address the points you make. From "if we're dealing with a bounded budget - in which case, one might prefer a suboptimal option with low variance", it sounds to me like maybe what you're getting at is risk-aversion and/or diminishing returns to a particular thing?

For example, let's say I can choose either A, which gives me $1 thousand in expectation, or B, which gives me $1 million in expectation. So far, B obviously seems way better. But what if B is way higher uncertainty (or way higher risk, if one prefers that phrasing)? Then maybe I'd prefer A.

I'd personally consider this biased if it's pure risk-aversion, and the dollar values perfectly correspond to my "utility" from this. But in reality, each additional dollar is less valuable. For example, perhaps I'm broke, and by far the most important thing is that I get $1000 to get myself out of a real hole - a quite low chance of much higher payoffs isn't worth it, because I get far less than 1000 times as much value out of 1000 times as much money.

If that's what you were getting at, I think that's all valid, and I think the optimizer's curse does probably magnify those reasons to sometimes not go with what you estimate will give you, in expectation, the most of some thing you value. But I think really that doesn't depend on the optimizer's curse, and is more about uncertainty in general. Also, I think it's really important to distinguish "maximising expected utility" from "maximising expected amount of some particular thing I value". My understanding is that "risk-aversion" based on diminishing returns to dollars, for example, can 100% make sense within expected utility maximisation - it's only pure risk-aversion (in terms of utility itself) that can't.

(Let me know if I was totally misunderstanding you.)

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-23T08:31:44.814Z · score: 3 (2 votes) · EA · GW

Yeah, I think all of that's right. I ended up coincidentally finding my way to a bunch of stuff about Goodhart on LW that I think is what you were referring to in another comment, and I've realised my explanation of the curse moved too fast and left out details. I think I was implicitly imagining that we'd already adjusted for what we know about the uncertainties of the estimates of the different options - but that wasn't made clear.

I've now removed the sentence you quote (as I think it was unnecessary there anyway), and changed my earlier claims to:

The implications of, and potential solutions to, the optimizer’s curse seem to be complicated and debatable. For more detail, see this post, Smith’s post, comments on Smith's post, and discussion of the related problem of Goodhart's law.
As best I can tell:
*The optimizer’s curse is likely to be a pervasive problem and is worth taking seriously.
*In many situations, the curse will just indicate that we're probably overestimating how much better (compared to the alternatives) the option we estimate is best is - it won't indicate that we should actually change what option we pick.
*But the curse can indicate that we should pick an option other than that which we estimate is best, if we have reason to believe that our estimate of the value of the best option is especially uncertain, and we don't model that information.
I've deliberately kept the above points brief (again, see the sources linked to for further explanations and justifications). This is because those claims are only relevant to the question of when to use EPs if the optimizer’s curse is a larger problem when using EPs than when using alternative approaches, and I don't think it necessarily is.

Now, that's not very clear, but I think it's more accurate, at least :D

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-21T23:23:55.072Z · score: 3 (2 votes) · EA · GW

That all seems to make sense to me. Thanks for the interesting reply!

Comment by michaela on Potential downsides of using explicit probabilities · 2020-01-20T23:25:17.750Z · score: 4 (3 votes) · EA · GW

I think I agree with pretty much all of that. And I'd say my position is close to yours, though slightly different; I might phrase mine like: "My understanding is that probabilities should always be used by ideal, rational agents with unlimited computational abilities etc. (Though that's still slightly 'received wisdom' for me.) And I also think that most people, and perhaps even most EAs and rationalists, should use probabilities more often. But I doubt they should actually be used for most tiny decisions, by actual humans. And I think they've sometimes been used with far too little attention to their uncertainty - but I also think that this really isn't an intrinsic issues with probabilities, and that intuitions are obviously also very often used overconfidently."

(Though this post wasn't trying to argue for that view, but rather to explore the potential downsides relatively neutrally and just see what that revealed.)

I'm not sure I know what you mean by the following two statements: "Probabilities [...] should eventually be used for most things" and "I think probabilities are a lot better now, but we could learn to get much better than them later." Could you expand on those points? (E.g., would you say we should eventually use probabilities even the 100th time we make the same decision as before about what to put in our sandwiches?)

Other points:

1. Yes, I share that view. But I think it's also interesting to note it's not a perfect correlation. E.g. Roser writes:

while I believe that we always have probabilities, this paper refrains from taking a stance on how we ought to decide on the basis of these probabilities. The question whether we have probabilities is completely separate from the question how we ought to make use of them. Here, I only ask the former question. The two issues are often not kept separate: the camp that is in favour of relying on probabilities is often associated with processing them in line with expected utility theory. I myself am in favour of relying on probabilities but I reject expected utility theory (and related stances such as cost-benefit analysis), at least if it comes as a formal way of spelling out a maximizing consequentialist moral stance which does not properly incorporate rights.

2. Yes, I agree. Possibly I should've emphasised that more. I allude to a similar point with "It seems the expected value of me bothering to do this EPM is lower than the expected value of me just reading a few reviews and then “going with my gut” (and thus saving time for other things)", and the accompanying footnote about utilitarianism.

4. I think I've seen what you'e referring to, e.g. in lukeprog's post on the optimizer's curse. And I think the basic idea makes sense to me (though not to the extent I could actually act on it right away if you handed me some data). But Chris Smith quotes the proposed solution, and then writes:

For entities with lots of past data on both the (a) expected values of activities and (b) precisely measured, realized values of the same activities, this may be an excellent solution.
In most scenarios where effective altruists encounter the optimizer’s curse, this solution is unworkable. The necessary data doesn’t exist.[7] The impact of most philanthropic programs has not been rigorously measured. Most funding decisions are not made on the basis of explicit expected value estimates. Many causes effective altruists are interested in are novel: there have never been opportunities to collect the necessary data.
The alternatives I’ve heard effective altruists propose involve attempts to approximate data-driven Bayesian adjustments as well as possible given the lack of data. I believe these alternatives either don’t generally work in practice or aren’t worth calling Bayesian.

That seems to me like at least a reason to expect the proposed solution to not work very well. My guess would be that we can still use our best guesses to make adjustments (e.g., just try to quantify our vague sense that a randomly chosen charity wouldn't be very cost-effective), but I don't think I understand the topic well enough to speak on that, really.

(And in any case, I'm not sure it's directly relevant to the question of whether we should use EPs anyway, because, as covered in this post, it seems like the curse could affect alternative approaches too, and like the curse doesn't mean we should abandon our best guess, just that we should be more uncertain about it.)

Comment by michaela on [AN #80]: Why AI risk might be solved without additional intervention from longtermists · 2020-01-20T03:37:50.305Z · score: 3 (2 votes) · EA · GW

I also found that passage odd, though I think for a somewhat different reason (or at least with a different framing).

For me, the passage reminded me of the O-ring theory of economic development, "which proposes that tasks of production must be executed proficiently together in order for any of them to be of high value".

For the sake of the argument, let's make the very extreme and unlikely assumption that, if no longtermists worked on them, each of AI risk, biorisk, and nuclear war would by themselves be enough to guarantee an existential catastrophe that reduces the value of the future to approximately 0. In that case, we might say that the EV of the future is ~0, and even if we were to totally fix one of those problems, or even two of them, the EV would still be ~0. Therefore, the EV of working on any one or two of the problems, viewed in isolation, is 0. But the EV of fixing all three would presumably be astronomical.

We could maybe say that existential catastrophe in this scenario is overdetermined, and so we need to remove multiple risks in order for catastrophe to actually not happen. This might naively make it look like many individual prevention efforts were totally worthless, and it might indeed mean that they are worthless if the other efforts don't happen, but it's still the case that, altogether, that collection of efforts is extremely valuable.

This also sort-of reminds me of some of 80k/Ben Todd's comments on attributing impact, e.g.:

A common misconception is that the credit for an action can’t add up to more than 100%. But it’s perfectly possible for both people to be responsible. Suppose Amy and Bob see someone drowning. Amy performs CPR while Bob calls the ambulance. If Amy wasn’t there to perform CPR, the person would have died while Bob called the ambulance. If Bob wasn’t there, the ambulance wouldn’t have arrived in time. Both Amy and Bob saved a life, and it wouldn’t have happened if either was there. So, both are 100% responsible.

I haven't taken the time to work through how well this point holds up when instead each x-risk causes less than 100%, e.g. 10%, chance of existential catastrophe if there were no longtermists working on it. But it seems plausible that there could be more than 100% worth of x-risks, if we add it up across the centuries/millenia, such that, naively, any specified effort to reduce x-risks that doesn't by itself reduce the total risk to less than 100% appears worthless.

So I think the point that, in a sense, only so many things can reduce the EV of the future by 10% does highlight that work on AI risk might be less valuable that one would naively think, if we expect other people/generations to drop the ball on other issues. But if we instead view this like a situation where there's a community of people working on AI risk, another working on biorisk, another working on nuclear, future generations working on whatever risks come up then, etc., then it seems totally possible that each community has to fix their problem. And so then it makes sense for our generation to work on the issues we can work on now, and more specifically for people with a comparative advantage for AI to work on that, those with a comparative advantage for general great power war reduction to work on that, etc.

So Paul's statement there seems to sort-of hold up as a partial argument for reducing focus on AI risk, but in a specific way (to ensure we also patch all the other leaks too). It doesn't seem like it holds up as an argument that AI safety work is less valuable than we thought in a simple sense, such that we should redirect efforts to non-longtermist work.

(I'm not saying Paul specifically intended to have the latter implication, but I think it'd be easy to make that inference from what he said, at least as quoted here. I'm also not sure if I'm right or if I've expressed myself well, and I'd be interested in other people's thoughts.)

Comment by michaela on [Link] The Optimizer's Curse & Wrong-Way Reductions · 2020-01-18T09:03:38.192Z · score: 1 (1 votes) · EA · GW

Late to the party, but I was re-reading this as it relates to another post I'm working on, and I realised I have a question. You write: (note that I say "you" in this comment a lot, but I'd also be interested in anyone else's thoughts on my questions)

The optimizer’s curse can show up even in situations where effective altruists’ prioritization decisions don’t involve formal models or explicit estimates of expected value. Someone informally assessing philanthropic opportunities in a linear manner might have a thought like:
Thing X seems like an awfully big issue. Funding Group A would probably cost only a little bit of money and have a small chance leading to a solution for Thing X. Accordingly, I feel decent about the expected cost-effectiveness of funding Group A.
Let me compare that to how I feel about some other funding opportunities…
Although the thinking is informal, there’s uncertainty, potential for bias, and an optimization-like process.

That makes sense to me, and seems a very worthwhile point. (It actually seems to me it might have been worth emphasising more, as I think a casual reader could think this post was a critique of formal/explicit/quantitative models in particular.)

But then in a footnote, you add:

Informal thinking isn’t always this linear. If the informal thinking considers an opportunity from multiple perspectives, draws on intuitions, etc., the risk of postdecision surprise may be reduced.

I'm not sure I understand what you mean by that, or if it's true/makes sense. It seems to me that, ultimately, if we're engaging in a process that effectively provides a ranking of how good the options seem (whether based on cost-effectiveness estimates or just how we "feel" about them), and there's uncertainty involved, and we pick the option that seems to come out on top, the optimizer's curse will be relevant. Even if we use multiple separate informal ways of looking at the problem, we still ultimately end up with a top ranked option, and, given that that option's ended up on top, we should still expect that errors have inflated its apparent value (whether that's in numerical terms or in terms of how we feel) more than average. Right?

Or did you simply mean that using multiple perspectives means that the various different errors and uncertainties might be more likely to balance out (in the same sort of way that converging lines of evidence based on different methodologies make us more confident that we've really found something real), and that, given that there'd effectively be less uncertainty, the significance of the optimizer's curse would be smaller. (This seems to fit with "the risk of postdecision surprise may be reduced".)

If that's what you meant, that seems reasonable to me, but it seems that we could get the same sort of benefits just by doing something like gathering more data or improving our formal models. (Though of course that may often be more expensive and difficult than cluster thinking, so highlighting that we also have the option of cluster thinking does seem useful.)

Comment by michaela on The ‘far future’ is not just the far future · 2020-01-18T06:43:47.343Z · score: 2 (2 votes) · EA · GW

This previous post by Gregory Lewis also seems relevant both to this point in particular, and to this post in general. E.g., Lewis writes:

there is a common pattern of thought along the lines of, "X-risk reduction only matters if the total view is true, and if one holds a different view one should basically discount it". Although rough, this cost-effectiveness guestimate suggests this is mistaken. Although it seems unlikely x-risk reduction is the best buy from the lights of a person-affecting view (we should be suspicious if it were), given ~$10000 per life year compares unfavourably to best global health interventions, it is still a good buy: it compares favourably to marginal cost effectiveness for rich country healthcare spending, for example.  
Second, although it seems unlikely that x-risk reduction would be the best buy by the lights of a person affecting view, this would not be wildly outlandish. Those with a person-affecting view who think x-risk is particularly likely, or that the cause area has easier wins available than implied in the model, might find the best opportunities to make a difference. It may therefore supply reason for those with such views to investigate the factual matters in greater depth, rather than ruling it out based on their moral commitments.
Comment by michaela on Opinion: Estimating Invertebrate Sentience · 2020-01-17T08:23:54.081Z · score: 3 (2 votes) · EA · GW
So, you might think, the absence of behavior X would really be evidence against sentience, while its presence alone in a creature might not be relevant to determining sentience.

I might be wrong about this or might be misunderstanding you, but I believe that, in any case where the absence of X is evidence against Y, the presence of X has to be evidence for Y. (Equivalently, whenever the presence of X is evidence for Y, the absence of X has to be evidence against Y.)

This does go against the common statement that "Absence of evidence is not evidence of absence." But we can understand that statement as having a very large kernel of truth, in that it is often the case that absence of evidence is only extremely weak evidence of absence. It depends on how likely it would be that we'd see the evidence if the hypothesis was true.

For an extreme example, let's say that an entity not being made of molecules would count as very strong evidence against that entity being sentient. But we also expect a huge number of other entities to be made of molecules without being sentient, and thus the fact that a given entity is made of molecules is extraordinarily weak evidence - arguably negligible for many purposes - that the entity is sentient. But it's still some evidence. If we were trying to bet on whether entity A (made of molecules) or entity B (may or may not be molecules; might be just a single atom or quark or whatever) is more likely to be sentient, we have reason to go with entity A.

This seems to sort-of mirror the possibility you describe (though here we're not talking behaviours), because being made of molecules is a necessary precondition for a huge number of what we'd take to be "indicators of sentience", but by itself is far from enough. Which does mean evidence of X is extremely weak evidence of sentience, but it's still some evidence, relative to a state in which we don't know whether X is true or not.

(I'm aware this is a bit of a tangent, and one that's coming fairly late. The post as a whole was very interesting, by the way - thanks to everyone who contributed to it.)

Comment by michaela on Making decisions when both morally and empirically uncertain · 2020-01-17T05:40:40.578Z · score: 2 (2 votes) · EA · GW

Those are two interesting distinctions. I don't have anything to add on that, but thanks for sharing those thoughts.

Not sure if it'll help but I have a short explanation and interactive widget trying to explain it here.

Oh, you're the person who made this value of information widget! I stumbled upon that earlier somehow, and am likely to link to it in a later post on applying VoI ideas to moral uncertainty.

Thanks for sharing the vNM widget; I intend to look at that soon.

Comment by michaela on [Link] The Optimizer's Curse & Wrong-Way Reductions · 2020-01-16T11:39:09.209Z · score: 2 (2 votes) · EA · GW

I think I agree with everything you've said there, except that I'd prefer to stay away from the term "Knightian", as it seems to be so often taken to refer to an absolute, binary distinction. It seems you wouldn't endorse that binary distinction yourself, given that you say "Knightian-ish", and that in your post you write:

we don’t need to assume a strict dichotomy separates quantifiable risks from unquantifiable risks. Instead, real-world uncertainty falls on something like a spectrum.

But I think, whatever one's own intentions, the term "Knightian" sneaks in a lot of baggage and connotations. And on top of that, the term is interpreted in so many different ways by different people. For example, I happened to have recently seen events very similar to those you contrasted against cases of Knightian-ish uncertainty used as examples to explain the concept of Knightian uncertainty (in this paper):

Finally, there are situations with so many unique features that they can hardly be grouped with similar cases, such as the danger resulting from a new type of virus, or the consequences of military intervention in conflict areas. These represent cases of (Knightian) uncertainty where no data are available to estimate objective probabilities. While we may rely on our subjective estimates under such conditions, no objective basis exists by which to judge them (e.g., LeRoy & Singell, 1987). (emphasis added)

So I see the term "Knightian" as introducing more confusion than it's worth, and I'd prefer to only use it if I also give caveats to that effect, or to highlight the confusions it causes. Typically, I'd prefer to rely instead on terms like more or less resilient, precise, or (your term) hazy probabilities/credences. (I collected various terms that can be used for this sort of idea here.)

[I know this comment is very late to the party, but I'm working on some posts about the idea of a risk-uncertainty distinction, and was re-reading your post to help inform that.]

Comment by michaela on Moloch and the Pareto optimal frontier · 2020-01-16T00:47:50.329Z · score: 4 (3 votes) · EA · GW

I think I had the same thought as Ozzie, if I'm interpreting his comment correctly. My thought was that this all seems to make sense, but that, from the model itself, I expected the second last sentence to be something like:

Given the above we’d expect that at first competitiveness and the accomplishment of humanity’s ultimate values are both improved but eventually they come apart and the trajectory skates along the Pareto frontier (that roughly speaking happens when we are at maximum technology or technological change becomes sufficiently slow) either towards maximizing competiveness or towards maximizing humanity's ultimate values (though it doesn't necessarily reach either extreme, and may skate back and forth).

And then that'd seem to lead to a suggestion like "Therefore, if the world is at this Pareto frontier or expected to reach it, a key task altruists should work on may be figuring out ways to either expand the frontier or increase the chances that, upon reaching it, we skate towards what we value rather than towards competitiveness."

That is, I don't see how the model itself indicates that, upon reaching the frontier, we'll necessarily move towards greater competitiveness, rather than towards humanity's values. Is that idea based on other considerations from outside of the model? E.g., that self-interest seems more common than altruism, or something like Robin Hanson's suggestion that evolutionary pressures will tend to favour maximum competitiveness (think I heard Hanson discuss that on a podcast, but here's a somewhat relevant post).

(And I think your reply is mainly highlighting that, at the frontier, there'd be a tradeoff between competitiveness and humanity's values, right? Rather than giving a reason why the competitiveness option would necessarily be favoured when we do face that tradeoff?)

Comment by michaela on Moloch and the Pareto optimal frontier · 2020-01-16T00:31:03.061Z · score: 4 (3 votes) · EA · GW

Your comment also reminded me of Robin Hanson's idea that policy debates are typically like tug of war between just two positions, in which case it may be best to "pull the rope sideways". Hanson writes: "Few will bother to resist such pulls, and since few will have considered such moves, you have a much better chance of identifying a move that improves policy."

That seems very similar to the idea that we may be at (or close to) the Pareto frontier when we consider only two dimensions, but not when we add a third, so it may be best to move towards the three-dimensional frontier rather than skating along the two-dimensional frontier.

Comment by michaela on Making decisions when both morally and empirically uncertain · 2020-01-14T05:41:58.088Z · score: 2 (2 votes) · EA · GW

Interesting. I hadn't explicitly made that connection, but it does seem worth thinking about.

I don't know if this is what you were implying, but that made me wonder about whether what I wrote in this post effectively entails that we could "cardinalise" the preferences of the ordinal theories under consideration. My first impression is that we probably still can't/shouldn't, but I'm actually not sure about that, so here's some long-winded thinking-aloud on the matter.

In MacAskill's thesis, he discusses a related matter:

Many theories do provide cardinally measurable choice-worthiness: in general, if a theory orders empirically uncertain prospects in terms of their choice-worthiness, such that the choice-worthiness relation satisfies the axioms of expected utility theory, then the theory provides cardinally measurable choice-worthiness. Many theories satisfy this condition. Consider, for example, decision-theoretic utilitarianism, according to which one should maximise expected wellbeing (and which therefore satisfies the axioms of expected utility theory). If, according to decision-theoretic utilitarianism, a guarantee of saving Person A is equal to a 50% chance of saving no-one and a 50% chance of saving both Persons B and C, then we would know that, according to decision-theoretic utilitarianism, the difference in choice-worthiness between saving person B and C and saving person A is the same as the difference in choice-worthiness between saving person A and saving no-one. We give meaning to the idea of ‘how much’ more choice-worthy one option is than another by appealing to what the theory says in cases of uncertainty.
However, this method cannot be applied to all theories, for two reasons. First, if the theory does not order empirically uncertain prospects, then the axioms of expected utility theory are inapplicable. This problem arises even for some consequentialist theories: if the theory orders options by the value of the consequences the option actually produces, rather than the value of the consequences it is expected to produce, then the theory has not given enough structure such that we can use probabilities to measure choice-worthiness on an interval scale. For virtue-ethical theories, or theories that focus on the intention of the agent, this problem looms even larger.
Second, the axioms of expected utility theory sometimes clash with common-sense intuition, such as in the Allais paradox. If a theory is designed to cohere closely with common-sense intuition, as many non-consequentialist theories are, then it may violate these axioms. And if the theory does violate these axioms, then, again, we cannot use probabilities in order to make sense of cardinal choice-worthiness. Plausibly, Kant’s ethical theory is an example of a merely ordinally measurable theory. According to Kant, murder is less choiceworthy than lying, which is less choice-worthy than failing to aid someone in need. But I don’t think it makes sense to say, even roughly, that on Kant’s view the difference in choice-worthiness between murder and lying is greater than or less than the difference in choice-worthiness between lying and failing to aid someone in need. So someone who has non-zero credence in Kant’s ethical theory simply can’t use expected choiceworthiness maximization over all theories in which she had credence. (line break added)

And later he adds:

Often, in responses to my work on taking into account normative uncertainty over merely ordinal theories, people make the following objection. They claim that we know that under empirical uncertainty, that expected utility theory or some variant is the correct decision theory. And we should treat normative uncertainty in the same way as empirical uncertainty. So if we encounter a merely ordinal theory, over which one cannot take an expectation, we should either ignore it or we should force some cardinalisation upon it. To this objection I replied that, under empirical uncertainty we rarely or never face merely ordinal choice-worthiness. This is a genuine disanalogy with empirical uncertainty. And to simply force merely ordinal theories to fit into the framework of expected utility theory, rather than to consider how to aggregate merely ordinal theories, is simply not to take one’s credence in those merely ordinal theories sufficiently seriously.

So it seems to me that he's arguing that we should respect that the theory is really meant to be ordinal, and we shouldn't force cardinality upon it.

Which leaves me with an initial, unclear thought along the lines of:

We can still do things as I suggested in this post.
If an ordinal moral theory really does only care about what action you take and not what it causes, then, as noted in this post, we can either (a) ignore empirical uncertainty or (b) set the probabilities to 100% (because the action is guaranteed to lead to the "outcome", which is that the outcome was taken); either way, we then use the Borda Rule as per usual.
But if an ordinal moral theory does care about outcomes, as most plausible theories do at least in part, then, as suggested in this post, we first look at the probabilities of each action under consideration leading to each outcome this theory "cares about". We then work out how each theory would rank these actions, with these probabilities of causing those outcomes in mind. We then use the Borda Rule on those rankings.
But we still haven't said the theories can tell us "how much" better one action is than another. And we haven't had to assume that the theories have a sufficiently complete/coherent/whatever [I only have a layperson's knowledge of the vNM utility theorem] set of preferences under empirical uncertainty that we can work out its cardinal preferences. It could have quite fuzzy or inconsistent ideas about what it would prefer in various situations of uncertainty, or it might very often consider an x% chance of A and a y% chance of A and B basically "equal" or "incomparable" or something like that.

But to be honest this seems to run into a bit of a roadblock related to me not really understanding how ordinal moral theories are really meant to work. I think that's a hard issue to avoid in general when thinking about moral uncertainty. There are these theories that seem like they just can't really be made to give us consistent, coherent preferences or follow axioms of rationality or whatever. But some of these theories are also very popular, including among professional philosophers, so, given epistemic humility, it does seem like it's worth trying to take them seriously - and trying to take them seriously for what they claim themselves to be (i.e., ordinal and arguably irrational).

(Plus there's the roadblock of me not having in-depth understanding of how the vNM utility theorem is meant to work.)

Additionally, in any case, it's also possible that MacAskill's Borda Rule effectively does implicitly cardinalize the theories. Tarsney seems to argue this, e.g.:

As I will argue at more length shortly, it seems to me that MacAskill’s approach fails to genuinely respect the phenomenon of merely ordinal theories, since the Borda counting approach (and MacAskill’s version of it in particular) is non-arbitrary only on the assumption that there are “hidden” cardinal values underlying the ordinal rankings of merely-ordinal theories.

If I'm interpreting Tarsney correctly, and if he's right, then that may be why when you poke around and consider empirical uncertainties it starts to look a lot like a typical method for getting cardinal preferences from preference orderings. But I haven't read that section of Tarsney's thesis properly, so I'm not sure.

(I may later write a post about Tarsney's suggested approach for making decision under moral uncertainty, which seems to have some advantages, and may also more fully respect ordinal theories ordinality.)

Comment by michaela on Andreas Mogensen's "Maximal Cluelessness" · 2020-01-05T07:12:34.715Z · score: 3 (2 votes) · EA · GW

Also, when reading Greaves and Mogensen's papers, I was reminded of the ideas of cluster thinking (also here) and model combination. I could be drawing faulty analogies, but it seemed like those ideas could be ways to capture, in a form that can actually be readily worked with, the following idea (from Greaves; the same basic concept is also used in Mogensen):

in the situations we are considering, instead of having some single and completely precise (real-valued) credence function, agents are rationally required to have imprecise credences: that is, to be in a credal state that is represented by a many-membered set of probability functions (call this set the agent’s ‘representor’)

That is, we can consider each probability function in the agent's representor as one model, and then either qualitatively use Holden's idea of cluster thinking, or get a weighted combination of those models. Then we'd actually have an answer, rather than just indifference.

This seems like potentially "the best of both worlds"; i.e., a way to capture both of the following intuitively appealing ideas:

  • perhaps we shouldn't present singular, sharp credence functions over extremely hard-to-predict long-term effects
  • we can still make educated guesses like "avoiding extinction is probably bad in expectation" and (perhaps) "giving to AMF is probably good in expectation".
    • (This second intuition can rest on ideas like "Yeah, ok, I agree that it's 'unclear' how to weigh up these arguments, but I weigh up arguments when it's unclear how to do so all the time. I'm still at least slightly more convinced by argument X, so I'm going to go with what it suggests, and just also remain extremely open to new evidence.")

Comment by michaela on Andreas Mogensen's "Maximal Cluelessness" · 2020-01-05T07:02:33.936Z · score: 7 (2 votes) · EA · GW

[I know I'm late to the party but...]

I'm certainly not an expert here, and I think my thinking is somewhat unclear, and my explanation of it likely will be too. But I share the sense that Knightian uncertainty can't be rational. Or more specifically, I have a sense that in these sorts of discussions, a lot of the work is being done by imprecise terms that imply a sort of crisp, black-and-white distinction between something we could call "regular" uncertainty and something we could call "extreme"/"radical"/"unquantifiable" uncertainty, without this distinction being properly made explicit or defended.

For example, in Hilary Greaves's paper on cluelessness (note: similar thoughts from me would apply to Mogensen's paper, though explained differently), she discusses cases of "simple cluelessness" and then argues they're not really a problem, because in such cases the "unforeseeable effects" cancel out in expectation, even if not in reality. E.g.,

While there are countless possible causal stories about how helping an old lady across the road might lead to (for instance) the existence of an additional murderous dictator in the 22nd century, any such story will have a precise counterpart, precisely as plausible as the original, according to which refraining from helping the old lady turns out to have the consequence in question; and it is intuitively clear that one ought to have equal credences in such precise counterpart possible stories.

Greaves is arguing that we can therefore focus on whether the "foreseeable effects" are positive or negative in expectation, just as our intuitions would suggest.

I agree with the conclusion, but I think the way it's juxtaposed with "complex cluelessness" (which she does suggest may be a cause for concern) highlights the sort of unwarranted (and implicit) sharp distinctions between "types" of uncertainty which I think are being made.

The three key criteria Greaves proposes for a case to involve complex cluelessness are:

(CC1) We have some reasons to think that the unforeseeable consequences of A1 would systematically tend to be substantially better than those of A2;
(CC2) We have some reasons to think that the unforeseeable consequences of A2 would systematically tend to be substantially better than those of A1;
(CC3) It is unclear how to weigh up these reasons against one another.

I think all of that actually applies to the old lady case, just very speculatively. One reason to think CC1 is that the old lady and/or anyone witnessing your kind act and/or anyone who's told about it could see altruism, kindness, community spirit, etc. as more of the norm than they previously did, and be inspired to act similarly themselves. When they act similarly themselves, this further spreads that norm. We could tell a story about how that ripples out further and further and creates huge amount of additional value over time.

Importantly, there isn't a "precise counterpart, precisely as plausible as the original", for this story. That'd have to be something like people seeing this act therefore thinking unkindness, bullying, etc. are more the norm that they previously thought they were, which is clearly less plausible.

One reason to think CC2 for the old lady case could jump off from that story; maybe your actions sparks ripples of kindness, altruism, etc., which leads to more people donating to GiveWell type charities, which (perhaps) leads to increased population (via reduced mortality), which (perhaps) leads to increased x-risk (e.g., via climate change or more rapid technological development), which eventually causes huge amounts of disvalue.

I'd argue you again can't tell a precise counterpart story that's precisely as plausible as this, and that's for reasons very similar to those covered in both Greaves and Mogensen's paper - there are separate lines of evidence and argument for GiveWell type charities leading to increased population vs them leading to decreased population, and for increased population increasing vs decreasing x-risk. (And again, it seems less plausible that witnessing your good deed would make people less likely to donate to GiveWell charities than more likely - or at least, a decrease would occur via different mechanisms than an increase, and therefore not be a "precise counterpart" story.)

I think both of these "stories" I've told are extremely unlikely, and for practical purposes aren't worth bearing in mind. But they do seem to me to meet the criteria in CC1 and CC2. And it doesn't seem to me there's a fundamental difference between their plausibility and worthiness-of-attention and that of the possibility for donations to AMF to increase vs decrease x-risk (or other claimed cases of [complex] cluelessness). I think it's just a difference of degree - perhaps a large difference in degree, but degree nonetheless. I can't see how we could draw some clear line somewhere to decide what uncertainties can just be dealt with in normal ways and which uncertainties make us count as clueless and thus unable to use regular expected value reasoning.

(And as for CC3, I'd say it's at least slightly "unclear" how to weigh up these reasons against one another.)

I think my thoughts here are essentially a criticism of the idea of a sharp, fundamental distinction between "risk" and "Knightian uncertainty"*, rather than of Greaves and Mogensen's papers as a whole. That is, if we did accept as a premise that distinction, I think that most of what Greaves and Mogensen say seems like it may well follow. (I also found their papers interesting, and acknowledge that I'm very much not an expert here and all of this could be unfounded, so for all these reasons this isn't necessarily a case of me viewing their papers negatively.)

*I do think that those can be useful concepts for heuristic-type, practical purposes. I think we should probably act differently when are credences are massively less well-founded than usual, and probably be suspicious of traditional expected value reasoning then. But I think that's because of various flaws with how humans think (e.g., overconfidence in inside-view predictions), not because different rules fundamentally should apply to fundamentally different types of uncertainty.

Comment by michaela on Making decisions under moral uncertainty · 2019-12-31T00:16:33.978Z · score: 1 (1 votes) · EA · GW

As I hope for this to remain a useful, accessible summary of these ideas, I've made various edits as I've learned more and gotten feedback (including on LessWrong), and expect to continue to do so. So please keep the feedback coming, so I can make this more useful for people!

(In the interest of over-the-top transparency, here's the version from when this was first published.)

Comment by michaela on Why I Don't Account for Moral Uncertainty · 2019-12-30T06:19:54.740Z · score: 1 (1 votes) · EA · GW

I think that's approximately true, but I also think it goes the other way around as well. In fact, just a few hours before reading your comment, I made a post using basically the same example, but in reverse (well, in both directions):

For example, I might wonder “Are fish conscious?”, which seems on the face of it an empirical question. However, I might not yet know precisely what I mean by “conscious”, and only really want to know whether fish are “conscious in a sense I would morally care about”. In this case, the seemingly empirical question becomes hard to disentangle from the (seemingly moral) question “What forms of consciousness are morally important?”

(Furthermore, my answers to that question may in turn may be influenced by empirical discoveries. For example, I may initially believe avoidance of painful stimuli demonstrates consciousness in a morally relevant sense, but then change that belief after learning that this behaviour can be displayed in a stimulus-response way by certain extremely simple organisms.)"

One idea informing why I put it that way around as well is that "consciousness" (like almost all terms) is not a fundamental element of nature, with clear and unambiguous borders. Instead, humanity has come up with the term, and can (to some extent) decide what it means. And I think one of the "criteria" a lot of people want that term to meet is "moral significance".

(From memory, and in my opinion, this sequence did a good job discussing how to think about words/concepts, their fuzzy borders, and then extent to which we are vs aren't free to use them however we want.)

(Also, I know some theories would propose consciousness is fundamental, but I don't fully understand them and believe they're not very mainstream, so I set them aside for now.)

This page is also relevant, e.g.:

A given worldview represents a combination of views, sometimes very difficult to disentangle, such that uncertainty between worldviews is constituted by a mix of empirical uncertainty (uncertainty about facts), normative uncertainty (uncertainty about morality), and methodological uncertainty (e.g. uncertainty about how to handle uncertainty, as laid out in the third bullet point above).
Comment by michaela on 2019 AI Alignment Literature Review and Charity Comparison · 2019-12-29T08:03:53.111Z · score: 5 (3 votes) · EA · GW

One question/nit-pick: In the discussion of Zabel & Muehlhauser's post, you write "Researchers from Google Brain were also named authors on the paper." Is that accurate? It seems to me that only Zabel & Muehlhauser are listed as authors of that post, and Google Brain itself (as opposed to Google more generally) isn't mentioned anywhere in the post. (But maybe I'm missing/misunderstanding something.)

Comment by michaela on 2019 AI Alignment Literature Review and Charity Comparison · 2019-12-29T06:49:23.641Z · score: 1 (1 votes) · EA · GW

Thanks for this - this seems a very valuable service, and will inform my donations. (Though it's harder to say what counterfactual influence it'll have, given I was already likely to donate to some of the organisations/funders you speak highly of and suggest you'll likely donate to.)

Comment by michaela on MichaelA's Shortform · 2019-12-22T05:35:17.611Z · score: 5 (4 votes) · EA · GW

Potential downsides of EA's epistemic norms (which overall seem great to me)

This is adapted from this comment, and I may develop it into a top-level post later. I welcome feedback on whether it'd be worth doing so, as well as feedback more generally.

Epistemic status: During my psychology undergrad, I did a decent amount of reading on topics related to the "continued influence effect" (CIE) of misinformation. My Honours thesis (adapted into this paper) also partially related to these topics. But I'm a bit rusty (my Honours was in 2017, and I haven't reviewed the literature since then).

This is a quick attempt to summarise some insights from psychological findings on the continued influence effect of misinformation (and related areas) that (speculatively) might suggest downsides to some of EA's epistemic norms (e.g., just honestly contributing your views/data points to the general pool and trusting people will update on them only to the appropriate degree, or clearly acknowledging counterarguments even when you believe your position is strong).

From memory, this paper reviews research on CIE, and I perceived it to be high-quality and a good intro to the topic.

From this paper's abstract:

Information that initially is presumed to be correct, but that is later retracted or corrected, often continues to influence memory and reasoning. This occurs even if the retraction itself is well remembered. The present study investigated whether the continued influence of misinformation can be reduced by explicitly warning people at the outset that they may be misled. A specific warning--giving detailed information about the continued influence effect (CIE)--succeeded in reducing the continued reliance on outdated information but did not eliminate it. A more general warning--reminding people that facts are not always properly checked before information is disseminated--was even less effective. In an additional experiment, a specific warning was combined with the provision of a plausible alternative explanation for the retracted information. This combined manipulation further reduced the CIE but still failed to eliminate it altogether. (emphasis added)

This seems to me to suggest some value in including "epistemic status" messages up front, but that this don't make it totally "safe" to make posts before having familiarised oneself with the literature and checked one's claims.

Here's a couple other seemingly relevant quotes from papers I read back then:

  • "retractions [of misinformation] are less effective if the misinformation is congruent with a person’s relevant attitudes, in which case the retractions can even backfire [i.e., increase belief in the misinformation]." (source) (see also this source)
  • "we randomly assigned 320 undergraduate participants to read a news article presenting either claims both for/against an autism-vaccine link [a "false balance"], link claims only, no-link claims only or non-health-related information. Participants who read the balanced article were less certain that vaccines are safe, more likely to believe experts were less certain that vaccines are safe and less likely to have their future children vaccinated. Results suggest that balancing conflicting views of the autism-vaccine controversy may lead readers to erroneously infer the state of expert knowledge regarding vaccine safety and negatively impact vaccine intentions." (emphasis added) (source)
    • This seems relevant to norms around "steelmanning" and explaining reasons why one's own view may be inaccurate. Those overall seem like very good norms to me, especially given EAs typically write about issues where there truly is far less consensus than there is around things like the autism-vaccine "controversy" or climate change. But it does seem those norms could perhaps lead to overweighting of the counterarguments when they're actually very weak, perhaps especially when communicating to wider publics who might read and consider posts less carefully than self-identifying EAs/rationalists would. But that's all my own speculative generalisations of the findings on "falsely balanced" coverage.
Comment by michaela on Should you familiarize yourself with the literature before writing an EA Forum post? · 2019-12-21T01:31:43.774Z · score: 5 (3 votes) · EA · GW

Epistemic status: During my psychology undergrad, I did a decent amount of reading on relevant topics, in particular under the broad label of the "continued influence effect" (CIE) of misinformation. My Honours thesis (adapted into this paper) also partially related to these topics. But I'm a bit rusty (my Honours was in 2017).

From this paper's abstract:

Information that initially is presumed to be correct, but that is later retracted or corrected, often continues to influence memory and reasoning. This occurs even if the retraction itself is well remembered. The present study investigated whether the continued influence of misinformation can be reduced by explicitly warning people at the outset that they may be misled. A specific warning--giving detailed information about the continued influence effect (CIE)--succeeded in reducing the continued reliance on outdated information but did not eliminate it. A more general warning--reminding people that facts are not always properly checked before information is disseminated--was even less effective. In an additional experiment, a specific warning was combined with the provision of a plausible alternative explanation for the retracted information. This combined manipulation further reduced the CIE but still failed to eliminate it altogether. (emphasis added)

This seems to me to suggest some value in including "epistemic status" messages up front, but that this don't make it totally "safe" to make posts before having familiarised oneself with the literature and checked one's claims.

From memory, this paper reviews research on CIE, and I perceived it to be high-quality and a good intro to the topic.

Here's a couple other seemingly relevant quotes from papers I read back then:

  • "retractions [of misinformation] are less effective if the misinformation is congruent with a person’s relevant attitudes, in which case the retractions can even backfire [i.e., increase belief in the misinformation]." (source) (see also this source)
  • "we randomly assigned 320 undergraduate participants to read a news article presenting either claims both for/against an autism-vaccine link [a "false balance"], link claims only, no-link claims only or non-health-related information. Participants who read the balanced article were less certain that vaccines are safe, more likely to believe experts were less certain that vaccines are safe and less likely to have their future children vaccinated. Results suggest that balancing conflicting views of the autism-vaccine controversy may lead readers to erroneously infer the state of expert knowledge regarding vaccine safety and negatively impact vaccine intentions." (emphasis added) (source)
    • This seems relevant to norms around "steelmanning" and explaining reasons why one's own view may be inaccurate. Those overall seem like very good norms to me, especially given EAs typically write about issues where there truly is far less consensus than there is around things like the autism-vaccine "controversy" or climate change. But it does seem those norms could perhaps lead to overweighting of the counterarguments when they're actually very weak, perhaps especially when communicating to wider publics who might read and consider posts less carefully than self-identifying EAs/rationalists would. But that's all my own speculative generalisations of the findings on "falsely balanced" coverage.

I've been considering brushing up on this literature to write a post for the forum on how to balance risks of spreading misinformation/flawed ideas with norms among EAs and rationalists around things like just honestly contributing your views/data points to the general pool and trusting people will update on them only to the appropriate degree. Reactions to this comment with inform whether I decide investing time into that would be worthwhile.

Comment by michaela on Will the Treaty on the Prohibition of Nuclear Weapons affect nuclear deproliferation through legal channels? · 2019-12-11T05:22:56.404Z · score: 6 (4 votes) · EA · GW
TPNW supporters are more likely to be engaged in civil war, but are less prone to inter-state conflict and territorial disputes

I initially interpreted that to mean that you found specific evidence that TPNW supporters are more likely to engage in civil war. However, the section that follows doesn't seem to contain such evidence, and in fact seems to include evidence that they are involved in less total armed conflict, which weakly suggests less civil war. (Though of course it doesn't prove that, as supporters might have slightly more civil war than non-supporters, yet far less interstate war, resulting in less armed conflict in total.)

So was that claim actually meant not as a confident claim based on specific evidence, but more as a guess, as the following quote seems to suggest?

As far as I can tell, TPNW supporters are just unusually peaceful countries. Though I suspect they may be involved in disproportionately more civil war, their lack of history of interstate violence and territorial disputes makes me think that are less likely to be involved in the kind of conflict where nuclear weapons are particularly valuable. (emphasis added)

One reason I'm curious about this is that it seems that a higher likelihood of civil war in a country might cause its government to have less desire to develop nuclear weapons (because the government might fear the weapons would at some point end up in the hands of some internal enemy or hated regime).