The academic contribution to AI safety seems large 2020-07-30T10:30:19.021Z
[Link] The option value of civilization 2019-01-06T09:58:17.919Z
Existential risk as common cause 2018-12-05T14:01:04.786Z


Comment by technicalities on What are some potential coordination failures in our community? · 2020-12-12T08:24:34.725Z · EA · GW

The cycle of people coming up with ideas about how to organise people into projects, or prevent redundant posts, or make the Forum more accretive, being forgotten a week later. i.e. We fail to coordinate on coordination projects.

Comment by technicalities on Progress Open Thread: December 2020 · 2020-12-02T15:44:16.503Z · EA · GW

Can anyone in clean meat verify this news? The last time I checked, we were still years off market release.

Conditional on it being a real shock, hooray!

Comment by technicalities on The academic contribution to AI safety seems large · 2020-11-22T12:29:30.439Z · EA · GW

Follow-up post to ARCHES with ranking of existing fields, lots more bibliographies.

Comment by technicalities on The Case for Space: A Longtermist Alternative to Existential Threat Reduction · 2020-11-18T16:14:02.979Z · EA · GW

Some more prior art, on Earth vs off-world "lifeboats". See also 4.2 here for a model of mining Mercury (for solar panels, not habitats).

Comment by technicalities on The academic contribution to AI safety seems large · 2020-07-31T17:15:45.460Z · EA · GW

This makes sense. I don't mean to imply that we don't need direct work.

AI strategy people have thought a lot about the capabilities : safety ratio, but it'd be interesting to think about the ratio of complementary parts of safety you mention. Ben Garfinkel notes that e.g. reward engineering work (by alignment researchers) is dual-use; it's not hard to imagine scenarios where lots of progress in reward engineering without corresponding progress in inner alignment could hurt us.

Comment by technicalities on The academic contribution to AI safety seems large · 2020-07-31T17:00:31.239Z · EA · GW


research done by people who are trying to do something else will probably end up not being very helpful for some of the core problems.

Yeah, it'd be good to break AGI control down more, to see if there are classes of problem where we should expect indirect work to be much less useful. But this particular model already has enough degrees of freedom to make me nervous.

I think that it might be easier to assign a value to the discount factor by assessing the total contributions of EA safety and non-EA safety.

That would be great! I used headcount because it's relatively easy, but value weights are clearly better. Do you know any reviews of alignment contributions?

... This doesn't seem to mesh with your claim about their relative productivity.

Yeah, I don't claim to be systematic. The nine are just notable things I happened across, rather than an exhaustive list of academic contributions. Besides the weak evidence from the model, my optimism about there being many other academic contributions is based on my own shallow knowledge of AI: "if even I could come up with 9..."

Something like the Median insights collection, but for alignment, would be amazing, but I didn't have time.

those senior researchers won't necessarily have useful things to say about how to do safety research

This might be another crux: "how much do general AI research skills transfer to alignment research?" (Tacitly I was assuming medium-high transfer.)

I think the link is to the wrong model?

No, that's the one; I mean the 2x2 of factors which lead to '% work that is alignment relevant'. (Annoyingly, Guesstimate hides the dependencies by default; try View > Visible)

Comment by technicalities on The academic contribution to AI safety seems large · 2020-07-31T13:33:16.556Z · EA · GW

An important source of capabilities / safety overlap, via Ben Garfinkel:

Let’s say you’re trying to develop a robotic system that can clean a house as well as a human house-cleaner can... Basically, you’ll find that if you try to do this today, it’s really hard to do that. A lot of traditional techniques that people use to train these sorts of systems involve reinforcement learning with essentially a hand-specified reward function...
One issue you’ll find is that the robot is probably doing totally horrible things because you care about a lot of other stuff besides just minimizing dust. If you just do this, the robot won’t care about, let’s say throwing out valuable objects that happened to be dusty. It won’t care about, let’s say, ripping apart a couch cushion to find dust on the inside... You’ll probably find any simple line of code you write isn’t going to capture all the nuances. Probably the system will end up doing stuff that you’re not happy with.
This is essentially an alignment problem. This is a problem of giving the system the right goals. You don’t really have a way using the standard techniques of making the system even really act like it’s trying to do the thing that you want it to be doing. There are some techniques that are being worked on actually by people in the AI safety and the AI alignment community to try and basically figure out a way of getting the system to do what you want it to be doing without needing to hand-specify this reward function...
These are all things that are being developed by basically the AI safety community. I think the interesting thing about them is that it seems like until we actually develop these techniques, probably we’re not in a position to develop anything that even really looks like it’s trying to clean a house, or anything that anyone would ever really want to deploy in the real world. It seems like there’s this interesting sense in which we have the storage system we’d like to create, but until we can work out the sorts of techniques that people in the alignment community are working on, we can’t give it anything even approaching the right goals. And if we can’t give anything approaching the right goals, we probably aren’t going to go out and, let’s say, deploy systems in the world that just mess up people’s houses in order to minimize dust.
I think this is interesting, in the sense in which the processes to give things the right goals bottleneck the process of creating systems that we would regard as highly capable and that we want to put out there.

He sees this as positive: it implies massive economic incentives to do some alignment, and a block on capabilities until it's done. But it could be a liability as well, if the alignment of weak systems is correspondingly weak, and if mid-term safety work fed into a capabilities feedback loop with greater amplification.

Comment by technicalities on The academic contribution to AI safety seems large · 2020-07-31T11:39:23.633Z · EA · GW

Thanks for this, I've flagged this in the main text. Should've paid more attention to my confusion on reading their old announcement!

Comment by technicalities on The academic contribution to AI safety seems large · 2020-07-30T10:31:25.525Z · EA · GW

If the above strikes you as wrong (and not just vague), you could copy the Guesstimate, edit the parameters, and comment below.

Comment by technicalities on How can I apply person-affecting views to Effective Altruism? · 2020-04-29T09:33:30.983Z · EA · GW


It's a common view. Some GiveWell staff hold this view, and indeed most of their work involves short-term effects, probably for epistemic reasons. Michael Plant has written about the EA implications of person-affecting views, and emphasises improvements to world mental health.

Here's a back-of-the-envelope estimate for why person-affecting views might still be bound to prioritise existential risk though (for the reason you give, but with some numbers for easier comparison).

Dominic Roser and I have also puzzled over Christian longtermism a bit.

Comment by technicalities on What would a pre-mortem for the long-termist project look like? · 2020-04-12T07:58:34.208Z · EA · GW

Great comment. I count only 65 percentage points - is the other third "something else happened"?

Or were you not conditioning on long-termist failure? (That would be scary.)

Comment by technicalities on (How) Could an AI become an independent economic agent? · 2020-04-04T19:40:05.401Z · EA · GW

IKEA is an interesting case: it was bequeathed entirely to a nonprofit foundation with a very loose mission and no owner(?)

Not a silly question IMO. I thought about Satoshi Nakamoto's bitcoin - but if they're dead, then it's owned by their heirs, or failing that by the government of whatever jurisdiction they were in. In places like Britain I think a combination of "bona vacantia" (unclaimed estates go to the government) and "treasure trove" (old treasure also) cover the edge cases. And if all else fails there's "finders keepers".

Comment by technicalities on What posts do you want someone to write? · 2020-03-29T14:08:06.017Z · EA · GW

A nice example of the second part, value dependence, is Ozy Brennan's series reviewing GiveWell charities.

Why might you donate to GiveDirectly?
You need a lot of warmfuzzies in order to motivate yourself to donate.
You think encouraging cash benchmarking is really important, and giving GiveDirectly more money will help that.
You want to encourage charities to do more RCTs on their programs by rewarding the charity that does that most enthusiastically.
You care about increasing people’s happiness and don’t care about saving the lives of small children, and prefer a certainty of a somewhat good outcome to a small chance of a very good outcome.
You believe, in principle, that we should let people make their own decisions about their lives.
You want an intervention that definitely has at least a small positive effect.
You have just looked at GDLive and are no longer responsible for your actions.
Comment by technicalities on What posts do you want someone to write? · 2020-03-24T08:40:03.140Z · EA · GW

Collating predictions made by particularly big pundits and getting calibration curves for them. Bill Gates is getting a lot of attention now for warning of pandemic in 2015; what is his average though? (This is a bad example though, since I expect his advisors to be world-class and to totally suppress his variance.)

If this could be hosted somewhere with a lot of traffic, it could reinforce good epistemics.

Comment by technicalities on What posts do you want someone to write? · 2020-03-24T08:35:03.042Z · EA · GW

A case study of the Scientific Revolution in Britain as intervention by a small group. This bears on one of the most surprising facts: the huge distance, 1.5 centuries, between the scientific and industrial revs. Could also shed light on the old marginal vs systemic argument: a synthesis is "do politics - to promote nonpolitical processes!"

Comment by technicalities on What are some 1:1 meetings you'd like to arrange, and how can people find you? · 2020-03-20T14:52:38.622Z · EA · GW

Who am I?

Gavin Leech, a PhD student in AI at Bristol. I used to work in international development, official statistics, web development, data science.

Things people can talk to you about

Stats, forecasting, great books, development economics, pessimism about philosophy, aphorisms, why AI safety is eating people, fake frameworks like multi-agent mind. How to get technical after an Arts degree.

Things I'd like to talk to others about

The greatest technical books you've ever read. Research taste, and how it is transmitted. Non-opportunistic ways to do AI safety. How cluelessness and AIS interact; how hinginess and AIS interact.

Get in touch . I also like the sound of this open-letter site.

Comment by technicalities on Open Thread #46 · 2020-03-14T10:36:45.956Z · EA · GW

Suggested project for someone curious:

There are EA profiles of interesting influential (or influentially uninfluential) social movements - the Fabians, the neoliberals, the General Semanticists. But no one has written about the biggest: the scientific revolution in Britain as intentional intervention, a neoliberal style coterie.

A small number of the most powerful people in Britain - the Lord Chancellor, the king's physicians, the chaplain of the Elector Palatine / bishop of Chester, London's greatest architect, and so on - apparently pushed a massive philosophical change, founded some of the key institutions for the next 4 centuries, and thereby contributed to most of our subsequent achievements.


  • Elizabethan technology and institutions before Bacon. Scholasticism and mathematical magic
  • The protagonists: "The Invisible College"
  • The impact of Gresham College and the Royal Society (sceptical empiricism revived! Peer review! Data sharing! efficient causation! elevating random uncredentialed commoners like Hooke)
  • Pre-emptive conflict management (Bacon's and Boyle's manifestos and Utopias are all deeply Christian)
  • The long gestation: it took 100 years for it to bear any fruit (e.g. Boyle's law, the shocking triumph of Newton); it took 200 years before it really transformed society. This is not that surprising measured in person-years of work, but otherwise why did it take so long?
  • Counterfactual: was Bacon overdetermined by economic or intellectual trends? If it was inevitable, how much did they speed it up?
  • Somewhat tongue in cheek cost:benefit estimate.

This was a nice introduction to the age.

Comment by technicalities on Launching An Introductory Online Textbook on Utilitarianism · 2020-03-10T13:40:20.052Z · EA · GW

To my knowledge, most of the big names (Bentham, Sidgwick, Mill, Hare, Parfit) were anti-speciesist to some degree; the unusual contribution of Singer is the insistence on equal consideration for nonhumans. It was just not obvious to their audiences for 100+ years afterward.

My understanding of multi-level U is that it permits not using explicit utility estimation, rather than forbidding using it. (U as not the only decision procedure, often too expensive.) It makes sense to read (naive, ideal) single-level consequentialism as the converse, forbidding or discouraging not using U estimation. Is this a straw man? Possibly, I'm not sure I've ever read anything by a strict estimate-everything single-level person.

Comment by technicalities on What are the key ongoing debates in EA? · 2020-03-09T15:00:47.738Z · EA · GW

I read it as 'getting some people who aren't economists, philosophers, or computer scientists'. (:

(Speaking as a philosophy+economics grad and a sort-of computer scientist.)

Comment by technicalities on What are the key ongoing debates in EA? · 2020-03-09T13:28:40.601Z · EA · GW

Not sure. 2017 fits the beginning of the discussion though.

Comment by technicalities on What are the key ongoing debates in EA? · 2020-03-08T17:00:16.259Z · EA · GW

I've had a few arguments about the 'worm wars', whether the bet on deworming kids, which was uncertain from the start, is undermined by the new evidence.

My interlocutor is very concerned about model error in cost-benefit analysis, about avoiding side effects (and 'double effect' in particular); and not just for the usual PR or future credibility reasons.

Comment by technicalities on What are the best arguments that AGI is on the horizon? · 2020-02-16T11:55:56.723Z · EA · GW

It can seem strange that people act decisively about speculative things. So the first piece to understand is expected value: if something would be extremely important if it happened, then you can place quite low probability on it and still have warrant to act on it. (This is sometimes accused of being a decision-theory "mugging", but it isn't: we're talking about subjective probabilities in the range of 1% - 10%, not infinitesimals like those involved in Pascal's mugging.)

I think the most-defensible outside-view argument is: it could happen soon; it could be dangerous; aligning it could be very hard; and the product of these probabilities is not low enough to ignore.

1. When you survey general AI experts (not just safety or AGI people), they give a very wide distribution of predicting when we will have human-level AI (HLAI), with a central tendency of "10% chance of human-level AI... in the 2020s or 2030s". (This is weak evidence, since technology forecasting is very hard; these surveys are not random samples; but it seems like some evidence.)

2. We don't know what the risk of HLAI being dangerous is, but we have a couple of analogous precedents:

* the human precedent for world domination through intelligence / combinatorial generalisation / cunning

* the human precedent for 'inner optimisers': evolution was heavily optimising for genetic fitness, but produced a system, us, which optimises for a very different objective ("fun", or "status", or "gratification" or some bundle of nonfitness things).

* goal space is much larger than the human-friendly part of goal space (suggesting that a random objective will not be human-friendly, which combined with assumptions about goal maximisation and instrumental drives implies that most goals could be dangerous) .

* there's a common phenomenon of very stupid ML systems still developing "clever" unintended / hacky / dangerous behaviours

3. We don't know how hard alignment is, so we don't know how long it will take to solve. It may involve certain profound philosophical and mathematical questions, which have been worked on by some of the greatest thinkers for a long time. Here's a nice nontechnical statement of the potential difficulty. Some AI safety researchers are actually quite optimistic about our prospects for solving alignment, even without EA intervention, and work on it to cover things like the "value lock-in" case instead of the x-risk case.

Comment by technicalities on Can I do an introductory post? · 2020-02-14T07:12:19.247Z · EA · GW

Welcome! This is a fine thing - you could link to your story here, for instance:

Comment by technicalities on Growth and the case against randomista development · 2020-01-16T13:55:52.751Z · EA · GW

Great work. I'm very interested in this claim

the top ten most prescribed medicines many work on only a third of the patients

In which volume was this claim made?

Comment by technicalities on In praise of unhistoric heroism · 2020-01-08T11:11:10.539Z · EA · GW

Some (likely insufficient) instrumental benefits of feeling bad about yourself:

  • When I play saxophone I often feel frustration at not sounding like Coltrane or Parker; but when I sing I feel joy at just being able to make noise. I'm not sure which mindset has led to better skill growth. : Evaluations can compare up (to a superior reference class) or compare down. I try to do plenty of both. e.g. "Relative to the human average I've done a lot and know a lot." Comparing up is more natural to me, so I have an emotional-support Anki deck of achievements and baselines.
  • Impostor syndrome is always painful and occasionally useful. Most people can't / won't pay attention to what they're bad at, and people with impostor syndrome sometimes do, and so at least have a chance to improve. If I had the chance to completely "cure" mine I might not, instead halving the intensity. (Soares' Replacing Guilt is an example of a productive mindset which dispenses with this emotional cost though, and it might be learnable, I don't know.)
  • It's really important for EAs to be modest, if only to balance out the arrogant-seeming claim in the word "Effective".
  • My adult life was tense and confusing until I blundered into two-level utilitarianism, so endorsing doing most actions intuitively, not scoring my private life. (I was always going to do most things intuitively, because it's impossible not to, but I managed to stop feeling bad about it.) Full explicit optimisation is so expensive and fraught that it only makes sense for large or rare decisions, e.g. career, consumption habits, ideology.
Comment by technicalities on Against value drift · 2019-11-05T12:34:29.319Z · EA · GW

Sure, I agree that most people's actions have a streak of self-interest, and that posterity could serve as this even in cases of sacrificing your life. I took OP to be making a stronger claim, that it is simply wrong to say that "people have altruistic values" as well.

There's just something up with saying that these altruistic actions are caused by selfish/social incentives, where the strongest such incentive is ostracism or the death penalty for doing it.

Comment by technicalities on Against value drift · 2019-10-30T19:05:32.213Z · EA · GW

How does this reduction account for the many historical examples of people who defied local social incentives, with little hope of gain and sometimes even destruction? (Off the top of my head: Ignaz Semmelweis, Irena Sendler, Sophie Scholl.)

We can always invent sufficiently strange posthoc preferences to "explain" any behaviour: but what do you gain in exchange for denying the seemingly simpler hypothesis "they had terminal values independent of their wellbeing"?

(Limiting this to atheists, since religious martyrs are explained well by incentives.)

Comment by technicalities on What book(s) would you want a gifted teenager to come across? · 2019-08-07T15:10:23.810Z · EA · GW

Actually I think Feynman has the same risk. (Consider his motto: "disregard others" ! All very well, if you're him.)

Comment by technicalities on What book(s) would you want a gifted teenager to come across? · 2019-08-07T08:32:23.013Z · EA · GW

I think I would have benefitted from Hanson's 'Elephant in the Brain', since I was intensely frustrated by (what I saw as) pervasive, inexplicable, wilfully bad choices, and this frustration affected my politics and ethics.

But it's high-risk, since it's easy to misread as justifying adolescent superiority (having 'seen through' society).

Comment by technicalities on Call for beta-testers for the EA Pen Pals Project! · 2019-07-26T20:49:55.988Z · EA · GW

I suggest randomising in two blocks: people who strongly prefer video calls vs people who strongly prefer text, with abstainers assigned to either. Should prevent one obvious failure mode, people having an incompatible medium.

Comment by technicalities on Who are the people that most publicly predicted we'd have AGI by now? Have they published any kind of retrospective, and updated their views? · 2019-06-29T14:29:50.169Z · EA · GW

I was sure that Kurzweil would be one, but actually he's still on track. ("Proper Turing test passed by 2029").

I wonder if the dismissive received view on him is because he states specific years (to make himself falsifiable), which people interpret as crankish overconfidence.

Comment by technicalities on Effective Altruism is an Ideology, not (just) a Question · 2019-06-29T10:50:12.682Z · EA · GW

Fair. But without tech there would be much less to fight for. So it's multiplicative.

Comment by technicalities on Effective Altruism is an Ideology, not (just) a Question · 2019-06-29T08:08:01.562Z · EA · GW


Comment by technicalities on Effective Altruism is an Ideology, not (just) a Question · 2019-06-29T08:02:59.054Z · EA · GW

Good call. I'd add organised labour if I was doing a personal accounting.

We could probably have had trans rights without Burou's surgeries and HRT but they surely had some impact, bringing it forward(?)

No, I don't have a strong opinion either way. I suspect they're 'wickedly' entangled. Just pushing back against the assumption that historical views, or policy views, can be assumed to be unempirical.

Is your claim (that soc > tech) retrospective only? I can think of plenty of speculated technologies that swamp all past social effects (e.g. super-longevity, brain emulation, suffering abolitionism) and perhaps all future social effects.

Comment by technicalities on Effective Altruism is an Ideology, not (just) a Question · 2019-06-28T12:29:20.672Z · EA · GW
[80k and others claim that]: "The development and introduction of disruptive new technologies is a more fundamental and important driver of long-term change than socio-political reform or institutional change."

1) Why do you assume this is ideological (in the sense of not being empirically grounded)?

Anecdotally, I was a protest-happy socialist as a teenager, but changed my mind after reading about the many historical failures, grasping the depth of the calculation problem, and so on. This at least felt like an ideological shift dependent on facts.

2) 80,000 Hours have been pushing congressional staffer and MP as top careers (literally #1 or #2 on the somewhat-deprecated quiz) for years. And improving institutions is on their front page of problem areas.

Comment by technicalities on Cash prizes for the best arguments against psychedelics being an EA cause area · 2019-05-24T19:57:22.675Z · EA · GW

This comment is a wonderful crystallisation of the 'defensive statistics' of Andrew Gelman, James Heathers and other great epistemic policemen. Thanks!

Comment by technicalities on Existential risk as common cause · 2019-05-19T19:34:10.467Z · EA · GW

Thanks for this. I'm not very familiar with the context, but let me see if I understand. (In a first for me, I'm not sure whether to ask you to cite more scripture or add more formal argument.) Let's assume a Christian god, and call a rational consequence-counting believer an Optimising Christian.

Your overall point is that there are (or might be) two disjoint ethics, one for us and one for God, and that ours has a smaller scope, falling short of long-termism, for obvious reasons. Is this an orthodox view?

1. "The Bible says not to worry, since you can trust God to make things right. Planning is not worrying though. This puts a cap on the intensity of our longterm concern."

2. "Humans are obviously not as good at longtermism as God, so we can leave it to Him."

3. "Classical theism: at least parts of the future are fixed, and God promised us no (more) existential catastrophes. (Via flooding.)"

4. "Optimising Christians don't need to bring (maximally many) people into existence: it's supererogatory." But large parts of Christianity take population increase very seriously as an obligation (based on e.g. Genesis 1:28 or Psalm 127). Do you know of doctrine that Christian universalism stops at present people?

5. "Optimising Christians only need to 'satisfice' their fellows, raising them out of subsistence. Positive consequentialism is for God." This idea has a similar structure to negative utilitarianism, a moral system with an unusual number of philosophical difficulties. Why do bliss or happiness have no / insufficient moral weight? And, theologically: does orthodoxy say we don't need to make others (very) happy?

If I understand you, in your points (1) through (4) you appeal to a notion of God's agency outside of human action or natural laws. (So miracles only?) But a better theology of causation wouldn't rely on miracles, instead viewing the whole causal history of the universe as constituting God's agency. That interpretation, which at least doesn't contradict physics, would keep optimising Christians on the hook for x-risk.

Many of your points are appropriately hedged - e.g. "it might also be God’s job" - but this makes it difficult to read off actions from the claims. (You also appeal to a qualitative kind of Bayesian belief updating, e.g. "significant but not conclusive reason".) Are you familiar with the parliamentary model of ethics? It helps us act even while holding nuanced/confused views - e.g. for the causation question I raised above, each agent could place their own subjective probabilities on occasionalism, fatalism, hands-off theology and so on, and then work out what the decision should be. This kind of analysis could move your post from food-for-thought into a tool for moving through ancient debates and imponderables.

Comment by technicalities on How does one live/do community as an Effective Altruist? · 2019-05-16T20:50:24.359Z · EA · GW

Yudkowsky once officiated at a wedding. I find it quite beautiful.

Comment by technicalities on Which scientific discovery was most ahead of its time? · 2019-05-16T19:50:41.182Z · EA · GW

More engineering than science, but Turing's 1944 'Delilah' system for portable speech encipherment had no equivalent for more than a decade. (His biographer claims "30 years" but I don't know what he's comparing it to.) It was never deployed and was classified by the British government, so it had no impact.

Comment by technicalities on Legal psychedelic retreats launching in Jamaica · 2019-04-19T08:45:38.098Z · EA · GW

People probably won't give those examples here, for civility reasons. The SSC post linked above covers some practices Greg probably means, using historical examples.

Comment by technicalities on Who is working on finding "Cause X"? · 2019-04-11T19:59:41.035Z · EA · GW

One great example is the pain gap / access abyss. Only coined around 2017, got some attention at EA Global London 2017 (?), then OPIS stepped up. I don't think the OPIS staff were doing a cause-neutral search for this (they were founded 2016) so much as it was independent convergence.

Comment by technicalities on What skills would you like 1-5 EAs to develop? · 2019-03-28T19:44:52.059Z · EA · GW

Related news for the suffering engineering idea (but sadly also related for the cognition engineering one).

Comment by technicalities on How to Understand and Mitigate Risk (Crosspost from LessWrong) · 2019-03-14T20:45:49.568Z · EA · GW

I really like this, particularly how the conceptual splits lead to the appropriate mitigations.

The best taxonomy of uncertainty I've ever seen is this great paper by some physicists reflecting on the Great Recession. It's ordinal and gives a bit more granularity to the stats ("Opaque") branch of your tree, and also has a (half-serious) capstone category for catching events beyond reason:

1. "Complete certainty". You are in a Newtonian clockwork universe with no residuals, no observer effects, utterly stable parameters. So, given perfect information, you yield perfect predictions.
2. "Risk without uncertainty". You know a probability distribution for an exhaustive set of outcomes. No statistical inference needed. This is life in a hypothetical honest casino, where the rules are transparent and always followed. This situation bears little resemblance to financial markets.
3. "Fully Reducible Uncertainty". There is one probability distribution over a set of known outcomes, but parameters are unknown. Like an honest casino, but one in which the odds are not posted and must therefore be inferred from experience. In broader terms, fully reducible uncertainty describes a world in which a single model generates all outcomes, and this model is parameterized by a finite number of unknown parameters that do not change over time and which can be estimated with an arbitrary degree of precision given enough data. As sample size increases, classical inference brings this down to level 2.
4. "Partially Reducible Uncertainty". The distribution generating the data changes too frequently or is too complex to be estimated, or it consists in several nonperiodic regimes. Statistical inference cannot ever reduce this uncertainty to risk. Four sources:
(1) stochastic or time-varying parameters that vary too frequently to be estimated accurately;
(2) nonlinearities too complex to be captured by existing models, techniques, and datasets;
(3) non-stationarities and non-ergodicities that render useless the Law of Large Numbers, Central Limit Theorem, and other methods of statistical inference and approximation;
and (4) the dependence on relevant but unknown and unknowable conditioning information...
5. "Irreducible uncertainty". Ignorance so complete that it cannot be reduced using data: no distribution, so no success in risk management. Such uncertainty is beyond the reach of probabilistic reasoning, statistical inference, and any meaningful quantification. This type of uncertainty is the domain of philosophers and religious leaders, who focus on not only the unknown, but the unknowable.
Comment by technicalities on Suffering of the Nonexistent · 2019-03-03T10:33:11.684Z · EA · GW

There's a small, new literature analysing the subset of nonexistence I think you mean, under the name "impossible worlds". (The authors have no moral or meta-ethical aims.) It might help to use their typology of impossible situations: Impossible Ways vs Logic Violators vs Classical Logic Violators vs Contradiction-Realizers.

To avoid confusion, consider 'necessarily-nonexistent' or 'impossible moral patients' or some new coinage like that, instead of just 'nonexistent beings' otherwise people will think you're talking about the old Nonidentity Problem.

I think you'll struggle to make progress, because the intuition that only possible people can be moral patients is so strong, stronger than the one about electrons or microbial life and so on. In the absence of positive reasons (rather than just speculative caution), the project can be expected to move attention away from moral patients to nonpatients - at least, your attention.

Meta: If you don't want to edit out the thirteen paragraphs of preamble, maybe add a biggish summary paragraph at the top; the first time I read it (skimming, but still) I couldn't find the proposition.

Comment by technicalities on Existential risk as common cause · 2019-03-01T15:34:08.128Z · EA · GW

Ah I see. Agreed - thanks for clarifying.

Comment by technicalities on What skills would you like 1-5 EAs to develop? · 2019-03-01T15:28:50.980Z · EA · GW

Re: CSR. George Howlett started Effective Workplace Activism a couple of years ago, but it didn't take off that much. Their handbook is useful.

I tried quite hard to change my large corporation's charity selection process (maybe 50 hours' work), but found the stubborn localism and fuzzies-orientation impossible to budge (for someone of my persuasiveness and seniority).

Comment by technicalities on Three Biases That Made Me Believe in AI Risk · 2019-02-19T20:11:17.462Z · EA · GW

Example of a practical benefit from taking the intentional stance: this (n=116) study of teaching programming by personalising the editor:

Comment by technicalities on Why do you reject negative utilitarianism? · 2019-02-17T19:27:30.152Z · EA · GW

Re: 2. Here's a few.

Comment by technicalities on Existential risk as common cause · 2019-02-15T10:38:23.191Z · EA · GW

True - but how many people hold these inverses to be their primary value? (That is, I think the argument above is useful because almost everyone has something in the Goods set.)

Comment by technicalities on Existential risk as common cause · 2019-02-13T00:31:25.143Z · EA · GW

I mean that the end of the world isn't a bad outcome to someone who only values the absence of suffering, and who is perfectly indifferent between all 'positive' states. (This is Ord's definition of absolute NU, so I don't think I'm straw-manning that kind.) And if something isn't bad (and doesn't prevent any good), a utilitarian 'doesn't have to work on it' in the sense that there's no moral imperative to.

(1) That makes sense. But there's an escalation problem: worse risk is better to ANU (see below).

(2) One dreadful idea is that self-replicators would do the anti-suffering work, obviating the need for sentient guardians, but I see what you're saying. Again though, this uncertainty about moral patients licences ANU work on x-risks to humans... but only while moving the degenerate 'solution' upward, to valuing risks that destroy more classes of candidate moral patients. At the limit, the end of the entire universe is indisputably optimal to an ANU. So you're right about Earth x-risks (which is mostly all people talk about) but not for really farout scifi ones, which ANU seems to value.

Actually this degenerate motion might change matters practically: it seems improbable that it'd be harder to remove suffering with biotechnology than to destroy everything. Up to you if you're willing to bite the bullet on the remaining theoretical repugnance.

(To clarify, I think basically no negative utilitarian wants this, including those who identify with absolute NU. But that suggests that their utility function is more complex than they let on. You hint at this when you mention valuing an 'infinite game' of suffering alleviation. This doesn't make sense on the ANU account, because each iteration can only break even (not increase suffering) or lose (increase suffering).)

Most ethical views have degenerate points in them, but valuing the greatest destruction equal to the greatest hedonic triumph is unusually repugnant, even among repugnant conclusions.

I don't think instrumentally valuing positive states helps with the x-risk question, because they get trumped by a sufficiently large amount of terminal value, again e.g. the end of all things.

(I'm not making claims about other kinds of NU.)