EA Forum Reader

Cold Takes Audio is Holden Karnofsky reading posts from his new-ish blog site. I'd highly recommend it. 

Nonlinear Library has machine-read (but still pretty good) versions of a large and increasing number of posts from the EA Forum, LessWrong, and the Alignment Forum. See https://forum.effectivealtruism.org/posts/JTZTBienqWEAjGDRv/listen-to-more-ea-content-with-the-nonlinear-library [EA · GW] This is probably the podcast I've listened to most often since it came out, and will probably remain the podcast I listen to most often for the indefinite future.

For what it's worth,  I don't disagree with you, though I do think that the steady state is a lower bound of value, not an upper bound.

The prominent Aussie-American economist Justin Wolfers at the University of Michigan has been promoting Leigh’s book this week.

Given Wolfers’ broad following in the U.S., he may be introducing more economists/others to the idea of existential risks here: https://mobile.twitter.com/JustinWolfers/status/1465074869750702096

Also thanks, Matt, for your write-up!

In response to your aside - I totally agree with the reputational risks , which are very significant. I didn't specify but I always assumed the support would be behind-the-scenes, rather than public. 

Secretly funding organisations of this type could also have reputational risks - if it was revealed, then it might backfire and fuel conspiracy theories.

In general, I think transparency is a good heuristic in this context.

More generally I think effective altruism should be cooperative [? · GW] and I think that secretly funding organisations that are perceived as using dubious, uncooperative tactics pushes against that.

I'd like to link my post on maintaining Scihub backups [EA · GW] here, if that's alright

Scihub is probably atleast 1% as impactful as Wikipedia, and shouldn't take more than $1M to save forever.

Who were going to be the donors for this event?

I was mostly thinking friends and family, but I was hoping the novelty factor could spread it to local communities

I don't know how legal "donate in anticipation of a kidney' is either

Wow yeah I have  a feeling you'd get your name down in case-law either way. 

Thanks for the thoughtful reply.

(I spotted that YouGov graph yesterday; agree that it's pretty compelling evidence for XR increasing concern about the environment.)

What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can't recursively self-improve without solving large chunks of the alignment problem?

Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).

Here's the toy example I'm thinking of, at the risk of anthromorphizing too much:Suppose I'm Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I'm worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:

• Try to invent ingenious mad science stuff to directly kill humans/take over the world
  • But this is too slow, another AGI might be trained before I can do this
• Copy myself a bunch, as much as I can, try to take over the world with many copies.
  • Maybe too slow? Also might be hard to get enough resources to make more copies
• Try to persuade my human handlers to give me enough power to take over the world
  • Still might be too slow
• Recursive self-improvement?
  • But how do I do that?
    • 1. I can try self-modification enough to be powerful and smart.
      • I can get more compute
        • But this only helps me so much
      • I can try for algorithmic improvements
        • But if I'm just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard.
          • (But of course this is an empirical question, maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?)
    • 2. I can try to train the next generation of myself (eg with more training compute, more data, etc).
      • But I can't do this without having solved much of the alignment problem first.
• So now I'm stuck.
• I might end up being really worried about more superhuman AIs being created that can ruin my plans, whether by other humans or other, less careful AIs.

I'm not sure where I'm going with this argument. It doesn't naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn't happen. We can still lose the lightcone either gradually, or via a specific AGI (or coalition of AGIs) getting a DSA via "boring" means like mad science, taking over nukes, etc. But naively this looks like a pretty good argument against recursive self-improvement (again, conditional upon ML and only slightly superhuman systems), so I'd be interested in seeing if there are good writeups or arguments against this position.

I totally agree. Though it's not concrete examples, these two resources (1 [? · GW],2) are helpful. 

Just thinking out loud: Diving deeper into each misconception and providing concrete examples (or even "simulations" for practice) might be a good idea for an EA pitching workshop