EA Forum Reader

The prominent Aussie-American economist Justin Wolfers at the University of Michigan has been promoting Leigh’s book this week.

Given Wolfers’ broad following in the U.S., he may be introducing more economists/others to the idea of existential risks here: https://mobile.twitter.com/JustinWolfers/status/1465074869750702096

Also thanks, Matt, for your write-up!

In response to your aside - I totally agree with the reputational risks , which are very significant. I didn't specify but I always assumed the support would be behind-the-scenes, rather than public. 

Secretly funding organisations of this type could also have reputational risks - if it was revealed, then it might backfire and fuel conspiracy theories.

In general, I think transparency is a good heuristic in this context.

More generally I think effective altruism should be cooperative [? · GW] and I think that secretly funding organisations that are perceived as using dubious, uncooperative tactics pushes against that.

I'd like to link my post on maintaining Scihub backups [EA · GW] here, if that's alright

Scihub is probably atleast 1% as impactful as Wikipedia, and shouldn't take more than $1M to save forever.

Who were going to be the donors for this event?

I was mostly thinking friends and family, but I was hoping the novelty factor could spread it to local communities

I don't know how legal "donate in anticipation of a kidney' is either

Wow yeah I have  a feeling you'd get your name down in case-law either way. 

Thanks for the thoughtful reply.

(I spotted that YouGov graph yesterday; agree that it's pretty compelling evidence for XR increasing concern about the environment.)

I have domain expertise in digital marketing and couldn't want anything more than to help EA associations. I'm not qualified to say something regarding the inquiries of informing or how sweeping the mission ought to be, however when those things are sorted out I can let you know the most ideal method for utilizing YouTube and Google Search to achieve your objectives.

What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can't recursively self-improve without solving large chunks of the alignment problem?

Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).

Here's the toy example I'm thinking of, at the risk of anthromorphizing too much:Suppose I'm Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I'm worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:

• Try to invent ingenious mad science stuff to directly kill humans/take over the world
  • But this is too slow, another AGI might be trained before I can do this
• Copy myself a bunch, as much as I can, try to take over the world with many copies.
  • Maybe too slow? Also might be hard to get enough resources to make more copies
• Try to persuade my human handlers to give me enough power to take over the world
  • Still might be too slow
• Recursive self-improvement?
  • But how do I do that?
    • 1. I can try self-modification enough to be powerful and smart.
      • I can get more compute
        • But this only helps me so much
      • I can try for algorithmic improvements
        • But if I'm just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard.
          • (But of course this is an empirical question, maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?)
    • 2. I can try to train the next generation of myself (eg with more training compute, more data, etc).
      • But I can't do this without having solved much of the alignment problem first.
• So now I'm stuck.
• I might end up being really worried about more superhuman AIs being created that can ruin my plans, whether by other humans or other, less careful AIs.

I'm not sure where I'm going with this argument. It doesn't naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn't happen. We can still lose the lightcone either gradually, or via a specific AGI (or coalition of AGIs) getting a DSA via "boring" means like mad science, taking over nukes, etc. But naively this looks like a pretty good argument against recursive self-improvement (again, conditional upon ML and only slightly superhuman systems), so I'd be interested in seeing if there are good writeups or arguments against this position.

I totally agree. Though it's not concrete examples, these two resources (1 [? · GW],2) are helpful. 

Just thinking out loud: Diving deeper into each misconception and providing concrete examples (or even "simulations" for practice) might be a good idea for an EA pitching workshop

I've reached out to people several times via the forum, and I've helped mentor one person I contacted that way, as well as meeting with and getting input from a few others.

On the other hand, I think more topic-specific EA Facebook groups are generally better places for this, and would encourage people to use them more. (I would not really recommend using Facebook for anything else.)

Regarding understanding Mutiny, I'll point to Scott Siskind/Alexander's point about Trump and breaking the system, and   something far too long which I wrote a few years ago to explain what Holden is now calling mutiny;

...To finish the earlier Scott Alexander quote, “If your goal is to replace the current systems with better ones, then destroying the current system is 1% of the work, and building the better ones is 99% of it.” If there are goals other than a Tyler-Durden-esque smashing of the system in order to let anarchy reign, you need to be incredibly sure your goals are clearly specified.

And this is where I mention Trump, the elephant-nominee in the room. His goals are unclear, and he’s strategically refusing to be coherent long enough to change that. That means he’s building a coalition with shared dissatisfaction, without needing to clarify where he’s headed..

So in my view, Holden's rejected (a) is absolutely correct as the interpretation - but the problem is that it's an often dangerous / destructive rhetorical strategy for taking over, not a vision for how to actually change things. The vision for how to change things is separate.

But leaving aside Trump, the problem for anyone with a clear vision is that specifying a Utopia which would actually work - a feat which has never been accomplished - is still  the easiest part of radical reform. The far trickier parts are about getting from the present system to the future one. Doing radical reform without revolution is akin to rebuilding a bicycle into a motorcycle, without stopping along the way. 

On the other hand, the idea that we can tear things down and build the new system once we get rid of the current one is more akin to stopping and smashing the bicycle, and then thinking we can make new parts and somehow wind up with a motorcycle because we have some diagrams of how to build one. However seductively promising revolution seems, it requires implementing radical new systems correctly, in a single try, after breaking all of the necessary systems which it's replacing. And so I would suggest that the feasibility of managing it once you smashed all the working pieces is somewhere between  never-before-done and impossible.