Posts
Comments
I think, in hindsight, the Fire Alarm first started ringing in a DeepMind building in 2017. Or perhaps an OpenAI building in 2020. It's certainly going off all over Microsoft now. It's also going off in many other places. To some of us it is already deafening. A huge, ominous, distraction from our daily lives. I really want to do something to shut the damn thing off.
See also the recent Lex Fridman Twitter poll [H/T Max Ra]:
Loudly and publicly calling for a global moratorium should have the effect of slowing down race-like behaviour, even if it is ultimately unsuccessful. We can at least buy some more time, it's not all or nothing in that sense. And more time can be used to buy yet more time, etc.
Factory farming is an interesting analogy, but the trade-off is different. You can think about whether abolitionism or welfarism has higher EV over the long term, but the stakes aren't literally the end of the world if factory farming continues to gain power for 5-15 more years (i.e. humanity won't end up in them).
The linked post is great, thanks for the reminder of it (and good to see it so high up the All Time top LW posts now). Who wants to start the institution lc talks about at the end? Who wants to devote significant resources to working on convincing AGI capabilities researchers to stop?
Would be good to see a breakdown of 1-10 into 1-5 and 5-10 years. And he should also do one on x-risk from AI (especially aimed at all those who answered 1-10 years).
Wow. I know a lot of his audience are technophiles, but that is a pretty big sample size!
Yes, but they will become increasingly cheaper. A taboo is far stronger than regulation.
GPT-4 is advanced enough that it will be used to meaningfully speed up the development of GPT-5. If GPT-5 can make GPT-6 on it's own, it's game over.
Spoofing accounts to combine multiple of them together (as in the Clippy story linked, but I'm imagining humans doing it). The kind of bending of the rules that happens when something is merely regulated but not taboo. It's not just Microsoft and Google we need to worry about. If the techniques and code are out there (open source models are not far behind cutting edge research), many actors will be trying to run them at scale.
Interesting, yes such moratorium on training new LLMs could help. But we also need to make the research morally unacceptable too - I think stigmatisation of AGI capabilities research could go a long way. No one is working on human genetic enhancement or cloning, mainly because of the taboos around them. It's not like there is a lot of underground research there. (I'm thinking this is needed, because any limits on compute that are imposed could easily be got around).
Cap the model size and sophistication somewhere near where it is now? Seems like there's easily a decade worth of alignment research that could be done on current models (and other theoretical work), which should be done before capabilities are advanced further. A moratorium would help bridge that gap. Demis Hassabis has talked about hitting the pause button as we get closer to the "grey zone". Now is the time!
Who else thinks we should be aiming for a global moratorium on AGI research at at this point? I'm considering ending every comment I make with "AGI research cessandum est", or "Furthermore, AGI research must be stopped".
What is the solution? I think we need a global moratorium on AGI research, ASAP. I know this will be very difficult. But it seems like it could be the best option we have at this point. Get the UN Security Council involved. Get the public involved (stigmatise the research).
I think we're at the stage now where we should be pushing for a global moratorium on AGI research. Getting the public on board morally stigmatizing it is an important part of this (cf. certain bio research like human genetic engineering).
Longtermism is wider than catastrophic risk reduction - e.g. it also encompasses "trajectory changes". It's about building a flourishing future over the very long term. (Personally I think x-risk from AGI is a short-term issue and should be prioritised, and Longtermism hasn't done great as a brand so far.)
You can't build a temple that lasts 1000 years without first ensuring that it's on solid ground and has secure foundations. (Or even a house that lasts 10 years for that matter.)
I agree that the focus on competency on existential risk research specifically is misplaced. But I still think the general competency argument goes through. And as I say elsewhere in the thread - tabooing "existential risk" and instead looking at Longtermism, it looks (and is) pretty bad that a flagship org branded as "longtermist" didn't last a year!
Fair enough. The implication is there though.
There is a certain hubris in claiming you are going to "build a flourishing future" and "support ambitious projects to improve humanity's long-term prospects" (as the FFF did on its website) only to not exist 6 months later and for reasons of fraud to boot.
This. We can taboo the words "existential risk" and focus instead on Longtermism. It's damning that the largest philanthropy focused on Longtermism -- the very long term future of humanity -- didn't even last a year. A necessary part of any organisation focused on the long term is a security mindset. It seems that this was lacking in the Future Fund. In particular, nothing was done to secure funding.
Cowen is saying that he thinks EA is less generally competent because of not seeing the x-risk to the Future Fund.
Even a 10 year delay is worth a huge amount (in expectation). We may well have a very different view of alignment by then (including perhaps being pretty solid on it's impossibility? Or perhaps a detailed plan for implementing it? (Or even the seemingly very unlikely "..there's nothing to worry about")), which would allow us to iterate on a better strategy (we shouldn't assume that our outlook will be the same after 10 years!)
but we should try to do both if there are sane ways to pursue both options today.
Yes! (And I think there are sane ways).
If you want MIRI to update from "both seem good, but alignment is the top priority" to your view, you should probably be arguing (or gathering evidence) against one or more of these claims:
- AGI alignment is a solvable problem.
There are people working on this (e.g. Yampolskiy, Landry & Ellen), and this is definitely something I want to spend more time on (note that the writings so far could definitely do with a more accessible distillation).
Absent aligned AGI, there isn't a known clearly-viable way for humanity to achieve a sufficiently-long reflection
I really don't think we need to worry about this now. AGI x-risk is an emergency - we need to deal with that emergency first (e.g. kick the can down the road 10 years with a moratorium on AGI research); then when we can relax a little, we can have the luxury to think about long term flourishing.
Humanity has never succeeded in any political task remotely as difficult as the political challenge of creating an enforced and effective 50+ year global moratorium on AGI.
I think this can definitely be argued against (and I will try and write more as/when I make a more fleshed out post calling for a global AGI moratorium). For a start, without all the work on nuclear proliferation and risk, we may well not be here today. Yes there has been proliferation, but there hasn't been an all-out nuclear exchange yet! It's now 77 years since a nuclear weapon was used in anger. That's a pretty big result I think! Also, global taboos around bio topics such as human genetic engineering are well established. If such a taboo is established, enforcement becomes a lesser concern, as you are then only fighting against isolated rogue elements rather than established megacorporations. Katja Grace discusses such taboos in her post on slowing down AI.
- EAs have not demonstrated the ability to succeed in political tasks that are way harder than any political task any past humans have succeeded on.
Fair point. I think we should be thinking much wider than EA here. This needs to become mainstream, and fast.
Also, I should say that I don't think MIRI should necessarily be diverting resources to work on a moratorium. Alignment is your comparative advantage so you should probably stick to that. What I'm saying is that you should be publicly and loudly calling for a moratorium. That would be very easy for you to do (a quick blog post/press release). But it could have a huge effect in terms of shifting the Overton Window on this. As I've said, it doesn't make sense for this not to be part of any "Death with Dignity" strategy. The sensible thing when faced with ~0% survival odds is to say "FOR FUCK'S SAKE CAN WE AT LEAST TRY AND PULL THE PLUG ON HUMANS DOING AGI RESEARCH!?!", or even "STOP BUILDING AGI YOU FUCKS!" [Sorry for the language, but I think it's appropriate given the gravity of the situation, as assumed by talk of 100% chance of death etc.]
I'm saying all this because I'm not afraid of treading on any toes. I don't depend on EA money (or anyone's money) for my livelihood or career[1] . I'm financially independent. In fact, my life is pretty good, all apart from facing impending doom from this! I mean, I don't need to work to survive[2], I've got an amazing partner and and a supportive family. All that is missing is existential security! I'd be happy to have "completed it mate" (i.e I've basically done this with the normal life of house, car, spouse, family, financial security etc); but I haven't - remaining is this small issue of surviving for a normal lifespan, having my children survive and thrive / ensuring the continuation of the sentient universe as we know it...
I think what's happened with Google/Deepmind and OpenAI/Microsoft has been much worse than safety washing. In effect it's been "existential safety washing"! The EA and AI x-risk communities have been far too placated by the existence of x-safety teams at these big AGI capabilities companies. I think at this point we need to be trying other things, like pushing for a moratorium on AGI development.
This is good, but I don't think it goes far enough. And I agree with your comments re "might not want MIRI to say "that move isn't available to us"". It might not be realistic to get the entire world to take a break on AGI work, but it's certainly conceivable, and I think maybe at this point more realistic than expecting alignment to be solved in time (or at all?). It seems reasonable to direct marginal resources toward pushing for a moratorium on AGI rather than more alignment work (although I still think this should at least be tried too!)
Your's and Nate's statement still implicitly assumes that AGI capabilities orgs are "on our side". The evidence is that they are clearly not. Demis is voicing caution at the same time that Google leadership have started a race with OpenAI (Microsoft). It's out of Demis' (and his seemingly toothless ethics board's) hands. Less accepting what has been tantamount to "existential safety washing", and more realpolitik, is needed. Better now might be to directly appeal to the public and policymakers. Or find a way to strategise with those with power. For example, should the UN Security Council be approached somehow? This isn't "defection".
The idea that Bostrom or Yudkowsky ever thought "the alignment problem is a major issue, but let's accelerate to AGI as quickly as possible for the sake of reaching the Glorious Transhumanist Future sooner" seems like revisionism to me
I'm not saying this is (was) the case. It's more subtle than that. It's the kind of background worldview that makes people post this (or talk of "pivotal acts") rather than this.
The message of differential technological development clearly hasn't had the needed effect. There has been no meaningful heed paid to it by the top AI companies. What we need now is much stronger statements. i.e. ones that use the word "moratorium". Why isn't MIRI making such statements? It doesn't make sense to go to 0 hope of survival without even seriously attempting a moratorium (or at the very least, publicly advocating for one).
If the AGI is so intelligent and powerful that it represents an existential risk to humanity, surely it is definitionally impossible for us to rein it in? And therefore surely the best approach would be ... to prevent work to develop AI
I'm starting to think that this intuition may be right (further thoughts in linked comment thread).
Yes, I think it's good that there is basically consensus here on AGI doom being a serious problem; the argument seems to be one of degree. Even OP says p(AGI doom by 2070) ~ 10%.
Don't get me wrong, I'd love to live in a glorious transhuman future (like e.g. Iain M Bank's Culture), but I just don't think it's worth the risk of doom, as things stand. Maybe after a few decades of moratorium, when we know a lot more, we can reassess (and hopefully we will still be able to have life extension so will personally still be around).
It now seems unfortunate that the AI x-risk prevention community was seeded from the transhumanist/techno-utopian community (e.g. Yudkowsky and Bostrom). This historical contingency is probably a large part of the reason why a global moratorium on AGI has never been seriously proposed/attempted.
The conjunctive/disjunctive dichotomy seems to be a major crux when it comes to AI x-risk. How much do belief in human progress, belief in a just world, the Long Peace, or even deep-rooted-by-evolution (genetic) collective optimism (all things in the "memetic water supply") play into the belief that the default is not-doom? Even as an atheist, I think it's sometimes difficult (not least because it's depressing to think about) to fully appreciate that we are "beyond the reach of God".
Social/moral consensus? There is precedent with e.g. recombinant DNA or human genetic engineering (if only the AI Asilomar conference was similarly focused on a moratorium!) It might be hard to indefinitely enforce globally, but we might at least be able to kick the can down the road a couple of decades (as seems to have happened with the problematic bio research).
(It should be achieved without such AGIs running around, if we want to minimise x-risk. Indeed, we should have started on this already! I'm starting to wonder whether it might actually be the best option we have, given the difficulty, or perhaps impossibility(?) of alignment.)
Yeah, I thought about nuclear risk, but Hiroshima and Nagasaki seem like good enough evidence for the possibility of widespread annihilation (or even Trinity for that matter). This would only be a good example if there was widespread appreciation for GCR potential from nuclear risk before any nuclear detonations. I don't think there was? (Especially considering that there was only a few short years (1933 - 1945) from theory to practice with the nuclear chain reaction.)
Also, the ending rubbed me up the wrong way a bit(!):
"Lastly, I would loathe it if the same selection effects applied to this document: If I spent a few days putting this document together, it seems easy for the AI safety community to easily put a few cumulative weeks into arguing against this document, just by virtue of being a community."
Is this basically saying that you aren't interested in engaging in the object level arguments? (Or the meta level arguments either, for that matter?) As you say:
Readers might want to keep in mind that parts of this post may look like a bravery debate.
And:
It’s not clear to me whether I have bound myself into a situation in which I can’t update from other people’s object-level arguments.
Deep learning also increases the probability of things like "roughly human-level AGIs run around for a few years before we see anything strongly superhuman", but this doesn't affect my p(doom) much because I don't see a specific path for leveraging this to prevent the world from being destroyed when we do reach superintelligence
But it does leave a path open to prevent doom: not reaching superintelligence! i.e. a global moratorium on AGI.
(The rest of the comment is great btw :))
This seems like a pretty general argument for being sceptical of anything, including EA!
This is an important point. The difficulty with AGI x-risk is that experimental verification isn't really possible (short of catastrophic-but-not-existential warning shots, that the most doomy people think are unlikely). Can anyone steelman with any strongly held beliefs that are justified without resort to overwhelming empirical verification? Maybe certain moral beliefs like the Golden Rule? But what about risks? Is there precedent with non-empirically-verified commonly accepted belief in a risk?
Very worrying! Can you get OpenAI to do something!? What's the plan? Is a global moratorium on AGI research possible? Should we just be trying for it anyway at this point? Are Google/DeepMind and Microsoft/OpenAI even discussing this with each other?
Of course! (I missed that)
Sounds good! Why the name Cavendish? Made me think of the Cavendish Laboratory (aka Cambridge Univeristy's Physics department - https://www.phy.cam.ac.uk/)
Wow, they gave to a LOT of charities - I make it >260,000! (~2100 pages *125/page). Many thousands of charities only getting $5 (and over half getting <$40). Seems like a lot of administrative overhead.
No one is saying that their inferences are "infallible" (and pretty much everyone I know in EA/AI Safety are open to changing their minds based on evidence and reason). We can do the best we can, that is all. My concern is that that won't be enough, and there won't be any second chances. Personally, I don't value "dying with dignity" all that much (over just dying). I'll still be dead. I would love it if someone could make a convincing case that there is nothing to worry about here. I've not seen anything close.
The whole field of existential risk is made up of hypotheses that aren't "testable", in that there would be no one there to read the data in the event of an existential catastrophe. This doesn't mean that there is nothing useful that we can say (or do) about existential risk. Regarding AI, we can use lines of scientific evidence and inference based on them (e.g. evolution of intelligence in humans etc). The post you link to provides some justifications for the claims it makes.
How much further? And how much relative to the stock market as a whole?
Thanks (posted job here).
Is there a current version of this? (And also the counterpart?)
CEEALAR is hiring for a full-time Operations Manager, please share with anyone you think may be interested: https://ceealar.org/job-operations-manager
To start mid-late February. £31,286 – £35,457 per year (full time, 40 hours a week).
MSFT is already valued at $1.7T (it's the 3rd largest company in the world). What multiple do you think is realistic over the next 5-10 years? Or would you suggest call options? (If Microsoft owns 1/3 of OpenAI with this latest deal, that still only represents 0.6% of MSFT marketcap. Also OpenAI profits are meant to be capped at 100x; with that you'd only get a 1.6x on your MSFT.)
Cool. I made some GWWC merch way back in 2011 on Zazzle. I see some of it is still there :)
Ok, well I took that course, and it most definitely did not have that kind of content in it (can you link to a relevant quote?). Better to think of the AI as an unconscious (arbitrary) optimiser, or even an indifferent natural process. There is nothing religious about AI x-risk.
Are you sure it was that course?!
This being will then rain down fire and brimstone on humanity for the original sin of being imperfect which is manifested by specification of an imperfect goal.
Doesn't sound very like it to me.
For those who want to deepen their knowledge of AI x-risk, I recommend reading the AGI Safety Fundamentals syllabus. Or better yet, signing up for the next iteration of the course (deadline to apply is 5th Jan).