Posts
Comments
You're right - I wasn't very happy with my word choice calling Google the 'engine of competition' in this situation. The engine was already in place and involves the various actors working on AGI and the incentives to do so. But these recent developments with Google doubling down on AI to protect their search/ad revenue are revving up that engine.
It's somewhat surprising to me the way this is shaking out. I would expect DeepMind and OpenAI's AGI research to be competing with one another*. But here it looks like Google is the engine of competition, less motivated by any future focused ideas about AGI more just by the fact that their core search/ad business model appears to be threatened by OpenAI's AGI research.
*And hopefully cooperating with one another too.
I think it's not quite right that low trust is costlier than high trust. Low trust is costly when things are going well. There's kind of a slow burn of additional cost.
But high trust is very costly when bad actors, corruption or mistakes arise that a low trust community would have preempted. So the cost is lumpier, cheap in the good times and expensive in the bad.
(I read fairly quickly so may have missed where you clarified this.)
If anyone consults a lawyer about this or starts the process with FTXrepay@ftx.us , it could be very useful to many of us if you followed up here and shared what your experience of the process was like.
I'm a long-time fan of renting over buying. I've been happily renting apartments since I started living on my own around ~2006. I've never owned a place and don't have any wishes or plans to. I skimmed the John Halstead post you linked to - a lot of his overall points have been motivations for me as well.
Last time I really looked into this (it's been a few years), the price-to-rent ratio varied a lot depending on the kind of place you live in. Generally if you lived in a major city, the ratio greatly favored renters. But in some lower-populated / suburban or rural areas, the ratio favored buyers: https://www.mrmoneymustache.com/2015/07/27/rent-vs-buy/ This is just one factor, but one I haven't found many people are aware of and so I like to bring up.
I do think it's a very personal question, and buying a home makes sense for some people. But I think it's generally overrated, and at least in North America I see way more people mistakenly buying property than mistakenly renting. But everyone needs to think through it for themselves. Some things that might make me think someone could be a good fit for ownership are a) if they like spending a lot of time fixing/working on their home, b) they want to be a landlord, c) they really don't like using parks/public areas and want a lot of private space, and/or d) they don't mind a long commute.
I'm surprised you were putting such high odds on it being a mistake at this point (even before the arrest). From my understanding (all public info), FTX's terms of service agreed that they would not touch customer funds. But then FTX loaned those funds to Alameda, who made risky bets with them.
IANAL but this seems to me like pretty clear case of fraud from FTX. I didn't think any of those aspects of the story were really disputed, but I have not been following the story as closely in the past week or so.
Will all the results of the survey be shared publicly on EA Forum? I couldn't find mention about this in the couple announcements I've seen for this survey.
It looks like at least some of the 2020 survey results were shared publicly. [1, 2, 3] But I can't find 2021 survey results. (Maybe there was no 2021 EA Survey?)
Thanks for the link and highlights!
Sam claims that he donated to Republicans: "I donated to both parties. I donated about the same amount to both parties (...) That was not generally known (...) All my Republican donations were dark (...) and the reason was not for regulatory reasons - it's just that reporters freak the fuck out if you donate to Republicans [inaudible] they're all liberal, and I didn't want to have that fight". If true, this seems to fit the notion that Sam didn't just donate to look good (i.e. he donated at least partly because of his personal altruistic beliefs)
What do you mean that this donation strategy would be from Sam's "personal altruistic beliefs"? Donating equally to both political parties has been a common strategy among major corporations for a long time. It's a way for them to push their own agenda in government. It's generally an amoral self-interested strategy, not an altruistic one.
I am a big fan of gratitude practice. I try to write a little in a gratitude journal most nights, which has helped my overall state of mind since I started doing it. I would recommend anybody to try it, including people involved in EA. And I'm glad you suggested it, as a little gratitude during a crisis like this can be especially helpful.
I have some reservations about posting things I'm grateful for publicly on this forum though. Gratitude can be a bit vulnerable, and this forum has more eyes on it than usual lately. Posting to a community about why you're thankful for that community could also be misinterpreted as being obsequious or virtue signalling. I think most of the benefits of gratitude practice can be enjoyed privately or with someone you trust, but if other people felt inclined to share their gratitude here, I would probably enjoy reading it and not be judgmental. And I may change my mind later and post some of that here as well :)
I would probably more excited about this thread if the forum had a feature to post comments anonymously. I don't see any downside to an anonymous public gratitude thread, but I'm probably too lazy to create an anonymous account just for that purpose.
Ultimately this was a failure of the EA ideas more so than the EA community. SBF used EA ideas as a justification for his actions. Very few EAs would condone his amoral stance w.r.t. business ethics, but business ethics isn't really a central part of EA ideas. Ultimately, I think the main failure was EAs failing to adequately condemn naive utilitarianism.
So I disagree with this because:
- It's unclear whether it's right to attribute SBF's choices to a failure of EA ideas. Following SBF's interview with Kelsey Piper and based on other things I've been reading, I don't think we can be sure at this point whether SBF was generally more motivated by naive utilitarianism or by seeking to expand his own power and influence. And it's unclear which of those headspaces led him to the decision to defraud FTX customers.
- It's plausible there actually were serious ways that the EA community failed with respect to SBF. According to a couple accounts, at least several people in the community had reason to believe SBF was dishonest and sketchy. Some of them spoke up about it and others didn't. The accounts say that these concerns were shared with more central leaders in EA who didn't take a lot of action based on that information (e.g. they could have stopped promoting Sam as a shining example of an EA after learning of reports that he was dishonest, even if they continued to accept funding from him). [1]
If this story is true (don't know for sure yet), then that would likely point to community failures in the sense that EA had a fairly centralized network of community/funding that was vulnerable, and it failed to distance itself from a known or suspected bad actor. This is pretty close to the OP's point about the EA community being high-trust and so far not developing sufficient mechanisms to verify that trust as it has scaled.
--
[1]: I do want to clarify that in addition to this story still not being unconfirmed, I'm mostly not trying to place a ton of blame or hostility on EA leaders who may have made mistakes. Leadership is hard, the situation sounds hard and I think EA leaders have done a lot of good things outside of this situation. What we find out may reduce how much responsibility I think the EA movement should put with those people, but overall I'm much more interested in looking at systemic problems/solutions than fixating on the blame of individuals.
Can you say a bit more about what you think EA has lost that makes it valuable?
Thanks for clarifying. That helps me understand your concern about the unilateralist's curse with funders acting independently. But i don't understand why the OP proposal of evaluating/encouraging funding diversification for important cause areas would exacerbate it. Presumably those funders could make risky bets regardless of this evaluation. Is it because you think it would bring a lot more funders into these areas or give them more permission to fund projects that they are currently ignoring?
Was it this post by chance? https://forum.effectivealtruism.org/posts/AbohvyvtF6P7cXBgy/brainstorming-ways-to-make-ea-safer-and-more-inclusive This one seems to be on a very similar topic. But it has a different name so it's probably not the same one but possibly Richard revised the title at some point.
Thanks for explaining, but who are you considering to be the "regulator" who is "captured" in this story? I guess you are thinking of either OpenPhil or OpenAI's board as the "regulator" of OpenAI. I've always heard the term "regulatory capture" in the context of companies capturing government regulators, but I guess it makes sense that it could be applied to other kinds of overseers of a company, such as its board or funder.
I've also been very upset since the FTX scandal began, and I love this community too. I think you're right that EA will lose some people. But I am not so worried the community will collapse (although it's possible that ending the global EA community could be a good thing). People's memories are short, and all things pass. In one year, I would be willing to bet you there will still be lots of (and still not enough!) good people working on and donating to important, tractable, and neglected causes. There will still be an EA Forum with lively debates happening, and that arguments about FTX will by that point make up a small fraction of the content. There will be still new people discovering EA and getting inspired by the potential to increase their positive impact in the world.
To be sure, I do think we should be worried* about the future of EA right now. But more in the sense of worried about whether EA can remain true to its core values and ideals going forward than about whether it can survive in some form.
--
*Note that when I say "we should be worried", I actually mean "we should be putting careful attention toward" rather than "we should be consumed by anxiety about". Be kind to yourself, and if you're feeling more of the latter, now may be a good time to double down on self-care.
Can you clarify which "public hearings" were demanded? Not sure if you're talking about how quickly the bankruptcy process has been moving at FTX, or how the reactions from people on EA Forum since the news about FTX started.
I followed this link, but I don't understand what it has to do with regulatory capture. The linked thread seems to be about nepotistic hiring and conflicts of interest at/around OpenAI.
I have thoughts on other points you made but just wanted to comment on this one bit for the moment:
I think it is very strange that SBF had so many inconsistencies between his claimed moral positions and his behavior and nobody noticed it.
Habryka noticed. His full comment is at that link but here are some key excerpts (emphasis mine):
Like, to be clear, I think the vast majority of EAs had little they could have or should have done here. But I think that I, and a bunch of people in the EA leadership, had the ability to actually do something about this.
I sent emails in which I warned people of SBF. I had had messages written but that I never sent that seem to me like if I had sent them they would have actually caused people to realize a bunch of inconsistencies in Sam's story. I had sat down my whole team, swore them to secrecy, and told them various pretty clearly illegal things that I heard Sam had done [sadly all uncomfirmed, asking for confidentiality and only in rumors] that convinced me that we should avoid doing business with him as much as possible (this was when we were considering whether to do a bunch of community building in the Bahamas). Like, in my worldview, I did do my due diligence, and FTX completely failed my due diligence, and I just failed to somehow propagate that knowledge.
...
I do also think it is pretty unlikely we could have prevented FTX exploding, though I do think we could have likely prevented FTX being super embedded in the EA Community, having a ton of people rely on its funding, and having Sam be held up in tons of places as a paragon of the EA community. Like, I think we fucked up pretty hard by saying for a few years that we think Sam did great by our values, when I think it was already pretty clear by that point that he quite likely wasn't.
Thanks for the quality summary. I finally opened this post after a couple days of ignoring it in my doomscrolling, because I thought there would be nothing new in it vs. other recent posts on FTX. But I found this critique actually gave me some new things to think about.
I think it's a good question, but I'm unsure whether a public discussion calling out names is the right way to go. And I think it might be net negative.
On the one hand, publicly calling people out could raise red flags about potential bad actors in the EA community early and in a transparent way. On the other hand, it could lead to witch hunts, false positives, etc. You can also imagine that whatever bad actors are in the community would start retaliating and listing good actors here who are their enemies, so it could cause a lot of confusion and infighting. (Think about the way Donald Trump uses public discourse.)
I think private discussions about this with others you trust are probably a good idea, to help guide personal decisions about who else to trust, work for, donate to etc. And this issue might make more decentralization actionable. For example, to mitigate bad actor risk, we might just have a background expectation that there is always X% chance that any individual in the EA community--even someone who is currently beloved--actually turns out to be a bad actor. And so no individual in the community should be responsible for more than Y% of the money, power, reputation etc. of the community to give EA more resilience against the inevitability of bad actors popping up here and there.
There may be other systemic ways to manage the risk too without making publicly outing suspected bad actors a regular thing. I'm also still open to the idea that public bad actor callouts might be a good idea, but I think it's a really delicate thing and I'd like to see a convincing argument/plan for how to make the discussion be productive before I would support it.
Commending Habryka for willing to share about these things. It takes courage and I think reflections/discussions like this could be really valuable (perhaps essential) to the EA community having the kind of reckoning about FTX that we need.
Maybe, but not so clear. He may have cared about consequentialist utilitarianism but just faked the deontology part.
Thanks, I appreciate this and bet some others will too.
There was a post Linch did earlier this year about their experience as a junior grantmaker in EA. It had an interesting part about conflicts of interest:
4. Conflicts of Interests are unavoidable if you want to maximize impact as a grantmaker in EA.
a. In tension with the above point, the EA community is just really small, and the subcommunities of fields within it (AI safety, or forecasting, or local community building) are even smaller.
b. Having a CoI often correlates strongly with having enough local knowledge to make an informed decision, so (e.g) if the grantmakers with the most CoIs in a committee always recuse themselves, a) you make worse decisions, and b) the other grantmakers have to burn more time to be more caught up to speed with what you know.
c. I started off with a policy of recusing myself from even small CoIs. But these days, I mostly accord with (what I think is) the equilibrium: a) definite recusal for romantic relationships, b) very likely recusal for employment or housing relationships, c) probable recusal for close friends, d) disclosure but no self-recusal by default for other relationships.
d. To avoid issues with #3, I’ve also made a conscious effort to do more recusals in reverse: that is, I’ve made a conscious effort to avoid being too friendly with grantees.
It makes sense to me what they were saying, even though it's troubling, because conflicts of interest are a problem.
It seems natural that more due diligence and organizational safeguards (like avoiding CoIs) will be some of the lessons learned from the FTX disaster. This could be an opportunity to prioritize such things more highly to improve EA's worst-case preparedness (e.g. corruption) , even at the expense of some impact/efficiency gains from having the best experts everywhere when things are going well.
That's true but hiding karma is an important thing for reducing bias (e.g. people being inclined to upvote something because it's already upvoted) and mental health. So it would be a great thing to have has a built-in feature (and perhaps even the default) so that it's accessible for many more users.
Karma scores are still important when it's hidden for sorting content by quality/popularity, it's just more of a behind-the-scenes thing.
(Thanks for all your great work on these forums.)
LessWrong has no such feature built-in that I'm aware of. Though there are external tools people have created to hide karma.
TIL that the code for LessWrong/EA Forum is open source.
The extension for blinding karma and author names has been a game changer for me. Massively improves my forum experience. Strong upvote, it'd be great to have these as native features so that they are much more accessible and others can enjoy the debiasing and mental health benefits.
I tend to prefer blinding karma instead of the author name. But they're both useful at different times. I think adding both and making independently controllable would be a huge step forward. Then the community can experiment with favorite configurations for different contexts.
Thanks to the OP, I've been meaning to post about this for months. And thanks to the forum devs who are doing a ton of work behind the scenes to make everything on these forums possible.
Strongly agree about the vote counts, I've been using a browser plugin to hide them for a couple months now. I think it should be a forum option and probably the default.
Stuart Russell is probably the most prominent example.
I think Dan Hendryks is doing good work in this area as well, as well as a bunch of people on the AI alignment team at DeepMind.
But yea, it'd be great if a lot more ML researchers/engineers engaged with the AI x-risk arguments and alignment research.
It's short for the Alignment Forum: https://www.alignmentforum.org/
The U.S. National Institute of Standards and Technology (“NIST”) is developing an AI Risk Management Framework.
Just a sidenote for anyone interested in this. There is an existing effort from some folks in the AI safety community to influence the development of this framework in a positive direction. See Actionable Guidance for High-Consequence AI Risk Management (Barett et al. 2022).
I see what you mean, but if you value cause prioritization seriously enough, it is really stifling to have literally no place to discuss x-risks in detail. Carefully managed private spaces are the best compromise I've seen so far, but if there's something better then I'd be really glad to learn about it.
That is high-value work. Holden Karnofsky's list of "important, actionable research questions" about AI alignment and strategy includes one about figuring out what should be done in deployment of advanced AI and leading up to it (1):
How do we hope an AI lab - or government - would handle various hypothetical situations in which they are nearing the development of transformative AI, and what does that mean for what they should be doing today?
Luke Muehlhauser and I sometimes refer to this general sort of question as the “AI deployment problem”: the question of how and when to build and deploy powerful AI systems, under conditions of uncertainty about how safe they are and how close others are to deploying powerful AI of their own.
My guess is that thinking through questions in this category can shed light on important, non-obvious actions that both AI labs and governments should be taking to make these sorts of future scenarios less daunting. This could, in turn, unlock interventions to encourage these actions.
You could avoid such infohazards by drawing up the scenarios in a private message or private doc that's only shared with select people.
I love the idea of reading silently together, but I don't think it wouldn't have occurred to me without this post.
In particular, the analogy with alchemy seems apropos given that concepts like sentience are very ill posed.
I took another look at that section, interesting to learn more about the alchemists.
I think most AI alignment researchers consider 'sentience' to be unimportant for questions of AI existential risk - it doesn't turn out to matter whether or not an AI is conscious or has qualia or anything like that. [1] What matters a lot more is whether AI can model the world and gain advanced capabilities, and AI systems today are making pretty quick progress along both these dimensions.
What would you say are good places to get up to speed on what we've learned about AI risk and the alignment problem in the past 8 years?
My favorite overview of the general topic is the AGI Safety Fundamentals course from EA Cambridge. I found taking the actual course to be very worthwhile, but they also make the curriculum freely available online. Weeks 1-3 are mostly about AGI risk and link to a lot of great readings on the topic. The weeks after that are mostly about looking at different approaches to solving AI alignment.
As for what has changed specifically in the last 8 years. I probably can't do the topic justice, but a couple things that jump out at me:
- The "inner alignment" problem has been identified and articulated. Most of the problems from Bostrom's Superintelligence (2014) fall under the category of what we now call "outer alignment", as the inner alignment problem wasn't really known at that time. Outer alignment isn't solved yet, but substantial work has been done on it. Inner alignment, on the other hand, is something many researchers consider to be more difficult.
Links on inner alignment: Canonical post on inner alignment, Article explainer, Video explainer - AI has advanced more rapidly than many people anticipated. People used to point to many things that ML models and other computer programs couldn't do yet as evidence that we were a long way from having anything resembling AI. But AI has now passed many of those milestones.
Here I'll list out some of those previously unsolved problems along with AI advances since 2015 that have solved them: Beating humans at Go (AlphaGo), beating humans at StarCraft (AlphaStar), biological protein folding (AlphaFold), having advanced linguistic/conversational abilities (GPT-3, PaLM), generalizing knowledge to competence in new tasks (XLand), artistic creation (DALL·E 2), multi-modal capabilities like combined language + vision + robotics (SayCan, Socratic Models, Gato).
Because of these rapid advances, many people have updated their estimates of when transformative AI will arrive to many years sooner than they previously thought. This cuts down on the time we have to solve the alignment problem.
--
[1]: It matters a lot whether the AI is sentient for moral questions around how we should treat advanced AI. But those are separate questions from AI x-risk.
That's a very good point.
With the assumption of longtermist ethics which I mentioned in the post, I think the difference in likelihoods has to be very large to make a difference though. Because placing equal value on future human lives to present ones makes extinction risks astronomically worse than catastrophic non-extinction risks.
(I don't 100% subscribe to longtermist ethics, but that was the frame I was taking for this post.)
You may have better luck getting responses to this posting on LessWrong with the 'AI' and 'AI Governance' (https://www.lesswrong.com/tag/ai-governance) tags, and/or on the AI Alignment Slack.
I skimmed the article. IMO it looks like a piece from circa 2015 dismissive of AI risk concerns. I don't have time right now to go through each argument, but it looks pretty easily refutable esp. with all that we've continued to learn about AI risk and the alignment problem in the past 8 years.
Was there a particular part from that link you found particularly compelling?
Agreed. The trend of writing "Epistemic status" as one of the first things in a post without a definition or explanation (kudos to Lizka for including one) has bothered me for some time. It immediately and unnecessarily alienates readers by making them feel like they need to be familiar with the esoteric word "epistemic", which usually has nothing to do with the rest of the post.
Would be happy to see this frequent jargon replaced with something like "How much you should trust me", "Author confidence" or "Post status" (maybe there's a better phrase, just some examples that come to mind).
Welcome to the field! Wow, I can imagine this post would be an intense crash course! :-o
There are some people who spend time on these questions. It's not something I've spent a ton of time on, but I think you'll find interesting posts related to this on LessWrong and AI Alignment Forum, e.g. using the value learning tag. Posts discussing 'ambitious value learning' and 'Coherent Extrapolated Volition' should be pretty directly related to your two questions.
Great post, thank you.
Really interesting observations.
I would say the conversion rate is actually shockingly low. Maybe CEA has more information on this, but I would be surprised if more than 5% of people who do Introductory EA fellowships make a high impact career change.
Do you have any sense of how many of those people are earning to give or end up making donation to effective causes play a significant role in their lives? I wonder if 5% is at least a little pessimistic for the "retention" of effective altruists if it's not accounting for people who take this path to making an impact.
My first 2 posts for this project went live on the Alignment Forum today:
1. Introduction to the sequence: Interpretability Research for the Most Important Century
2. (main post) Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios
I learned a lot from reading this post and some of the top comments, thanks for the useful analysis.
Throughout the post and comments people are tending to classify AI safety as a "longtermist" cause. This isn't wrong, but for anyone less familiar with the topic, I just want to point out that there are many of us who work in the field and consider AI to be a near-to-medium term existential risk.
Just in case "longtermism" gave anyone the wrong impression that AI x-risk is something we definitely won't be confronted with for 100+ years. Many of us think it will be much sooner than that (though there is still considerable uncertainty and disagreement about timelines).
See the related post "Long-Termism" vs. "Existential Risk" by Scott Alexander.
You're right, that paragraph was confusing. I just edited it to try and make it more clear.
These are thoughtful data points, but consider that they may just be good evidence for hard takeoff rather than soft takeoff.
What I mean is that most of these examples show a failure of narrow AIs to deliver on some economic goals. In soft takeoff, we expect to see things like broad deployment of AIs contributing to massive economic gains and GDP doublings in short periods of time well before we get to anything like AGI.
But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers (or it could just be limitations of the narrow AI). In fact, these limitations could even point more forcefully to the massive benefits of an AI that can generalize. And having the recipe for that AGI discovered and deployed in a lab doesn't depend on the success of prior narrow AIs in the regulated marketplace. AGI is a different breed and may also become powerful enough that it doesn't have to play by the rules of the regulated marketplace and national legal systems.
Machines will need to learn in open-ended play with the world, where today they mostly learn from labeled examples.
Have you seen DeepMind's Generally capable agents emerge from open-ended play? I think it is a powerful demonstration of learning from open-ended play actually working in a lab (not just a possible future approach). Though it is still in a virtual environment rather than the real physical world.
I think very, very few people really believe that superintelligence systems will be that influential.
A lot of prominent scientists, technologists and intellectuals outside of EA have warned about advanced artificial intelligence too. Stephen Hawking, Elon Musk, Bill Gates, Sam Harris, everyone on this open letter back in 2015 etc.
I agree that the number of people really concerned about this is strikingly small given the emphasis longtermist EAs put on it. But I think these many counter-examples warn us that it's not just EAs and the AGI labs being overconfident or out of left field.
What relatively well-scoped research activities are particularly likely to be useful for longtermism-oriented AI alignment?
[...]
(3) Activity that is likely to be relevant for the hardest and most important parts of the problem, while also being the sort of thing that researchers can get up to speed on and contribute to relatively straightforwardly (without having to take on an unusual worldview, match other researchers’ unarticulated intuitions to too great a degree, etc.)
I'm planning to spend some time working on this question, or rather part of it. In particular I'm going to explore the argument that interpretability research falls into this category, with some attention to which specific aspects or angles of interpretability research seem most useful.
Since I don't plan to spend much time thoroughly examining other research directions besides interpretability, I don't expect to have a complete comparative answer to the question. But by answering the question for interpretability, I hope to at least put together a fairly comprehensive argument for (or perhaps against, we'll see after I look at the evidence!) interpretability research that could be used by those considering it as a target for their funding or their time. I also hope that then someone trying to answer the larger question could use my work on interpretability as part of a comparative analysis across different research activities.
If someone is already working on this particular question and I'm duplicating effort, please let me know and perhaps we can sync up. Otherwise, I hope to have something to show on this question in a few/several weeks!
Why is this being severely downvoted? Is it because it's a scam or something, or because Red Cross is just not considered an effective charity?