EA Forum Reader

Epistemic status: Raw thoughts that I've just started to think about. I'm highly uncertain about a lot of this.

Some works that have inspired my thinking recently:

• Ben Garfinkel on scrutinising classic AI risk arguments - 80,000 Hours
• Unpacking Classic Arguments for AI Risk
• Reframing Superintelligence: Comprehensive AI Services as General Intelligence - I've read a little bit of this

Reading/listening to these works has caused me to reevaluate the risks posed by advanced artificial intelligence. While AI risk is currently the top cause in x-risk reduction, I don't think this is necessarily warranted. I think the CAIS model is a more plausible description of how AI is likely to evolve in the near future, but I haven't read enough to assess whether it makes AI more or less of a risk (to humanity, civilization, liberal democracy, etc.) than it would be under the classic "Superintelligence" model.

• User research case study: Developing Effective Altruism in Asia

I'm strongly interested in improving diversity in EA, and I think this is an interesting case study about how one could do that. Right now, it seems like there is a core/middle/periphery of the EA community where the core includes people and orgs in countries like the US, UK, and Australia, and I think the EA movement would be stronger if we actively tried to bring more people in more countries into the core.

I'm also interested in how we could use qualitative methods like those employed in user experience research (UXR) to solve problems in EA causes. I'm familiar enough with design thinking (the application of design methods to practical problems) that I could do some of this given enough time and training.

Hmm this doesn't any of your questions directly, but might be helpful context to set: My impression is that relatively few people actually set out to become generalists! I think it's more accurate of an explanation to think of some people being willing to do what needs to get done (or doing things they find interesting, or has high exploration value, or a myriad of other reasons). And if those things keep seeming like highly impactful things to do (or continues to be interesting, has high learning/exploration value, etc), they keep doing them, and then eventually become specialists in that domain.

If this impression is correct, specialists start off as generalists who eventually specialize more and more, though when they start specializing might vary a lot (where some people continue to be excited about the first thing they tried, so are set on their life path by the time they were 12, while others might have tried 30 before settling on the right one).

(I obviously can't speak for other EAs; these are just my own vague impressions. Don't take it too seriously, etc)

Advice for someone early as a generalist?

Hmm, I don't feel too strongly about this, but maybe be really inquisitive and questioning of received wisdom? 80,000 Hours doesn't have a monopoly on truth of what jobs are impactful! (the same goes for GiveWell and global health charities, or FHI and what is good for the future of humanity).

Often, in any given case they'll have more experience in the topic and have spent more time on it, so you should probably take their advice under consideration, but 1) you are a domain expert in being you, so there are always gaps and pieces of information that others will miss when trying to give you broadly applicable advice and 2) ultimately your life is your own, and you should probably take charge of your own life plans and epistemology.

(For what it's worth, I don't think this is in-principle always true. I can imagine a world very much like ours where the expert careers advice people are just much better than individuals at figuring out what careers are best for them, etc. I just think the world we live in is very far from that).

Did you stumble upon these different fields of interest by your own or did you surround yourself with smart people to get good understandings of various fields?

From the inside, it often feels like I just get interested in things and then explore them! But I don't think introspection is the most reliable source on these things. For example, if I had different friends, I'd undoubtedly have different interests, and be subconsciously guided to different things to focus on. There are also other unconscious biases, for example it is likely that I spend more time on interesting things that I perceive

I think once I do get interested in stuff, I'm often very willing to ping others to learn about it, eg, to ask dumb questions. For example I probably had at least 20 discussions about programming before my first programming job, and I had calls with Daniel Filan (Metaculus moderator) and David Manheim (superforecaster) before deciding on spending increasingly high amounts of time on forecasting. For specific questions I'm confused about, I also often ping other amateur forecasters, or occasionally biosecurity people I know in EA.

Thoughts on impact comparisons?

Too soon to tell.

can a generalist maybe bring knowledge/wisdom from intuitively non-adjacent disciplines into a project and help advance it?)

Yeah I think there are plenty of examples in history where lots of insights are gained from creatively applying fairly standard models in one field to another seemingly disparate field. Admittedly I think this as someone who's NOT a historian (or amateur historian), so I don't have a good idea of the true base rates.

Being a generalist is also not a free win. For example I think there's little stopping me from becoming a fairly good programmer, but obviously I wouldn't be as good at programming compared to if I started programming when I was 12. Something similar will be true for mathematics, or law, or entrepreneurship, or any number of other plausibly impactful things.

It's also easier for other people to interact with you if they think you're a generalist.

What skills are you lacking \ or which ones would you like to acquire to become a "Jack of all trades"?

Hmm. I think there's a bunch of skills in the general domain of "presentation" that I'm pretty bad at, and meant to improve this year, but the pandemic has become my excuse to not worry about it. For example, improving my accent, having nicer clothes, better general grooming, etc. Pandemic renders a lot of these things moot both because there seem to be better things to do in the short term, and because the default way to communicate is text plus Zoom calls, so I think people subconsciously judge on those things a lot less.

More broadly, better "people skills" seems pretty helpful, though at the moment I think I have not defined what I'm bad at well enough for me to train a lot in that domain.

I'd also like to get better at math, history and generalist research.

I think I'd like to get better at introspection and meta-cognition of my own feelings, though I'm not meta-cognitive enough to know if this would actually be really helpful (it's a vicious cycle!).

It'd be great if I can get really good at doing work consistently just because the work needs to be done at a high level, rather than because I'm personally in-the-moment excited about it.

Are you even aiming to become even more of a generalist? Yes or no - please elaborate.

I think my first paragraph might be a good answer to this? I roughly think I'll keep looking for things to do until I find something to "settle down" and specialize in (which might be in 6 months or 6 years).

The 'freedom to trade internationally' composite, which is basically not-being-protectionist, saw one of the largest improvements of any of their scores over this time. They are still protectionist relative to many richer countries, like the JV requirements. But they are dramatically less protectionist than they used to be, and this change coincided with / preceded their dramatic growth.

Thank you for your comment, great to hear that these suggestions seem to be helpful.

And great questions. I don't have a definitive answer (I might miss some things and the situation can differ a lot between organisations) but I can share some thoughts. Your second question is easier to answer, so I will start with that one.

I think that it's a bad idea to work with volunteers if:

• The organisation does not realise that volunteers aren't free [EA · GW]. They cost time, for coordination, recruitment, training, answering questions, etc. And sometimes the organisation needs to pay for an insurance, the volunteer's equipment, etc.
• The organisation is not willing to invest in the relationship with the volunteer. Even though some volunteers work independently, everybody is still human and would like to have some relation to the organisation. (Or to say it more bluntly: if you are a jerk, don't work with volunteers (or other people))
• The organisation has the resources to pay people for their work. I think that, in general, it is preferable to compensate people for their work. Payment is a strong sign of value and appreciation, and if an organisation has the means to pay for the work, but doesn't do it, I would question what the additional value of the volunteer's work is. Besides being a great motivation killer for the volunteer, it seems to be a better use of the volunteer's time to volunteer somewhere else.

Ok, so when do organisations benefit the most from working with volunteers? I think this question is more complex than the former and depends on several factors. In general I think two components are important: 1. the nature of the work and 2. the available volunteers. If the organisation's work is not too specialised and there are enough people who fit with your organisation (and are willing to donate their time), then an organisation can benefit from working with volunteers. Otherwise, I think an organisation will struggle and it can become (very) hard to have an impact while working with volunteers.

1. The nature of the work.

Usually volunteers can be very skilled, but they do not have much time to become specialised. There are of course exceptions, but generally volunteers can not spend too much time on their work, therefore getting to know the specific organisation (and the specific work) less well than a staff member. So if your organisation's work needs a lot of inside knowledge, or is specific, it will be harder to work with volunteers. Of course, volunteers can be trained, but as they tend to spend less time on the job (and the turn-over is sometimes high), it can make the volunteer work so costly that it's not efficient anymore (for these costs, the organisation can better hire 1 person full-time and train him/her/them). Another factor that plays a role is how easily the work can be coordinated and managed, as all these are costs as well.

As a rule of thumb I think for short-term volunteering the work should be easy and straight forward (if you give somebody a description of the work, they must be able to carry it out without further instructions). For tasks that take longer, the work should contribute towards volunteer satisfaction. That is easier if the work can be carried out autonomous, is flexible, the volunteer is responsible for the whole process, and the work is challenging. If the organisation's work does not have any of these characteristics I assume it could be hard to retain your volunteers.

2. The available volunteers.

Your question was about when organisations benefit most from working with volunteers, but I think that this question can not be separated from the volunteers themselves. Partly because volunteers have to fit with your organisation, partly because volunteers who donate their time to one organisation, can not do that for another organisation (what is the best way to distribute this resource in the community?), and partly because if there are no qualified people to volunteer, the organisation can not have their intended impact. And as an organisation usually needs more volunteers to carry out the same work as one paid employee (because of the time availability), and it can be hard to find reliable, high-quality volunteers, an organisation can be easily bottlenecked by a lack of volunteers (some people say that good volunteers are power-law distributed).

Another factor is if the volunteering is the best use of the volunteer's time. Somebody could not only donate their time to another (more effective?) organisation, but also use their time to e.g. skill-up or earn to give. For effective volunteering that means that volunteers often volunteer to learn something, improve a skill, or test out a career option. If your organisation can offer an opportunity for people to do one of these, it could be encouraged (and positive for the community) to create volunteer opportunities. But if the organisation's work provides an opportunity that people are not looking for, it can be really hard to fill your positions.

I think his point was related to development via protection, etc., that was then loosened somewhat. But not sure.

Hickel claims that China's very non-neoliberal policy enriched its people

China's post-1979 reforms are one of the textbook examples of neoliberalism! They privatized many businesses, allowed the creation of markets for many goods and services, opened up to international trade and reduced capital controls. While there is still a great deal of central control, the level is dramatically lower than it was in the 1970s. Their economic freedom ranking improved from a terrible 3.59 in 1980 to a respectable (though still not great) 6 in 2002, a very rapid rise. This is a similar increase to other countries undergoing neoliberal reforms at the same time, like the UK, Chile and Sweden, though many of these started from a higher base.

EDIT: Unless you are talking about the One Child Policy, which I would agree is very non-neoliberal, and is a major policy.

This sounds like a good battle plan! And do let us know what you finally decide on!

Oh, also I want to be clear that what I was concerned about is the broader structure of argument, rather than being critical of your post! :)

Sorry if this isn’t as polished as I’d hoped. Still a lot to read and think about, but posting as I won’t have time now to elaborate further before the weekend. Thanks for doing the AMA!

It seems like a crux that you have identified is how “sudden emergence” happens. How would a recursive self-improvement feedback loop start? Increasing optimisation capacity is a convergent instrumental goal. But how exactly is that goal reached? To give the most pertinent example - what would the nuts and bolts of it be for it happening in an ML system? It’s possible to imagine a sufficiently large pile of linear algebra enabling recursive chain reactions of both improvement in algorithmic efficiency, and size (e.g. capturing all global compute -> nanotech -> converting Earth to Computronium). Even more so since GPT-3. But what would the trigger be for setting it off?

Does the above summary of my take of this chime with yours? Do you (or anyone else reading) know of any attempts at articulating such a “nuts-and-bolts” explanation of “sudden emergence” of AGI in an ML system?

Or maybe there would be no trigger? Maybe a great many arbitrary goals would lead to sufficiently large ML systems brute-force stumbling upon recursive self-improvement as an instrumental goal (or mesa-optimisation)?

Responding to some quotes from the 80,000 Hours podcast:

“It’s not really that’s surprising, I don’t have this wild destructive preference about how they’re arranged. Let’s say the atoms in this room. The general principle here is that if you want to try and predict what some future technology will look like, maybe there is some predictive power you get from thinking about X percent of the ways of doing this involve property P. But it’s important to think about where there’s a process by which this technology or artifact will emerge. Is that the sort of process that will be differentially attracted to things which are let’s say benign? If so, then maybe that outweighs the fact that most possible designs are not benign.”

What mechanism makes AI be attracted to benign things? Surely only through human direction? But to my mind the whole Bostrom/Yudkowsky argument is that it FOOMs out of control of humans (and e.g. converts everything into Computronium as a convergent instrumental goal.)

“There’s some intuition of just the gap between something that’s going around and let’s say murdering people and using their atoms for engineering projects and something that’s doing whatever it is you want it to be doing seems relatively large.”

This reads like a bit of a strawman. My intuition for the problem of instrumental convergence is that in many take-off scenarios the AI will perform (a lot) more compute, and the way it will do this is by converting all available matter to Computronium (with human-existential collateral damage). From what I’ve read, you don’t directly touch on such scenarios. Would be interested to hear your thoughts on them.

“my impression is that you typically won’t get behaviours which are radically different or that seem like the system’s going for something completely different.”

Whilst you might not typically get radically different behaviours, in the cases where ML systems do fail, they tend to fail catastrophically (in ways that a human never would)! This also fits in with the notion of hidden proxy goals from “mesa optimisers” being a major concern (as well as accurate and sufficient specification of human goals).

A mixture of conversations and shared Google Docs. Nothing publicly citable as far as I know.

Thanks, I'll update the text when I get access to Metaculus again (I've blocked myself from it for productivity reasons lol)