Bottlenecks and Solutions for the X-Risk Ecosystem

post by FlorentBerthet · 2018-10-08T12:47:42.068Z · score: 34 (28 votes) · EA · GW · Legacy · 14 comments

Contents

  Background
  Summary 
    Conclusion for Solution Area 1 (optimise the senior talent pipeline)
    Conclusion for Solution Area 2 (optimise the project pipeline)
  Who should read this report?
      Solution Area 1 (optimise the senior talent pipeline):
      Solution Area 2 (optimise the project pipeline): 
  Prioritisation Uncertainties
      Confidence in our Findings
      Choice of Solution Areas 
      Causal Relationships Between Solution Areas
  Overview of Bottlenecks
      Credibility and Prestige
      Talent Gap
      Funding Gap
      Coordination Gap 
      Policy Implementation Gap
  Recommended next steps
      1. To onboard more senior talent:
      2. To launch more projects with a high positive expected value (EV):
    What we are doing to make progress 
    What can you do to help?
None
12 comments

This is a summary of a report we wrote to clarify the bottlenecks of the x-risk ecosystem in a pragmatic solution-oriented manner, outlining some of their top-level contours and possible solutions. The report focuses on what we consider to be the two current central bottlenecks: lack of senior talent and lack of projects that have a high positive expected value.

 

Background

This is the culmination of a strategic needs assessment of the x-risk ecosystem, which included a detailed literature review and interviews with 20+ individuals from across the x-risk ecosystem (including staff at FHI, CEA, CHAI and CSER, aspiring x-risk talent, and leaders of independent x-risk mitigation projects).  

Insights gleaned from interviews have been kept anonymous, to protect the privacy of the interviewees. Our literature review identified relatively scarce quantitative data with regards to the x-risk ecosystem. Our recommendations thus stem from the subjective analysis of largely qualitative data, much of which cannot be cited. Footnotes and details of our methodology are included in the appendices of the full report.

While the main bottlenecks are already well-known within the community, we think this report can be valuable by providing additional details and concrete action plans.

This was a voluntary self-study project conducted by principal investigators:

With strategic support from Konrad Seifert and Max Stauffer, co-founders of EA Geneva.

 

Summary 

X-risk mitigation is a fledgling field; there is a lot of work to be done, many people motivated to help, and extensive funding available. So why is more not being done now? What is holding back the progress of the field, and how can we overcome these obstacles?

We identified four key solution areas to optimise the x-risk ecosystem. In order of priority, with highest priority first:

  1. Optimise the senior talent pipeline
  2. Optimise the pipeline of projects with a high positive expected value (EV)
  3. Bridge the policy implementation gap
  4. Optimise the junior talent pipeline

 

Our analysis found that 1 and 2 currently limit 3 and 4; in order to make good progress on policy work and to onboard junior talent effectively, we need more senior talent and more projects with a high EV.

On top of this double-bottleneck, the field of x-risk mitigation and some sub-fields, including AI safety, are still at risk of failure due to insufficient credibility. This is an important consideration when reasoning about how the field should make progress.

We recommend ongoing work on all four solution areas; balanced progress on all these fronts is necessary for optimum long term growth of the field. For the sake of brevity and prioritisation, this report focuses on solutions to the most immediate problems, the current lack of senior talent and lack of projects that have a high positive expected value (EV+).

The x-risk ecosystem is young, rapidly evolving, and has lots of room for optimisation. The most critical needs right now are to expedite the senior talent pipeline and the project pipeline. In both cases, the solution requires professionals with specialist skills to make careful implementation plans, in consultation with x-risk leaders and the broader x-risk community.

 

Conclusion for Solution Area 1 (optimise the senior talent pipeline)

The lack of senior talent is currently the central bottleneck in the x-risk ecosystem. If suitable resources are dedicated to this challenge right away, we suspect that senior talent will remain the central bottleneck for no more than 3 years, and possibly a lot less. Many of the most promising ways to expedite this pipeline are highly neglected and fairly tractable; in the longer term, the x-risk community should be able to attract all the senior talent it needs.

In the short term, senior hires are most likely to come from finding and onboarding people who already have the required skills, experience, credentials and intrinsic motivation to reduce x-risks. This should include investment in an experienced professional services team to develop and help implement high quality branding, marketing, recruitment, talent management and partnership strategies for x-risk organisations and the community as a whole.

  

Conclusion for Solution Area 2 (optimise the project pipeline)

Expediting the pipeline for projects with a high positive EV is a very important, neglected, and tractable way to help mitigate x-risks.

We can increase the x-risk community’s standards of project planning and evaluation by creating user-friendly guidelines to develop high EV+ projects. We can matchmake top project ideas with top talent, with the help of a unified searchable priority project database. Community members could effectively develop promising project ideas from the project database, assisted by resources, events, and a dedicated project incubation team. We can partially automate and scale the project pipeline with an online project incubation platform, complete with step-by-step project planning guidelines and a project self-scoring system.

The project self-scoring system and project database would provide a robust, data-rich foundation for ongoing optimisation of the project pipeline. Feedback from potential users may result in a very different implementation path to what has been presented in this report.

 

Who should read this report?

 

This report is for both x-risk experts and aspiring x-risk talent. The following sections are particularly relevant to the following audiences:  

Solution Area 1 (optimise the senior talent pipeline):

Solution Area 2 (optimise the project pipeline): 

 

Prioritisation Uncertainties

 

Confidence in our Findings

We are confident that the general analysis and conclusions in this report are largely correct. We are also very confident that this report misses lots of subtleties and nuances, which are important to clarify in order to get the details right when it comes to implementing solutions.

 

Choice of Solution Areas 

We are aware that our four solution areas are not a comprehensive representation of the x-risk ecosystem, and we are unclear to what extent we may be missing other important factors.

We suspect that these four areas are sufficiently broad and overarching that most other important dimension of optimising the x-risk ecosystem would be implicitly entailed in their scope. Nonetheless, we may be neglecting some important dimensions due to our framing of the solution areas, for example:

 

Causal Relationships Between Solution Areas

Our prioritisation tries to take into account the complex causal relationships between each solution area, however, our analysis has probably missed some important feedback dynamics between these different solution areas, especially as they play out over the longer term.

 

Overview of Bottlenecks

 

Credibility and Prestige

 

Talent Gap

 

Funding Gap

 

Coordination Gap 

 

Policy Implementation Gap

 

Recommended next steps

 

1. To onboard more senior talent:

Set up a 12 month project to enable leading x-risk organisations to establish robust internal systems to effectively attract and retain senior talent. This would include:

  1. HR strategy
  2. Brand strategy (for AI safety, x-risk and individual organisations’ brands)
  3. Partnerships strategy

 

2. To launch more projects with a high positive expected value (EV):

Set up a 12 month project to enable leading x-risk organisations to establish robust project incubation infrastructure:

  1. Write clear detailed guidelines for what gives a project high positive EV
  2. Set up a centralised database of the most promising project ideas
  3. Create and train a project incubation team
  4. Consult with x-risk funders and partners to matchmake top talent with top projects

This work would increase the rate at which new projects with high positive EV get started.

 

What we are doing to make progress 

As part of a team of five, we have established a group of EA-aligned professional consultants, with the skills to implement the above recommendations. Together we created an EA consultancy called Good Growth (website).

We have already started helping some EA orgs. On the long term, contingent on funding and expert feedback, we intend to conduct ongoing strategic research and consultancy work across all four solution areas to:

 

What can you do to help?

To explore any of these options further, please email us directly:

For a more detailed picture of our recommendations, see the full report here.

 

14 comments

Comments sorted by top scores.

comment by Elityre · 2018-10-08T14:57:26.738Z · score: 12 (8 votes) · EA(p) · GW(p)

In the short term, senior hires are most likely to come from finding and onboarding people who already have the required skills, experience, credentials and intrinsic motivation to reduce x-risks.

Can you be more specific about the the required skills and experience are?

Skimming the report, you say "All senior hires require exceptionally good judgement and decision-making." Can you be more specific about what that means and how it can be assessed?

comment by Elityre · 2018-10-08T15:09:44.601Z · score: 13 (11 votes) · EA(p) · GW(p)

It seems to me that in many cases the specific skills that are needed are both extremely rare and not well captured by the standard categories.

For instance, Paul Christiano seems to me to be an enormous asset to solving the core problems of AI safety. If "we didn't have a Paul" I would be willing to trade huge amounts of EA resources to have him working on AI safety, and I would similarly trade huge resources to get another Paul-equivalent working on the problem.

But it doesn't seem like Paul's skillset is one that I can easily select for. He's knowledgeable about ML, but there are many people with ML knowledge (about 100 new ML PhDs each year). That isn't the thing that distinguishes him.

Nevertheless, Paul has some qualities, above and beyond his technical familiarity, that allow him to do original and insightful thinking about AI safety. I don't understand what those qualities are, or know how to assess them, but they seem to me to be much more critical than having object level knowledge.

I have close to no idea how to recruit more people that can do the sort of work that Paul can do. (I wish I did. As I said, I would give up way more than my left arm to get more Pauls).

But, I'm afraid there's a tendency here to goodhart on the easily measurable virtues, like technical skill or credentials.

comment by AdamGleave · 2018-10-09T03:20:49.348Z · score: 20 (11 votes) · EA(p) · GW(p)

There aren't many people with PhD-level research experience in relevant fields who are focusing on AI safety, so I think it's a bit early to conclude these skills are "extremely rare" amongst qualified individuals.

AI safety research spans a broad range of areas, but for the more ML-oriented research the skills are, unsurprisingly, not that different from other fields of ML research. There are two main differences I've noticed:

  • In AI safety you often have to turn ill-defined, messy intuitions into formal problem statements before you can start working on them. In other areas of AI, people are more likely to have already formalized the problem for you.
  • It's important to be your own harshest critic. This is cultivated in some other fields, such as computer security and (in a different way) in mathematics. But ML tends to encourage a sloppy attitude here.

Both of these I think are fairly easily measurable from looking at someone's past work and talking to them, though.

Identifying highly capable individuals is indeed hard, but I don't think this is any more of a problem in AI safety research than in other fields. I've been involved in screening in two different industries (financial trading and, more recently, AI research). In both cases there's always been a lot of guesswork involved, and I don't get the impression it's any better in other sectors. If anything I've found screening in AI easier: at least you can actually read the person's work, rather than everything behind behind an NDA (common in many industries).

comment by Ben Pace · 2018-10-09T06:53:53.965Z · score: 7 (6 votes) · EA(p) · GW(p)

Identifying highly capable individuals is indeed hard, but I don't think this is any more of a problem in AI safety research than in other fields.

Quite. I think that my model of Eli was setting the highest standard possible - not merely a good researcher, but a great one, the sort of person who can bring whole new paradigms/subfields into existence (Kahneman & Tversky, Von Neumann, Shannon, Einstein, etc), and then noting that because the tails come apart (aka regressional goodharting), optimising for the normal metrics used in standard hiring practices won't get you these researchers (I realise that probably wasn't true for Von Neumann, but I think it was true for all the others).

comment by Elityre · 2019-11-14T02:27:41.973Z · score: 1 (1 votes) · EA(p) · GW(p)

I like the breakdown of those two bullet points, a lot, and I want to think more about them.

Both of these I think are fairly easily measurable from looking at someone's past work and talking to them, though.

I bet that you could do that, yes. But that seems like a different question than making a scalable system that can do it.

In any case, Ben articulates the view that generated the comment above, above.

comment by toonalfrink · 2019-01-14T15:25:44.168Z · score: 3 (1 votes) · EA(p) · GW(p)

How about this: you, as someone already grappling with these problems, present some existing problems to a recrutee, and ask them to come up with some one-paragraph descriptions of original solutions. You read these, and introspect whether they give you a sense of traction/quality, or match solutions that have been proposed by experts you trust (that they haven't heard of).

I'm looking to do a pilot for this. If anyone would like to join, message me.

comment by oliverbramford · 2018-10-09T12:17:21.615Z · score: 5 (5 votes) · EA(p) · GW(p)

The required skills and experience of senior hires vary between fields and roles; senior x-risk staff are probably best-placed to specify these requirements in their respective domains of work. You can look at x-risk job ads and recruitment webpages of leading x-risk orgs for some reasonable guidance. (we are developing a set of profiles for prospective high-impact talent, to give a more nuanced picture of who's required).

"Exceptionally good judgement and decision-making", for senior x-risk talent, I believe requires:

  • a thorough and nuanced understanding of EA concepts and how they apply to the context

  • good pragmatic foresight - an intuitive grasp of the likely and possible implications of one's actions

  • a conscientious risk-aware attitude, with the ability to think clearly and creatively to identify failure modes

Assessing good-judgement and decision-making is hard; it's particularly hard to assess the consistency of a person's judgement without knowing/working with them over at least several months. Some methods:

  • Speaking to a person can quickly clarify their level of knowledge of EA concepts and how they apply to the context of their role.

  • Speaking to references could be very helpful, to get a picture of how a person updates their beliefs and actions.

  • Actually working with them (perhaps via a work trial, partnership or consultancy project) is probably the best way to test whether a person is suitable for the role

  • A critical thinking psychometric test may plausibly be a good preliminary filter, but is perhaps more relevant for junior talent. A low score would be a big red flag, but a high score is far from sufficient to imply overall good judgement and decision-making.

comment by PeterMcIntyre · 2018-10-10T23:21:12.538Z · score: 11 (7 votes) · EA(p) · GW(p)

Thanks for writing this!

Just wanted to let everyone know that at 80,000 Hours we’ve started headhunting for EA orgs and I’m working full-time leading that project. We’re advised by a headhunter from another industry, and as suggested, are attempting to implement executive search best practices.

Have reached out to your emails listed above - looking forward to speaking.

Peter

comment by RomeoStevens · 2018-10-13T07:05:56.022Z · score: 4 (4 votes) · EA(p) · GW(p)

Was whether senior talent should have better access to high quality executive assistants explored?

Actually in general if there were any negative findings that is probably useful too.

comment by Sean_o_h · 2018-10-15T11:56:13.951Z · score: 2 (2 votes) · EA(p) · GW(p)

I can say anecdotally that at different points, access to excellent executive assistants (or people effectively functioning in such roles) has been hugely helpful for me and other folks in xrisk leadership positions I've known.

comment by Evan_Gaensbauer · 2018-10-17T23:48:56.205Z · score: 2 (2 votes) · EA(p) · GW(p)

Upvoted.

Questions:

  1. What's the definition of expertise in x-risk? Unless someone has an academic background in a field where expertise is well-defined by credentials, there doesn't appear to be any qualified definition for expertise in x-risk reduction.

  2. What are considered the signs of a value-misaligned actor?

  3. What are the qualities indicating "exceptionally good judgement and decision-making skills" in terms of x-risk reduction orgs?

  4. Where can we find these numerous public lists of project ideas produced by x-risk experts?

Comments:

  1. While 'x-risk' is apparently unprecedented in large parts of academia, and may have always been obscure, I don't believe it's unprecedented in academia or in intellectual circles as a whole. Prevention of nuclear war and and once-looming environmental catastrophes like the ozone holes posed arguably existential risks that were academically studied. The development of game theory was largely motivated by a need for better analysis of war scenarios between the U.S. and Soviet Union during the Cold War.

  2. An example of a major funder for small projects in x-risk reduction would be the Long-Term Future EA Fund. For a year its management was characterized by Nick Beckstead, a central node in the trust network of funding for x-risk reduction, not providing much justification for grants made mostly to x-risk projects the average x-risk donor could've very easily identified themselves. The way the issue of the 'funding gap' is framed seems to imply patches to the existing trust network may be sufficient to solve the problem, when it appears the existing trust network may be fundamentally inadequate.

comment by oliverbramford · 2018-10-26T09:07:20.791Z · score: 1 (1 votes) · EA(p) · GW(p)

1. I don't have a definition of x-risk expertise. I think the quality of x-risk expertise is currently ascribed to people i) with a track record of important contributions to x-risk reduction ii) subjective peer approval from other experts.

I think a more objective way to evaluate x-risk expertise would be extremely valuable.

2. Possible signs of a value mis-aligned actor:

if they don't value impact maximisation, they may focus on ineffective solutions, perhaps based on their interests

if they don't value high epistemic standards, they may hold beliefs that they cannot rationally justify, and may make more avoidable bad/risky decisions

if they don't value the far future, they may make decisions that hare high risk for the far future

3. see http://effective-altruism.com/ea/1tu/bottlenecks_and_solutions_for_the_xrisk_ecosystem/foo

I also think good judgement and decision-making results from a combination of qualities of the individual and qualities of their social network. Plausibly, one could make much better decisions if they have frequent truth-seeking dialogue with relevant domain experts with divergent views.