AI Safety Needs Great Engineers

post by Andy Jones · 2021-11-23T21:03:41.486Z · EA · GW · 5 comments


  Why engineers?
  What kind of engineers?
  How does engineering compare to research?
  Should I apply?
  Should I skill up?
  Take homes

Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.

I work for Anthropic, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.

I've spoken to several other AI safety research organisations who feel the same.

Why engineers?

May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.

As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.

In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.

What kind of engineers?

Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are

This is not a requirements list though. Based on the people working here already, 'great software engineer' and 'easy to get on with' are hard requirements, but the things in the list above are very much nice-to-haves, with several folks having just one or none of them. 

Right now our job listings are bucketed into 'security engineer', 'infrastructure engineer', 'research engineer' and the like because these are the noun phrases that a lot of the people we like identify themselves with. But what we're actually most concerned about are generally-great software engineers who - ideally - have some extra bit of deep experience that we lack. 

How does engineering compare to research?

At Anthropic there is no hard distinction between researchers and engineers. Some other organisations retain the distinction, but the increasing reliance of research on substantial, custom infrastructure is dissolving the boundary at every industrial lab I'm familiar with. 

This might be hard to believe. I think the archetypal research-and-engineering organisation is one where the researchers come up with the fun prototypes, and then toss them over the wall to the engineers to clean up and implement. I think the archetype is common enough that it dissuades a lot of engineers from applying to engineering roles, instead applying to research positions where they - when evaluated on a different set of metrics than the ones they're best at - underperform.

What's changed in modern AI safety is that the prototypes now require serious engineering, and so prototyping and experimenting is now an engineering problem from the get-go. A thousand-line nested for-loop does not carry research as far as it once did.  

I think this might be a hard sell to folks who have endured those older kinds of research organisations, so here are some anecdotes:

Should I apply?

It's hard to judge sight-unseen whether a specific person would suit AI safety engineering, but a good litmus test is the one given at the top of this post:

With a few weeks' work, could you - hypothetically! - write a new feature or fix a serious bug in a major ML library? 

Are you already there? Could you get there with a month or two of effort? 

I like this as a litmus test because it's very close to what my colleagues and I do all day. If you're a strong enough engineer to make a successful pull request to PyTorch, you're likely a strong enough engineer to make a successful pull request to our internal repos. 

Actually, the litmus test above is only one half of the actual litmus test I give folk that I meet out and about. The other half is 

Tell me your thoughts on AI and the future.

with a pass being a nuanced, well-thought-out response. 

Should I skill up?

This post is aimed at folks who already can pass the litmus test. I originally intended to pair it with another post on skilling up to the point of being able to pass the test, but that has turned out to be a much more difficult topic than I expected. For now, I'd recommend starting with 80k's software engineering guide.

Take homes

We want more great engineers.

If you could write a pull request for a major ML library, you should apply to one of the groups working on empirical AI safety: Anthropic, Cohere, DeepMind Safety, OpenAI Safety and Redwood Research

If that's not you but you know one or more great engineers, ask them if they could write a pull request for a major ML library. If yes, tell them to apply to Anthropic. 

If that's not you but you'd like it to be, watch this space - we're working on skilling up advice.

This post is twinned with the same one on LessWrong [LW · GW]


Comments sorted by top scores.

comment by Yonatan Cale (hibukki) · 2021-11-23T17:34:18.532Z · EA(p) · GW(p)

Hey, I think I'm a good software developer, but I have roundable-to-zero knowledge in ML.

Can you help me estimate how far I am from the bar of being useful to such a project? [Update: I'm consulting with friends about this]

It would also be useful for me to understand the requirements well since lately I talk to other engineers [EA · GW] who are sometimes interested in working on AI Safety if they can


Replies from: Andy Jones
comment by Andy Jones · 2021-11-25T18:40:58.010Z · EA(p) · GW(p)

I'm hoping to have some comprehensive advice out on this Soon(TM), but my sense is that if you're familiar with multivariable calculus/linear algebra/probability then it's a few months' full-time-work-equivalent to skill up in ML to the point where it's no longer the constraining factor in being hired as an ML engineer. 

If you're not familiar with multivar calc/linear algebra/probability then it's a bit of a longer path and a harder one to predict, since I think ease-with-maths varies more widely than ease-with-ML.

Replies from: hibukki
comment by Yonatan Cale (hibukki) · 2021-11-27T19:51:41.927Z · EA(p) · GW(p)

Thank you



I asked a few friends about this and one of them decided to apply himself, so anyway I think this is turning out well. :)

comment by Benjamin_Todd · 2021-11-24T12:16:18.362Z · EA(p) · GW(p)

+1 to this!

If you're a software engineer considering transitioning into AI Safety, we have a guide about how to do it, and attached podcast interview.

There are also many other ways SWE can use their skills for direct impact, including in biosecurity and by transitioning into information security, building systems at EA orgs, or in various parts of govt.

To get more ideas, we have 180+ engineering positions on our job board.

comment by MathiasKB (MathiasKirkBonde) · 2021-11-29T08:50:02.821Z · EA(p) · GW(p)

Fantastic article, I ended up sending this to quite a few friends who are great programmers and considering careers in AI Safety.

I think the litmus test is great, we need more like this!

If someone is unsure about whether they are able to make a significant contribution to an AI library, they have to be able to come up with an idea for a meaningful contribution that they can implement. That requires extensive knowledge of the AI library and its shortcomings. Is this part of the litmus test?

If not, it might make sense for AI safety engineers to publicize a  difficult toy-problem or two that would allow a prospective candidates to immediately get a sense of their abilities, without needing to think too much about what type of pull-request they would do.