How to generate research proposals

post by Jsevillamol · 2019-08-01T16:38:53.790Z · score: 80 (35 votes) · EA · GW · 5 comments

Contents

  Brainstorming ideas
  Curating ideas
  Operationalizing the proposals
  Conclusions
None
5 comments

Epistemic status: The advice I give is based on anecdotic experience, and what works well for me might not transfer well to others, but my past self would have found this article quite useful

As part of the application to FHI’s Summer Fellows Program I had to submit a research proposal, and once I finished my original proposal I had to engage again with the process to select my next research project.

In this article I will explain the particular approach I chose for this task, and provide some commentary on what worked well and what did not.

I expect you will get a lot of mileage out of this article if you are an early career researcher struggling to generate research ideas or if you are considering pursuing a career in research and would like to peek into how an important chunk of a researcher’s time is spent.

In short, my top advice for early career researchers is:

Generating and prioritizing research proposals seems to be a critical part of strategic research, and my informal impression is that systematic approaches are quite underexplored.

The overall process I used is split up in three chunks: generating research ideas, curating the ideas and operationalizing the ideas. We will cover each of these parts in separate sections of this article.

The outcome of this process is a detailed outline of a research project, which clearly explains the methodology and value of a project and that can be reused as an introduction to a potential publication.

The whole process took ~2 weeks. Progress was uneven, on some days I progressed a lot on the research proposal generation and others I just focused on other projects.

Brainstorming ideas

For the brainstorming phase I set up a brainstorming document where I collected questions that had drawn my attention, together with useful context and notes on what inspired the idea in a bullet point format.

To fill the list I resorted to the following strategies:

Reading research agendas proved to be quite fruitful. With research agendas I mostly refer to collections of open questions compiled by other researchers. If you already have a topic of interest you can search research agendas focused precisely on that; if not they can provide an excellent introduction to new areas of research.

Some examples of research agendas are Allan Dafoe’s AI Governance Research Agenda [REF], Luke Muehlhauser’s How to study superintelligence strategy [REF] or OPP’s Important unresolved research questions in macroeconomic policy [REF].

Talking to people and finding disagreements was also really useful, albeit it is harder to execute on demand. I had some really good research ideas finding common topics of interest with other researchers, finding out we disagreed about a particular decision we both cared about and finding the root of the disagreement.

The core advantage of this method is that it can lead you to realize some internalized insights that may be useful to share with others, and points of confusion. This also has the upside of leading to finding good project collaborators.

For example, my latest article can be traced back to a disagreement between my coauthor Pablo Moreno and me on the importance of quantum computing for AI safety reasons.

One downside of the talking method is that sometimes it is tempting to dismiss ideas. I partially solved this by having quite a low bar to put ideas in the brainstorming doc and treating the viewpoints of others as interesting evidence that I might be missing some important information about the world.

How I originally started brainstorming was creating a taxonomy of areas I was interested in (like AI Safety, Forecasting or Education) and then zooming in to particular topics (like Inverse Reinforcement Learning, how forecasting is used in policy or skill transfer). This was less useful than I expected but it was a good strategy to try overall.

This process ended up with ~30 research questions first time I started, plus I kept adding ideas afterwards. It was very useful to keep the ideas numbered for future reference and discussion.

One thing I would like to try in the future is to keep a log of interesting papers that I can then refer back to to think of possible improvements and questions that they inspire.

After compiling a low-bar list of research ideas, I invited people to my google doc to highlight the ones that seemed interesting to them and put comments about references. My primary goal was to test which ones passed the first test of attractiveness.

Based on the feedback I gathered, I started color coding the ideas, highlighting in green the ones that started to feel more exciting, in yellow the ones I wanted to think more about and in red the ones that I got discouraged from pursuing. I didn’t highlight the ones I wanted to seek more feedback on. The process was in flux, and several ideas changed colors several times.

The end result looked something like this:

Generating the original set of ideas took about ~4 days, and collecting feedback on them took ~4 days as well.

Curating ideas

The ideas that I became excited about by the end of the brainstorming and feedback process were transferred to a new document, the curation doc. Of the original ~30 research ideas, about ~15 made it to the second document.

I proceeded to think about characteristics that good research questions should possess. I came up with the following:

I scored each question on each characteristic, from LOW to MID to HIGH, leaving question marks on places I felt confused about how to score. I added a final column to summarize how I felt about a question after scoring it. This is how the score looked like in the end:

Overall I feel like I was trying to be too comprehensive with the characteristics I chose, and that the exercise would have been more useful if had I restricted myself to fewer categories. Of the categories above, the ones that felt more useful are tractability, actionability and organizational fit.

If I was doing it again I would consider splitting tractability into scope (having good, concrete and small research questions) and methodology (having clear strategies to tackle these questions); and I would merge personal and organizational fit in a single fit score.

After the table was done I checked again with myself how excited I would be about working on the ideas. I felt I was still excited about ~6 of the projects, so I picked them up and developed them further.

In order to do that I used the following bullet point template:

I was fairly liberal on deciding which parts of the template I expanded on for each question and which ones were kept brief or even empty.

When deciding what parts to expand on and which ones to keep brief it was useful to think in terms of what were my uncertainties around the projects (Maybe this question is really hard to answer?) and zero in on them (Can I outline a methodology in 5 minutes?).

At this point, I sought further feedback from others. This time I was primarily seeking feedback that would allow me to develop the methodology further or discard projects before investing too much effort in them.

I sought feedback via two strategies: sending the document to people and asking for general comments and seeking out people I knew were working on some particular ideas related to mine and questioning them / exposing my ideas to them.

As I received feedback I started color coding them by how excited I was, using the same conventions as before: RED for discouragement, YELLOW for further thought needed, GREEN for excitement and no highlight for further feedback needed.

This is what the ideas looked like at this stage:

Scoring the ideas took <1 day, and writing the first draft of the templates took ~2 days.

Operationalizing the proposals

After seeking feedback, of the ~6 ideas that I developed a bit further I was still excited about pursuing ~3 of them.

For each of these ~3, I scheduled conversations with other fellows and researchers with overlapping interests so we could talk at length about them.

The conversations took different shapes, from them telling me about their projects and reasons to pursue them to having MIRI-style deconfusing sessions where I tried making progress on the questions [REF].

All insights and references I got were used to improve the bullet point template of the previous section.

Every so often I would try to write a formal proposal. The goal of this exercise would be to defend the value of the project to a hypothetical board deciding if they would fund my project.

The proposal would be short (less than one page long), and include in redacted form the important points in the bullet-point template.

If the proposal felt hard to write or wrong in any sense, I would stop to check where that feeling was coming from. Was my methodology incomplete? Did I feel there was low hanging fruit information from scheduling another meeting? Should I try just solving the question with a crude model and see if I was still excited? Or was I simply tired and should take a break?

One unexpected benefit of this exercise is that I was able to copy with very little modification my original research proposal and reuse it as an introduction to the research article I subsequently coathored.

The resulting proposals would look something like this:

In order to make the final decision, it was helpful to allocate time to the process. That is, I explicitly decided I would dedicate two weeks to choosing a good research question. While I did make my decision before the deadline, the commitment pushed me towards seeking progress earlier.

The process of me making the final decision looked like me realizing that I was already making substantial progress on one particular project.

Having the conversations and playing around with the ideas until I wrote a concrete proposal took ~4 days.

Conclusions

In this piece I have exposed how I generated research proposals for the FHI Summer Fellows Program. The process consisted of three phases: brainstorming research ideas, curating them and operationalizing the proposals.

I think that the process of methodically thinking about research idea generation and prioritization has helped me develop a better research taste, and I would recommend it to people who are starting to do research. I have however an intuition that as your research aesthetic matures following explicit procedures becomes less useful versus just following your instinct.

The most salient benefit of engaging with this project has been creating a pipeline for noting down and developing research ideas that I expect to use and refine going forward. Doing it deliberately and explicitly has also lead to some interesting conversations with the other summer fellows and the rest of FHI’s staff.

Some concrete feedback I would give myself about the research generation pipeline, with the benefit of hindsight:

Thinking in terms of research questions and concrete methodology to answer them instead of in terms of areas I want to work on is probably the biggest learning of my experience doing research over the last year.

Reading research agendas and papers while having a document opened where I could dump research questions was more productive than trying to conjure research ideas from an empty page. Furthermore, the latter method did not lead to substantially more original ideas anyway.

Talking to people was very useful, and I should have done that earlier and more often. The commenting features of google docs are amazing.

Keeping the brainstorming doc and the curation doc separately was useful so I could selectively share them with different people for different levels of feedback.

Overall I think that scoring the ideas on the different characteristic was not that useful out of itself, but the process of thinking how do good ideas look like helped me develop a better taste for good research. Going forward I expect I will not emphasize such a methodical approach to sorting the ideas versus just talking to others. On the other hand, it might have been more useful had I restricted myself to fewer categories.

Allocating explicit time and setting a concrete goal for this process was useful, and made me seek progress earlier.

Hopefully this article has helped you reflect about how to create research questions and you can take away some ideas of things to try. Let me know your thoughts in the comments!

This article was written by Jaime Sevilla. I want to thank Linh Chi Nguyen for useful feedback and editing.

5 comments

Comments sorted by top scores.

comment by Jsevillamol · 2019-08-07T18:37:50.312Z · score: 4 (3 votes) · EA · GW

As further reading I recently came across Research as a Stochastic Decision Process, which discusses another systematic approach to research.

Summary copy pasted from the article:


Summary

Many of our default intuitions about how to pursue uncertain ideas are counterproductive:

  • We often try easier tasks first, when instead we should try the most informative tasks first.
  • We often conflate a high-level approach with a low-level instantiation of the approach.
  • We are often too slow to try to disprove our own ideas.

Building frameworks that reify the research process as a concrete search problem can help unearth these incorrect intuitions and replace them with systematic reasoning.

comment by [deleted] · 2019-08-06T11:55:31.392Z · score: 4 (3 votes) · EA · GW

Thanks for this! I agree that my past self would have found this, or something along those lines, a very useful starting point. That said, I think I would also have found this process a little overwhelming (esp during initial projects, when I was trying to assess personal fit more than anything else). So here's a quote from the GovAI agenda which I find useful whenever I get lost: _"Ultimately, perhaps the simplest rule of thumb is to just begin with those questions or ideas that most grab you, and start furiously working."_ (p.13)

comment by gavintaylor · 2019-08-22T12:18:59.078Z · score: 3 (2 votes) · EA · GW

Nice, I particularly like the table and bullet-point forms you used for curating your ideas - I often find myself with too many ideas to work on and this seems like a good way to take an objective overview.

During my PhD I read 'Becoming a successful scientist' - this presented a strategic approach to scientific discovery and problem selection (Section 3.1) that I haven't really seen elsewhere. It focused on science, but the ideas of looking for contradictions, paradoxes, new viewpoints or different scales may also be helpful for generating research questions in philosophy/economics.

I have a PDF of the book I'm happy to send by email, PM me.

comment by John_Maxwell_IV · 2019-08-03T06:07:31.716Z · score: 3 (2 votes) · EA · GW

Generating and prioritizing research proposals seems to be a critical part of strategic research, and my informal impression is that systematic approaches are quite underexplored.

This is also my impression.

I guess some people will probably be just as interested in pursuing the research ideas of others as pursuing their own. For those people, maybe creating a thread here or on Facebook with a message like "I'm looking for research ideas on Topic X" would work?

Personally, I've been noting down research ideas on EA topics that interest me (AI safety and improved institutions would be the two big ones I guess) for quite a while, and I'm pursuing the ideas at a much lower rate than I'm noting them down! So maybe it'd be good for me to connect with people who are hunting for ideas somehow?

comment by Jsevillamol · 2019-08-03T08:41:40.781Z · score: 1 (1 votes) · EA · GW

Totally second the motion of empowering junior researchers to ask for research ideas here and somewhere else.

Also, I'd encourage you to write down your own research agenda in the form of a blogpost listing some open questions in this forum!

It will be useful for other researchers and you will get interesting feedback on your ideas :)