Retrospective on Catalyst, a 100-person biosecurity summit

post by Tessa (tessa), Spiracular, Fin, Jeffrey Ladish (landfish), Brian Wang (brianwang712), decodyng · 2021-05-26T13:10:22.942Z · EA · GW · 11 comments

Contents

  tl;dr
  How did the event go?
    Outcomes from the day of the summit
    Additional outcomes
  Should other people run a similar event?
  Detailed Notes and Advice
    Timeline
      Full Timeline
        2018
        2019
        2020
    Budget
      Expected vs Actual Budget
    Programming and Event Structure
    Venue, Food and other Logistics
    Recruiting Speakers
    Recruiting and Communicating with Participants
    Organizing Team
      Getting help and mentorship
      Managing meetings and information
    Other Resources
None
11 comments

This is a retrospective on Catalyst, a one-day collaborative biosecurity summit that was held in February 2020 in the San Francisco Bay Area, written by the organizers[1] for anyone who might run a similar event in the future.

tl;dr

  1. It is possible for a small group of volunteers to run a high-quality conference without making it any of their full-time jobs, but give yourself 8 to 10 months to bring an event of this scale from concept to execution. (See: Timeline.)
  2. Define the outcomes you want from your event early on. This will shape so many decisions, from contacting advisors to choosing a venue. Check out Gather for a good step-by-step guide. That said, measuring outcomes is really hard!
  3. There is an unmet need for non-hierarchical and connection-focused conferences. We prioritized discussion, shared problem-solving, and networking between participants, and, from the feedback we received, we think these efforts contributed to more dynamic conversations and a stronger sense of community. (See: How did the event go? and Programming and Event Structure).
  4. Experienced advisors are great. You don’t need a lot of their time to get huge benefits; a few 1-hour meetings and permission to put their names on the website was enough (see: Getting help and mentorship).
  5. If you have funding for the conference you should try to understand the tax implications before starting to spend money. This worked out fine for us, but could easily not have (see: Budget).
  6. Leave slack in your budget, timeline, and day-of program. We made incorrect predictions about how long things would take (especially booking speakers and finding a venue) and about how much things would cost (especially catering). (See: Timeline and Expected vs Actual Budget, which we hope will help future organizers to make better predictions).

How did the event go?

We ran Catalyst because we wanted to create a venue for discussing biosecurity in the SF Bay Area. The theme of the summit was that biotechnology is advancing quickly, many different futures seem possible, and that we must thoughtfully collaborate in order to ensure that we get a positive future.

Two photographs from Catalyst. The first is of a whiteboard with the words "Draw Your Connections" at the top. Instant photos of people are on the board, and lines are drawn between them and various words on the board (e.g. "synthetic biology"). The second photo is a group photo of all the attendees standing together.

Catalyst was attended by about 100 people representing various levels of engagement with biosecurity (from undergrads to experienced biosecurity professionals) across several different communities (effective altruists, biohackers, academics, industrial biotechnologists, and policymakers), with a slight preference for those coming from the SF Bay Area in order to facilitate continued post-Catalyst momentum. The event was funded by a grant from the Long Term Future Fund [? · GW].

Outcomes from the day of the summit

We did an immediate post-event survey to gather outcomes from the event. We intended to run an additional six-month follow-up survey, but then 2020 happened.

We don’t feel totally satisfied with the robustness of our outcome data, but also aren’t sure what a solid methodology for assessing event impact would have been. Participants seem to have considered the event a good use of their time; when asked if they would attend a similar event in SF again, 30/35 respondents said “Yes”, and the remainder said “Maybe”. One experienced organizer told us that Net Promoter Score was a standard assessment: our NPS was 74[2], but to be honest we’re not sure if that’s a good score or not. (Edited to add: according to a comment from Ben West [EA(p) · GW(p)] this is quite a good score.)

Two photographs from Catalyst. In the first, a group of people sits around a table, seeming to be engaged in conversation. In the second, two people in a Long Conversation sit in chairs facing  each other and speaking.

In both the survey and while talking to participants, we heard that Catalyst exceeded expectations. Many people shared enthusiastic positive feedback with organizers during the reception, and the energy in the room was very high. Participants appreciated how small the summit was, how many cool people were there, the exciting conversations, and the diverse backgrounds of attendees and speakers. Some illustrative quotes:

A few more concrete outcomes, each reported by a different participant:

Additional outcomes

The main outcome of Catalyst was the day-of experience for the participants. A few secondary outcomes:

We were hoping to have more post-event outcomes, but several difficult-to-anticipate things got in the way of this:

We think a counterfactual version of this event, in which the two things above did not happen, would have had greater impacts beyond the day-of experience of the participants.

Should other people run a similar event?

Basically yes. We think the event provided good value for time and money.

Time: organizing Catalyst was the main side project for several of the volunteer organizers for several months, but it wasn’t a ridiculous amount of work. A rough estimate is that we spent between 350 and 500 hours on the event, split across the 6 organizers. See more details under Organizing Team below.

Money: a much cheaper event could be run either online or with in-kind offers from an academic venue… but we think there’s quite a lot of value in having a nice venue (to put people in a novel, connection-oriented mood) and providing good food and drink (so people stay in the venue and talk with each other). See more details under Budget below.

One reason we might not recommend running a similar event is if your team has only minimal connections in the problem area you’re focused on. We’re not saying you need to be super well connected, but you should hit the threshold of “knows someone who knows someone’s email” for at least some of the speakers you’d want to invite, and having accessible, well-connected advisors is really helpful (see more details under Recruiting Speakers and Getting help and mentorship).

Detailed Notes and Advice

This section is long, and may not be interesting for anyone not trying to organize a similar event! You have been warned. We categorized each section as a success. near miss, or failure.

Timeline

Our timeline was a near miss. That is, nothing went wrong, but if we hadn’t for external reasons needed to reschedule, the event likely would have gone poorly. Our big mistakes were:

It’s also a near miss (though one entirely beyond our control) that we’d have cancelled Catalyst if it had been scheduled a one week later. As it was, Catalyst happened 4 days before the first reported community spread of COVID-19 in the USA and 22 days before the SF Bay Area went under a shelter-in-place order. It’s a little weird to have hosted an event on biosecurity and pandemic preparedness in late February 2020, haha??

Full Timeline

2018

2019

2020

Budget

Success in terms of handling available funds, but near miss when it comes to taxes. Organization and a conservative buffer were important to the former. If we could do things again, we’d clarify our tax situation (an individual organizer receiving a multi-thousand dollar grant) earlier on with our funders. Our general budget advice:

Expected vs Actual Budget

Our original budget was put together with advice from our advisors and other event organizers we knew, but it ended up being incorrect in a few ways.

Image of table comparing expected vs. actual budget. Expected cost for venue, AV and videography was $9000, which was $600 too low. Expected cost for food an drink was $4000, which was $7,765 too low. Expected cost for speaker travel and lodging was $16,000, which was $13,730 too high.

Programming and Event Structure

Our program and event structure was a success, especially the conversations between individuals. The event had the following kinds of structured programming:

You can see the complete schedule in this PDF. As the schedule PDF shows, we had a few bookable spaces in the venue, but in practice participants just talked informally with each other. Here is our main programming advice:

Venue, Food and other Logistics

Our conference logistics were a success. We think logistics are really important, since even if a conference goes well intellectually, if it’s chaotic on the day of, or if the food is bad, that can dominate peoples’ experience. We were pleased to have a number of people day-of who came up to the organizers and said they were impressed with how smooth and problem-free the day as a whole was for them.

We think our venue and food were solidly a success, despite a few minor issues. AV was a failure in its outcome, though it’s not necessarily clear what we should have done to prevent that. Onto some more detailed notes:

Recruiting Speakers

We’re really happy with the group of people who ended up speaking at Catalyst (success) but we had a near miss with cancellations (almost 20% of our invited speakers had to cancel somewhat last minute). Thankfully, we had a lot of flexibility in both our venue and schedule.

When deciding who to invite, we thought about the balance of perspectives we wanted (e.g. covering industry and academia, ideally local) and specific topics we wanted to cover. Our advisors were extremely helpful at this stage, as they often knew of good speakers who would provide desired perspectives. We also made a long list of relevant organizations (mostly local ones) and looked up people who worked at those organizations to contact.

Some additional advice on recruiting speakers:

Recruiting and Communicating with Participants

Overall, we feel happy with the participants who were present at Catalyst (success) and no one seemed terribly confused by our communications (success?). We sent a lot of emails in the few weeks leading up to the summit, with various actions for participants to take, and that seemed like a good way to build enthusiasm / engagement.

Organizing Team

Catalyst was organized by a team of six people (Tessa Alexanian, Cody Wild, Finan Adamson, Jeffrey Ladish, Megan Crawford, and Brian Wang), helped by three advisors (Megan Palmer, Kevin Esvelt, and Jun Axup). We estimate that we spent between 350 and 500 hours organizing the event, split unevenly across the six organizers[5].

This team grew around a core set of existing organizers of the East Bay Biosecurity meetup. All organizers on the team had some level of event organization or operations experience, including:

While we don’t have a clear counterfactual, we believe this experience was valuable to the execution of Catalyst, both in the form of practical logistics (e.g. having some notion of how to book a venue) and in terms of judgment and intuitions honed from prior similar situations.

Getting help and mentorship

Our advisors were super important, even though we only had a few meetings with them (maybe 5 hours in total).

We also had a really helpful 30-minute call with a CEA staff member early on about logistics and programming, which covered lots of useful ground (e.g. how to structure emails to speakers to increase the chance they’ll be interested).

Managing meetings and information

We have some additional advice specifically around meeting and information management:


Other Resources


  1. Most of the text was written by Tessa Alexanian [EA · GW] and Cody Wild [EA · GW], with substantial input and review from Finan Adamson [EA · GW], Jeffrey Ladish [EA · GW], Megan Crawford [EA · GW], and Brian Wang [EA · GW]. Thanks also to Aaron Gertler [EA · GW] for providing feedback on a draft. ↩︎

  2. To find this score, we asked “How likely are you to recommend (a similar event to) Catalyst to a friend, on a scale of 1-10?”, then took the ((# of 9 to 10 ratings) - (# 0 to 6 ratings)) / (total responses) = (27 - 1) / 35 = 74. ↩︎

  3. For anyone curious, here is the exact budget spreadsheet we used. It probably won’t be comprehensible to non-organizers, but should demonstrate the level of detail. ↩︎

  4. It was, ironically, focused on non-pharmaceutical interventions for pandemics... 4 days before the first reported case of community spread of COVID-19 in the USA. ↩︎

  5. Tessa estimated (based on detailed personal time tracking) that she spent 10-15 hours per month while actively working on the event (May-July 2019, November 2019 - January 2020), 65 hours in the three weeks immediately before the event, and a little over 12 hours on the day of the event itself. (She more or less took a break from August - October 2019 due to a family emergency, and estimates about 3 hours / month of work during that time.) Two other organizers estimated their time as a rough multiple of this, as “between 0.75 and 0.9 the amount of time Tessa spent” and “maybe 0.4 x Tessa's estimate”. ↩︎

11 comments

Comments sorted by top scores.

comment by Ben_West · 2021-05-28T00:31:58.643Z · EA(p) · GW(p)

Congratulations on such a successful event! 

  1. Regarding your NPS of 74: I think that's quite good; this page describes an NPS of 60+ as "very, very special, your only daughter probably just got married". 
  2. "We sent a lot of emails in the few weeks leading up to the summit, with various actions for participants to take, and that seemed like a good way to build enthusiasm / engagement." – could you say more about what you asked the attendees to do?
  3. I recently found out about Gather and was also pretty impressed. Do you happen to have filled out versions of the worksheets you could share? I'd be particularly interested in the "designer's agenda" you came up with
Replies from: tessa, tessa
comment by Tessa (tessa) · 2021-05-28T01:23:29.732Z · EA(p) · GW(p)

Could you say more about what you asked the attendees to do?

Sure! I looked through our emails and found the following messages to attendees:

  • T-(1 month to 1 week): "Catalyst Biosummit: You're In!" was a generic reminder / acceptance email asking people to confirm their attendance and register e.g. dietary preferences
  • (lots of emails with the most engaged applicants asking them to give lightning talks and host design jam groups)
  • T-7 Days: "Get ready to participate in Catalyst": sent out full agenda and logistics. Prompted attendees to sign up for meetups and share anything that might facilitate full participation (we noted that the venue had a gender-neutral restroom and gave the example of "space for lactation" as an accessibility need we'd be happy to meet if we were made aware of it)
  • T-4 Days: "Start shaping your Catalyst experience" prompted attendees to sign up for a design jam group, join the Slack, schedule breakout rooms and make a list of their goals for the day.
  • T-2 Days: "Sign up for your Catalyst design jam group today" reminded people to sign up for a design group (noting that we'd by default be randomly assigning everyone except invited speakers to groups early the next morning, and noting that people could opt out of this by replying to the email)
  • T-1 Day: "See you at Catalyst tomorrow" more of a logistical email (e.g. here is when and where you should show up) but included links to design jam briefs, the full agenda, and notes on how to recognize organizers in case an uncomfortable incident needed to be reported
comment by Tessa (tessa) · 2021-05-28T01:24:28.500Z · EA(p) · GW(p)

Do you happen to have filled out versions of the worksheets you could share?

No, alas; we only found Gather a few weeks before the event, at which point the schedule was largely finalized. But it was very clear that it would have been super useful to us if we'd found it earlier on.

comment by SiebeRozendal · 2021-05-28T14:07:59.486Z · EA(p) · GW(p)

Thanks for the highly detailed post! Seems like it was a cool event.

Nitpicking: this is the second time I see an evaluation described as "postmortem" and it puts me on the wrong foot. To me "postmortem" suggests the project was overall a failure, while it clearly wasn't! "Evaluation" seems like a better word?

Replies from: tessa
comment by Tessa (tessa) · 2021-05-29T00:28:38.047Z · EA(p) · GW(p)

Thanks for the feedback; nitpicking appreciated, since we also weren't sure about the title. We considered alternatives like "Learnings from a mid-sized event: Catalyst Biosecurity Summit Writeup" or "How to run a 100-person biosecurity event" but picked the current title for being short and containing the name of the summit.

I think we chose the word "postmortem" kind of following the naming trend of a few of the EAGx "postmortems" linked at the end of the post. I notice one of the current tags on the post is"Postmortems & Retrospectives". Would it have seemed more appropriate to you if the name was "Event Retrospective: Catalyst Biosecurity Summit"? Would "Retrospective: running a 100-person biosecurity summit" be better still? Further feedback welcome!

Replies from: tessa
comment by Tessa (tessa) · 2021-05-31T00:49:10.337Z · EA(p) · GW(p)

After more thought, we decided to rename the post from "Event Postmortem: Catalyst Biosecurity Summit" to "Retrospective on Catalyst, a 100-person biosecurity summit". Thanks again for the feedback!

Replies from: SiebeRozendal
comment by SiebeRozendal · 2021-06-01T11:49:52.963Z · EA(p) · GW(p)

That sounds like a better title to me :) Kudos on the adaptation.

comment by slg (Simon_Grimm) · 2022-01-19T19:47:35.219Z · EA(p) · GW(p)

Hey, I just wanted to leave a note of thanks for this excellent write-up!

I and some other EAs are planning an event with a similar format—your advice is super helpful to structure our planning and avoid obvious mistakes. 

In general, these kinds of project management retrospectives provide a lot of value (e.g., EAF's hiring retrospective [EA · GW]).

comment by Tristan Cook · 2021-05-26T21:25:19.264Z · EA(p) · GW(p)

Both links to Catalyst are broken (I think they're missing https://)

Replies from: aarongertler
comment by Aaron Gertler (aarongertler) · 2021-05-26T22:56:34.833Z · EA(p) · GW(p)

That was the issue -- just fixed the links.

Replies from: tessa
comment by Tessa (tessa) · 2021-05-27T01:16:38.114Z · EA(p) · GW(p)

Thanks for reporting and for fixing!