Lessons from a full-time community builder. Part 1 of 4. Impact assessment

post by weeatquince · 2017-10-04T18:14:12.357Z · score: 14 (14 votes) · EA · GW · Legacy · 5 comments

Contents

  Summary
      Raising awareness of EA -> engagement in our community -> positive behaviour changes
  Contents
  Background and initial plans
    THE VISION
    MODEL / THEORY OF CHANGE
    AIMS AND PLANS
    PREDICTIONS
  Breakdown of activities and costsACTIVITIES
    COSTS
  Impact – key data
    HEADLINE FIGURESThe total impact of Effective Altruism London this year is estimated to be:
    IMPACT OF PROVIDING FUNDING
    HOW THIS COMPARES TO OTHER EA ORGS
    Our measured impact on behaviour changes suggests we are in the same order of magnitude as other meta EA organisations but somewhat less effective.
    The returns to funding other EA orgs is in the ballpark of $1000 per pledge or equivalent (am uncertain estimate based on talking to people who work in EA organisations). We are getting a pledge level change in behaviour for about $2000 of funding (depends on how measured, see Cost benefit analysis section in Appendix A below).
  Impact – further analysis
    GROWTHGrowth rates are unclear but generally positive. Noise is added as the kinds of events being run has changed throughout the year as we have experimented with different things.
  Impact – Other benefits and stories of success
  Update on June – Sept
  Conclusions
  Annex A: Impact – elaboration & data collection
  Annex B: Subgroups
      You can read this here.
None
5 comments

NOTE: For the past year Sam Hilton has been funded by the EA community in London to grow, run and support the community. This is part 1 of a 4 part write up, broken down as follows:

Part 1. Impact assessment

Part 2.  General lessons on how to build EA communities.

Part 3. Specific lessons on running a large local community.

Part 4. Future plans and a request for funding

[Links will be provided to future articles when they are written]

 

 

Summary

Background: Sam was funded to work full time on the EA London community for a year.

What was done:

What was the estimated impact:

Conclusions:

 


Contents

- Background and initial plans

- Breakdown of activities and costs

- Impact – key data

- Impact – Further analysis

- Impact – Other benefits and stories of success

- Update on June 2017 – Sept 2017

- Conclusions

- Annex A: Impact – elaboration & data collection

- Annex B: London’s other EA communities


This document can be read as a Google Doc here
.
This is part 1 of 4. Other parts cover community building, lessons learned and future plans. 
[Links pending]   

 

 

Background and initial plans

This impact assessment covers the activities done by Effective Altruism London from 3 June 2016 until 7 June 2017.

HISTORY

In April 2013, the first Effective Altruism London social event was held. The community grew slowly but steadily, drawing from outreach efforts in London and growth in the wider EA community. In June 2016 the community began funding Sam Hilton to act as a full time movement builder for a year.

THE VISION

Our aspirational vision was written collectively by the community and has helped guide actions. It is:

Everyone in London working effectively towards a better world

MODEL / THEORY OF CHANGE

We had a 3 stage model of how we create impact. Awareness of the community leads to people engaging which leads to people changing their behaviour to be more altruistic and effective:

 

Awareness (of EAL)

Engagement

Changing behaviour

Happens through:

Meetup.com, word of mouth, wider EA

 

Event attendance, (some online activity)

 

Event attendance, (some personal support)

Measured by:

Numbers on Meetup, Facebook, email, etc

 

Event attendance, Facebook conversation

 

Surveys on self-reported change, projects started


This model has remained largely unchanged for the year, although we also tried to recognise (but do not have good measures for) alternative paths to impact. Including:

• Community benefits, such as networks and EA friendships leading to projects and actions taken.

• Benefits of keeping existing EAs within the EA community

AIMS AND PLANS

The aims set for this year were:

  1. Work out the best ways of building an engaged EA community in London

  2. Secondary aim: Support and inspire those in the EA London community to have a greater positive impact on the world.

  3. Tertiary aim: Grow the EA movement in London

Plans were made about how these aims would be achieved.  For example, we planned to:

Details of the initial plans can be found at: February 2016 _ Request for funding

Plans, adjusted following our mid-year progress review, are at: Jan 2017 _ Six month review

We had a budget of just over £30,000 to carry out this work

PREDICTIONS

Prior to the year beginning we predicted the following counterfactual impacts

 

 

Breakdown of activities and costs
ACTIVITIES

The following chart shows the breakdown of key activities carried out in the past year.

Time (apx)

Activity

Details

22%

Events

socials, talks, workshops

22%

Awareness raising

newsletter, advertising, website, auto-emails, speaking

16%

Subgroups

LSE, Imperial, UCL, policy x 2, finance, animals, others

12%

Strategy

Quarterly strategy reviews

7%

Impact measurement

 

7%

Operations

Charity, accounts, insurance, volunteer management

7%

Collaboration

Networking, writing up, talking to other EA orgs,

6%

Experiments / campaigns

Humanist experiment, buddies, pledge campaign

1%

Community

One-on-one support, Facebook groups, interns


This is roughly as expected
from the initial plans, although perhaps with:

• less time going into structured experiments

• more time than expected spent on impact measurement

• more time than expected spent on growing EA London’s subgroups


COSTS

The following chart shows the breakdown of financial costs for the past year.

 

Money Spent

Inc gifts in kind and lost wages

TOTAL

£19,510

£34,800

Staff time

£17,860

£32,860

Marketing

£855

£925

Events

£630

£850

Overheads

£165

£165


We spent £19,510 from 1 June 2016 until 1 June 2017. Adding in estimates for gifts in kind (books from CEA, printing) this brings the cost to £19,760. Sam took a low wage for the year. If accounting for the loss in salary of £15,000 the cost becomes approximately £35,000 (assumes 100% of additional salary earned would have been donated).

Note: We have not accounted for time spent by volunteers. This is difficult to do as it is unclear:




Impact – key data

HEADLINE FIGURES

The total impact of Effective Altruism London this year is estimated to be:

The above figures are calculated estimates rather than numbers known with certainty. We explain how they were calculated below in Annex A: Impact – elaboration & data collection.


IMPACT OF PROVIDING FUNDING

We estimate that if not funded we would have had 33% of the impact on behaviour and attendance, 25% of the impact on contactable individuals, and the finance subgroup would have been started. This is based on the amount of and size of events run in by volunteers in previous years.

This means that the impact on behaviour of funding EA London, rather than letting it be run by volunteers, is about 2/3 of the impact set out above, ie. 12 large behaviour changes of which 8 are GWWC pledges.

Additionally not being volunteer run means a reduced risk of community collapse, and the additional benefits created, as set out in the section: Impact – Other benefits and stories of success.




HOW THIS COMPARES TO OTHER EA ORGS


Our measured impact on behaviour changes suggests we are in the same order of magnitude as other meta EA organisations but somewhat less effective.

The returns to funding other EA orgs is in the ballpark of $1000 per pledge or equivalent (am uncertain estimate based on talking to people who work in EA organisations). We are getting a pledge level change in behaviour for about $2000 of funding (depends on how measured, see Cost benefit analysis section in Appendix A below).

 

 

Impact – further analysis

GROWTH

Growth rates are unclear but generally positive. Noise is added as the kinds of events being run has changed throughout the year as we have experimented with different things.

  

On the other hand, we did not find additional evidence that we are changing behaviour or getting more new people engaged at an increasing rate. This maybe a sign that we are not successfully engaging more new attendees despite larger events, but might also be because the data set is noisy, too small and has missing data.

 

HOW OUR ACTIONS CREATED IMPACT

Roughly 80% of project time went into marketing and running events, which led to the behaviour changes we saw (18 large and 174 significant behaviour changes). It is difficult to know exactly which actions lead to which changes as people often do not change their behaviour straight away.

We can see from the event reports that social events led to the most new attendees (people for whom this was their very first event) returning (attending another event with 50 days):

 

Number of attendees at events by type

Total number of each type of event

Average returning new attendees

Total (not unique) event attendees

New attendees

Returning new attendees

Career networking

199

106

8

10

0.8

Social

594

263

48

23

2.1

Talk

417

221

15

16

0.9

Workshop

240

156

19

14

1.4

Other

130

23

11

11

1.0

TOTAL

1580

769

101

74

1.4

NOTES: • Not adjusted to account for imperfect attendance data, total numbers returning is likely to be higher.
• Dates covered in the above table: 01 Aug 16 to 31 Jul 17.   • Returning means attended another event within 50 days.

More detail on exactly which events, campaigns and actions were more impactful than others is covered in part 3 of the write-up on Specific lessons on running a large local community [link pending].

Roughly 20% of the time went into supporting the sub groups, supporting individuals with their projects, and so forth. We believe this had other benefits, such as the existence of student EA groups at London universities. (Events at student groups and any resultant behaviour changes are not captured by our metrics, eg. we know of and have not counted 3 pledges at student groups).


WHY COUNTERFACTUAL IMPACT OF FUNDING WAS LOWER THAN PREDICTIONS

How our predictions match our results:

 

Metric

Prediction

Result*

Comparison to prediction

Within range predicted

Awareness

Contactable individuals

1,500-2,500

2250

Better

Yes

Engagement

Unique attendees

400-800

600

Same

Yes, mid range

Attending multiple events

50-250

160

Same

Yes, mid range

Behaviour

Change

Made a significant change

Not predicted

174

-

-

Made a huge change

30-85

12

Lower

No

GWWC pledge

20-55

8

Lower

No

* Predictions made comparing to if volunteer run. Results are are the marginal impact, so 33% lower than headline figures..

We overpredicted the expected behaviour change. We have a good understanding of why this happened. The main problem was that we based predictions on self-reported behaviour change, but this did not match up to actual behaviour changes. For example, people said on surveys that attending EA London events has already had a drastic (8 out of 8) effect on their behaviour but upon questioning failed to give clear evidence of such changes.

Also, our behaviour change metric does not capture future behaviour changes. For example, changing a career plan (or donation plan, etc) would not be captured by our metric, unless we believe the person could give evidence that they had already changed their career (or donations etc). Anecdotal evidence does however suggest that people who are attending regular events and report an intention to change their behaviour may well actually do so.

Predictions of attendance and repeat attendance very closely matched results.

 

 

 

 

Impact – Other benefits and stories of success

As well as introducing new people to EA we have had significant other successes, as illustrated by the anecdotes below:

Founding and supporting sub-communities. Some examples:

Networking effects. Attendees have commented that they have benefited from the networking effects of the EA London community. Some examples:

Retention effects. Attendees have commented that the EA London community helps make sure they continue to engage with EA ideas. However, this effect is hard to measure and we have no good evidence or stories of this. We have seen a few people attending events who had been involved in EA at university, not done much since and EA London gets them re-engaging with EA ideas.

Paved the way for future behaviour change. We have seen people who got involved in EA in part through EA London, prior to this year, play a bigger role in the EA community including:

These are not benefits created this year but are evidence that impressive actions and significant life changes may take longer than a year to manifest. Already we have seen:

Paved the way for future growth of the London community. We should grow more quickly in the future as we have gained:

Downsides. It is likely some of the people who have put time and effort to advising and supporting EA London may have had significant benefit elsewhere.

 



Update on June – Sept

In the last 3 months we have continued to put time into EA London and to grow (captured by the orange bars in the charts in the section on growth). We have also been training new staff. Sam took a step back (and will shortly be moving into a Government role) and Holly Morgan and David Nash took over the EA London work. In these 4 months we have seen:

 


Conclusions

Overall the conclusions from this year are rather weak. As set out above we believe:

 

SHOULD EAS CONTINUE TO FUND THE LONDON COMMUNITY

 As a result of our limited success this year we intend to change our plans. We hope that with slightly different plans and more focus we can have an even greater impact in future.

Against funding:

For funding:

Overall, we think it depends on what the other opportunities for funding meta EA work such as movement building work are. We believe such opportunities are limited and that CEA are not considerably funding-constrained. We have, collectively as a community, taken the decision to develop future plans and seek funding for another year.

See part 4 for our future plans [link pending].

 

SHOULD EAS FUND FULL TIME LOCAL COMMUNITY BUILDERS IN MORE CITIES

We believe the larger EA organisations providing small amounts of funding and support to local communities is useful. That said, this year has not made a strong case either for or against funding full time local community organisers. Overall, we have no strong recommendation either way and it may depend on what other opportunities are available.

We do however think there is value in funding outreach to innovative community groups with specific targets, such as HIPE which is reaching UK civil servants, REG which reached top poker players or Founders Pledge which is reaching founders. This is defended by our ideas of how communities grow (see part 2 of write up on: General lessons on how to build EA communities) and analysis of our subgroups (see section below: Annex B: Subgroups). We will look to do more of this in London next year.

If people in the EA movement do fund local city organisers they should be prepared for the fact that compared to other movement building actions that involve mass marketing, growing a local community will produce less rigorous quantitative data but will provide a greater insight into the individuals impacted. 

 

 

RECOMMENDATIONS FOR CEA / LEAN 

A few things we think that CEA and/or LEAN should be considering in their work to build the EA movement:




Annex A: Impact – elaboration & data collection

You can read this here.



 

Annex B: Subgroups

You can read this here.

 

5 comments

Comments sorted by top scores.

comment by Robert_Wiblin · 2017-10-05T01:40:57.884Z · score: 11 (11 votes) · EA(p) · GW(p)

Nice post. A few comments:

i) I'd focus more on the opportunity cost of the people involved than the financial cost.

ii) Being half as cost effective as CEA would still be pretty good! So long as CEA wasn't significantly funding constrained, it would make sense to fund others groups that could do that well. BTW I'd heard of marginal cost of pledge acquisitions as low as a few hundred dollars 3 years ago, but maybe that has gone up. I've also heard concerns the pledge quality could be declining as GWWC has scaled and made it easier to join.

iii) $2,000 for a pledge makes it better than funding AMF just from a fundraising point of view. You only need someone to follow through on giving 10% for an extra 1 year or so to break even. People who are part of a local community are probably above-average quality members as well (likely to donate to better places and for longer). But if you think about the staff's opportunity cost (i.e. their ability to do useful direct work if they weren't doing EA London) maybe it's not so good.

iv) I expect the long-term impact on the people involved to be more valuable again than the 12 pledge-equivalents, but people's mileage varies on that quite a lot.

comment by weeatquince · 2017-10-29T23:10:19.480Z · score: 2 (2 votes) · EA(p) · GW(p)

Thanks for the feedback Rob

i) The opportunity cost of time has been low.

• For me, there were minimal opportunities to do something higher impact at this stage in my career. For example, I may have stayed in government and I doubt this would have had much impact (also this year out has not significantly damaged my civil service career, I was able to return on a promotion). It is not clear that I had the credibility on any other EA project that I could have found funders willing to cover my costs for the year. I could have worked part time in the civil service and tried to found a different type of other organisation but I think it is unlikely to have gone as well.

• David Nash has invested time but it is helping him move career-wise in a direction he wants to be going in.

• I expect the interns taken on would not have spent time as effectively otherwise.

• Time invested by others was minimal.

ii)-iv) Agree

comment by Michael_PJ · 2017-10-14T21:44:53.067Z · score: 1 (1 votes) · EA(p) · GW(p)

This is great stuff! Really appreciate the effort you put into measuring things.

comment by zdgroff · 2017-10-05T19:58:22.201Z · score: 1 (1 votes) · EA(p) · GW(p)

Thanks for sharing this! Out of curiosity, was there any particular evidence that drove the basic theory of outreach (awareness -> engagement -> behavior change)? This seems like actually a hotly contested empirical area so I'm curious. Thanks!

comment by weeatquince · 2017-10-29T23:14:37.339Z · score: 0 (0 votes) · EA(p) · GW(p)

It roughly came from the idea of treating movement builidng as a marketing funnel. It is similar to the marketing funnel you'd expect of any other organisation except "buy our junk" is replaced with "behaviour change".

I did not have specific evidence on community building that this was a particularly good theory of change, although nothing I read when looking for data on this suggested it would not be a good theory of change.

What is it that it contested about this?