SHIC Will Suspend Outreach Operations

post by cafelow · 2019-03-07T16:07:03.465Z · score: 153 (63 votes) · EA · GW · 8 comments

Contents

    A Q1 update and 2018 in review
      By Baxter Bullock and Catherine Low
  Summary
  2018 in review
  Our results
  Our reasoning for suspending workshop operations
  Looking Forward
  Endnotes
8 comments

A Q1 update and 2018 in review

By Baxter Bullock and Catherine Low

Since launching in 2016, Students for High-Impact Charity (SHIC), a project of Rethink Charity, has focused on educational outreach for high school students (primarily ages 16-18) through our interactive content. In January 2018, we began implementing instructor-led workshops, mostly in the Greater Vancouver area. Below, we summarize our experiences of 2018 and explain why we are choosing to suspend our outreach operations.

Summary

2018 in review

In late 2017 we made a strategic shift towards a high-fidelity model of student engagement through instructor-led workshops. We tested this model throughout 2018, with our instructors visiting schools in Greater Vancouver, Canada[1] [EA · GW].

Most students (56%) participated in a single-session workshop lasting approximately 80 minutes. These workshops consisted of a giving game[2] [EA · GW], followed by an overview of the core ideas of effective altruism[3] [EA · GW], including coverage of key cause areas. The remaining 44% of participants participated in multi-session (typically three), in-depth workshops which usually included a giving game, interactive explorations of the topics mentioned above, a cause prioritization activity, and a discussion of effective career paths.

Our goal for the second half of 2018 was to identify high-potential students from our school visits, and engage them further with supplementary advanced workshops at a central location in Vancouver. To gauge interest initially, we began with an opt-in approach for all interested students who provided an email address in order to obtain more information. We ran a workshop in November which primarily consisted of an in-depth activity on cause prioritization, and a workshop in December focused on effectively creating online fundraisers for the holidays.

Our results

The metrics we identified to gauge our success were:

  1. Teachers and school uptake
  2. Student survey results indicating shifts of opinion and/or behavior
  3. The number of students who continue to engage with the material beyond our initial visit, and the extent to which they remain involved.

We exceeded our expectation with metric 1 - Demand from teachers was higher than initially expected, with 25% of the schools we contacted booking at least one workshop. We also saw some success through representation at local teacher conferences. All told, in 2018 we reached a total of 2,580 students at 40 institutions, mostly in high school Social Studies and Math classes throughout Greater Vancouver[4] [EA · GW]. Qualitative feedback from teachers and students, which includes our reads on the reactions of workshop participants, suggested that students were engaged. Teachers generally thought the content and delivery was excellent and well-suited to their curriculum area. The number of teachers who asked us to return for additional classes and/or the for the autumn semester after our initial visit serves as further evidence of continued interest in the program.

We met our expectations for metric 2 - Qualitative and quantitative data gathered from pre- and post-workshop surveys [EA · GW] showed small but significant improvements in knowledge, attitudes, and values for the majority of our questions, suggesting that students generally shift their perspectives as a result of our program. However, there is likely a social desirability effect to consider[5] [EA · GW]. Additionally, it was unclear to what extent these students are likely to act upon these newfound perspectives.

We attempted to gauge whether these shifts were temporary, but only 37 out of the 388 students who were emailed a 3-month survey chose to start the survey, so we were unable to draw conclusions from such a small sample.

We did not meet our expectations for the third, and most important metric - Despite the success of our first two metrics, we struggled to further engage students with opt-in advanced workshops. We feel the turnout was poor enough to suggest that this was not an effective method for SHIC to engage students beyond the initial workshop. 1247 students were eligible to be informed about our advanced workshops, and we sent out 392 invitations[6] [EA · GW]. 17 students expressed interest, and five ended up attending our first advanced workshop in November, and two attended the December workshop (both returning from the first). There may have been logistical reasons for why the December workshop was less attended by students. The decrease in attendance could be attributed to a change in the workshop time (moved to begin on a Sunday morning) or its proximity to the holidays, or it could have been the particularly discouraging weather (cold, raining heavily).

Based on these results, we’re left with one or more of the following three conclusions:

  1. The students were engaged in the SHIC program in class, but our methods for engaging students beyond the classroom were ineffective.
  2. Students had the will, but not the bandwidth to engage further with the SHIC program.
  3. Students were not as engaged by the SHIC program as our post-program survey data and experience suggested, and therefore uninclined to participate in further programming.

Our best guess is that all three of these conclusions are true to some extent. High school-aged students in North America tend to engage in a plethora of extracurricular activities, often leaving them with little bandwidth. We also believe that using email as our primary method of contacting students was not the optimal form of correspondence, as email tends not to be a widely used form of communication with this age group[7] [EA · GW].

Our reasoning for suspending workshop operations

Our decision to suspend SHIC’s operations was an unusually difficult one because, despite our failure to deliver on our key metric, we still believe educational outreach in some form could be effective. For example, we could have shifted our audience to more targeted groups beyond high schools, or used other methods of engagement within high schools. It’s also possible that SHIC workshops had more of an effect than we were able to measure, such as increasing the chances that students will donate or fundraise for effective charities in the future, or be more receptive next time they encounter EA ideas.

However, as a result of not reaching our goals, we came to believe SHIC is now less marketable to new and existing donors who were also interested in long-term engagement of students. Had this not been the case, we would have more strongly considered exploring the effectiveness of our program with new audiences, such as university students and adults.

Finally, the section below outlines our belief that organizational bandwidth and resources would be more valuable if reallocated to other promising projects, such as RC Forward and the Local Effective Altruism Network (LEAN).

We are nonetheless proud of what SHIC has accomplished, and believe there is a possibility that this program made an impact on many students’ lives, despite our difficulties in tracking that impact.

Looking Forward

A SHIC website containing our materials and a contact form will remain active - While active outreach will be suspended, we think it’s important to maintain accessibility to the user-friendly content SHIC has created. We would also like to put volunteer time and minimal staff time towards consulting those with inquiries about how best to implement EA education.

A detailed evaluation of SHIC and high school outreach is forthcoming - By May 1, we plan on releasing an in-depth look at not only our experiences with high school workshops, but all programs SHIC has conducted, as well as aggregate data from other similar projects related to EA educational outreach. It is our hope that this post will provide a foundation upon which experimentation with educational outreach can continue within EA.

Repurposed SHIC resources could be very valuable - A dissolution of one of our projects allows us to reallocate some of the more promising elements of that project towards other initiatives. In SHIC, we have the resources and staff to effectively make complex philosophical and mathematical concepts accessible to the public. We believe this could prove very valuable for other projects within Rethink Charity, such as educating high net worth donors in Canada about effective giving opportunities as part of RC Forward, and providing high quality materials and guidance for group leaders who wish to run workshops on EA concepts as part of the Local EA Network (LEAN).

A potential return to educational outreach in the future - We still believe that educational outreach could be very impactful, and are pleased that others within the EA community are continuing with their outreach programs. Pending clearance of several hurdles from a strategic standpoint, this could be worth returning to in the future. Those hurdles include reconsidering the optimal audience, methods of data collection, and methods of long-term engagement.

_______________________________________________

Endnotes

1. Prior to becoming SHIC Manager, Catherine Low tested SHIC materials in the classroom as a high school teacher in 2016 and 2017. Her experiences provided evidence that an instructor-based model may be more effective than student-led models.

2. SHIC’s giving game involves students analyzing 3 to 4 charities comparatively, then deciding which charity will receive a donation.

3. In most cases, the term “effective altruism” was not used as part of the SHIC program.

4. In Vancouver high schools specifically, we reached 2,228 students in 31 institutions.

5. We included a social desirability test in our survey to attempt to measure this bias, however the results have not been factored into our survey analysis [EA · GW].

6. For students who participated in the full SHIC program in the Winter and Spring terms of early 2018, providing a contact email was mandatory. In Summer and Fall 2018, 19% of the students voluntarily gave us a contact email.

7. We considered other (social media focused) methods of communication that ultimately felt inappropriate.

8 comments

Comments sorted by top scores.

comment by Larks · 2019-03-08T03:01:56.667Z · score: 49 (24 votes) · EA · GW

This was very interesting. Retrospectives on projects that didn't work can be extremely helpful to others, but I imagine can also been tough to write, so thanks very much!

comment by Ozzie Gooen (oagr) · 2019-03-08T22:29:20.816Z · score: 37 (16 votes) · EA · GW

Kudos for pursuing this and not getting too attached to it as to not be willing to give it up when the evidence showed such made sense.

Here's a question: do you have thoughts on how this could have failed faster? If you were to go about this again, what would you have done differently in order to spend even fewer resources on it?

comment by cafelow · 2019-03-11T09:46:12.545Z · score: 36 (10 votes) · EA · GW

Good question Ozzie. In the start of 2018 we mostly focussed on getting into schools and on the surveys (metrics 1 and 2 above), because they were our first hurdles and we were very uncertain on how these would go until we started the project.

However that meant we didn't optimise our workshops for engaging students long term (metric 3) for several months after starting the project. That meant we weren't confident in making decisions based on the first indications that we were not meeting metric 3, and ran the project for several more months as a result. If we had planned our long term engagement strategy at the start of 2018 and set success criteria earlier we could have learnt what we needed to in less time.

comment by aarongertler · 2019-03-08T00:53:17.050Z · score: 26 (13 votes) · EA · GW

Strong upvote for publishing this summary. Reading it, I feel like I have a good sense of the program's timeline, logistics, and results. I also really appreciated the splitting up of metrics by "success level" and "importance" -- a lot of progress updates don't include the second of those, making them a lot less useful.

Sounds like any future project meant to teach EA values to high-school students will have to deal with the measurement problem (e.g. "high school students are busy and will often flake on non-high-school things"). Maybe some kind of small reward attached to surveys? At $10/person, that seems affordable for 380 students given the scale of the program, though it might make social desirability bias even stronger.

comment by baxterb · 2019-03-08T15:09:23.363Z · score: 16 (7 votes) · EA · GW

Thanks Aaron. Measurement problems were a big issue. We experimented with incentives a bit, particularly offering to randomly select from students who completed the post-program survey, and those selected would receive money to give to a charity of their choice, but that didn't seem to make a difference, or at least we weren't in a place to offer a significant enough incentive to make a noticeable difference.

The other measurement problem that we ran into was knowing that, given the age of workshop participants, in most cases we wouldn't be able to measure actionable impact for another ~5 years.

comment by Jon_Behar · 2019-03-08T18:20:21.001Z · score: 23 (12 votes) · EA · GW

Others have said this but it bears repeating: thank you for writing this up! This sort of detailed post-mortem is a resource the whole community can learn from. Kudos!

comment by MichaelPlant · 2019-03-07T19:07:40.099Z · score: 19 (11 votes) · EA · GW

Thanks for writing this up. Great to see people testing things and then adjusting their plans in light of the results.

This is probably a relatively minor question but this wasn't something you mentioned so I thought I'd ask: was transportation a problem in people getting to the advanced workshops? I can imagine that, if the a student needed to be driven to the workshop, that would make it much harder to attend.

comment by David Vatousios · 2019-03-07T20:28:25.413Z · score: 27 (11 votes) · EA · GW

Hi Michael, I'm the Vancouver-based Educator for SHIC and one of two people who ran the advanced workshops (the other being Baxter).

Your question is a good one! Whilst we did our best to accommodate students from all parts of Greater Vancouver - picking a central location that was very close to a train station - the trek for many students would have been quite a long one (Greater Vancouver is very spread out!). Since we currently don't have any feedback regarding this, it's hard to say to what extent students were deterred by the prospect of a long commute, and whether or not having access to a car factored into their decision to come or not (I should probably note that some of the students who did attend one or both of the advanced workshops had a long way to come, and made the journey by public transit). My own guess is that it was probably a minor factor overall, but one still worth addressing if we, or anyone else, ever decides to take another crack at something like this.