I'll Fund You to Give Away 'Doing Good Better' - Surprisingly Effective?

post by Jeremy (captainjc) · 2019-03-19T18:21:04.086Z · score: 34 (17 votes) · EA · GW · 14 comments

Contents

  Summary
  Surprising Response
  Other EA Book Giveaways
  Surveying Recipients
  Why You Should Consider Doing It
  How to Do it
    Writing/Posting
    Responding
    Keeping Track
    Ordering Books
    Sending Surveys
    Financial Aid
  Evaluating and Publishing Results
  Feedback Requested
None
14 comments

This is an expanded version of a post I made to the EA Facebook group in December 2018. Thanks to Aaron Gertler for reading a draft of this post, finding many typos (I’m sure I’ve added more since), and making many good suggestions. Besides commenting here, you can contact me about this project at eabookgiveaway [at] gmail [dot] com.

Summary

I got a surprising number of responses when I offered to send out free copies of Doing Good Better on social media. It seemed like it could be a scalable, cost-effective form of outreach, so I have created a survey to assess impact. In the following post, I try to convince others (you!) to make the offer - in order to gain a larger sample size. I detail what I did. I give some reasons why you should consider it, and suggestions how to do it. I estimate how long it will take to publish meaningful results, and I request feedback.

Surprising Response

Somewhat impulsively, back in December, I posted to Facebook and Instagram, offering to send a copy of Doing Good Better to anyone who wanted to read it. It seemed like a nice thing to do around the holidays, but I wasn't expecting much response - especially since I rarely post on Facebook, and often hear crickets when I do.

But the algorithm gods must have smiled. 21 people responded to my offer, 3 more to a repost, and another 3 to a friend who was inspired to make the same offer. In total 28 people, that I know, of received the book. It broke down like this:

Additionally, I sent articles to 2 people who expressed interest in reading about EA, but who didn’t want to commit to a book.

Other EA Book Giveaways

When I posted about it to the EA Facebook group, people were interested to read survey results, and generally positive about the idea. I also learned that EA New Zealand has an ongoing offer of a free copy of the book, and that EA London has also handed out books (DGB, as well as Superintelligence, Poor Economics, The Life You Can Save and 80,000 Hours) at their events. They both shared their survey questions.

Catherine from EA NZ told me that they have now given away over 900 copies. She said that they are in the midst of analyzing recent data, but she has concerns it may not be as effective as they initially thought (or perhaps not as effective now as at the beginning).

According to EA London's summary, "...it seemed that handing out books at EA London events increased the amount of engagement", though, to my knowledge, they didn't test against any kind of control group (I.E. attendees who weren't offered a book). It's also worth noting that their book recipients were attending an EA event and, therefore, had at least some familiarity with EA.

I was interested in how they decided when to send the survey, as I was concerned that it often takes me a while to get around to reading a new book. Catherine confirmed that a lot of people responded that they hadn't read it, but that the survey was a reminder to do so. She said she was planning to change it to instruct people NOT to fill out the survey if they haven't read it, then to send a reminder to those people later. This seemed like a really great idea.

Surveying Recipients

My goal was to measure, as much as possible, the value from recipients reading the book vs counterfactual. I found it helpful to break it down in a very obvious way:

(A) the value of recipients' actions after reading the book
(minus)
(B) the value of what they would have done anyway
(minus)
(C) the cost

For A, I didn’t want to ask what people intend to do, as I am very uncertain how that correlates with what they actually do. Rather, I asked what changes recipients have already made. The obvious drawback to this approach is that much of the value of their actions may not yet be realized, because it doesn’t capture anything done after taking the survey.

To remedy that, the last question asks if people would be willing to take a follow-up survey after a year (the follow-up will then ask for another one, etc. etc.). Some may not be willing to do this, but the survey is pretty short and unimposing, and it seems like willingness to do it would correlate highly with interest in EA - which, in turn, would correlate highly with actions taken/value.

This certainly won't catch everything everyone does as a result of reading the book, but it seems better than tracking people's intentions, then trying to figure out how to value them. I recognize that this is a bit of an unusual way to do it, so I would appreciate any feedback.

For B, I was initially thinking of asking people if the book was their introduction to EA, or to somehow gauge their involvement in EA before and after, but it's unclear how that information would fit cleanly into the equation - especially if they already had some knowledge of EA. I decided to just ask people what they have done, that they wouldn't have otherwise. I do have some concerns that people might misunderstand, or not be able to estimate this accurately. If this instruction does not seem clear, or you feel you have a better idea, please comment.

C is obviously not a part of the survey, but just carefully tracking the time and money spent, which I have done.

The Survey can be found here. (I will wait a couple of weeks to send it out, in order to incorporate any changes from community feedback.)

Why You Should Consider Doing It

How to Do it

If you’d like to do it, please comment on this post, or email me at eabookgiveaway [at] gmail [dot] com. I plan to can keep track of who's doing it, so I can follow up to see how it went. The following guide is based entirely on my intuitions and my own experience doing it. It will be valuable to integrate feedback from other people who have done it as well.

Writing/Posting

Here's what I wrote (posted along with an image of the book cover):

"Content warning: Sincerity. For the holidays this year, I am giving out copies of this book - just message me your address and I'll send you one! Effective Altruism is a movement I've become increasingly excited about over the past couple of years. Besides the inherent #warmfuzzies of helping others, it's also an excellent antidote to the #culturewars and reading about #trump #brexit #opioidepidemic or whatever depressing news item. This book is a great primer. Well written, more entertaining than you'd think, not too technical and pretty much inspiring. Seriously, message me and I will mail you a copy for Christmakwaanzakah."

Here is some of my reasoning and a few other ideas and suggestions:

Responding

Keeping Track

Here's a blank version of the Google sheet I used to track things. To use it, choose File > Make a Copy, to save it to your own account.

It contains fields for:

Ordering Books

Sending Surveys

And that should do it! Of course, feedback on all this is very much welcome - especially if you’ve found something else that worked or didn't. I consider this post a work in progress and will updated it as needed.

Financial Aid

While I think it’s definitely best if it is seen as a personal gift/offer, I understand that this may be difficult for some people to afford. For the moment, I am happy to reimburse for the cost of the books, or order the copies for those who are unable to (this will come out of my personal donation budget, likely otherwise headed for GiveWell unrestricted). Email me at eabookgiveaway [at] gmail [dot] com to coordinate.

Evaluating and Publishing Results

Assuming the survey approach described above, there may not be much to report initially. I will probably wait until the first follow up surveys are completed (Fall 2020?) to write a full report - unless results are particularly remarkable before then.

I will publish the results to the EA forum, and, if it turns out to be cost-effective, I will build a website (updated regularly) for them as well, where I will also post most of the information contained herein. (As my day job is web developer, this would be cheap/easy for me to create and maintain.)

Feedback Requested

I would love feedback/comments on any of the above - but specifically:

14 comments

Comments sorted by top scores.

comment by aarongertler · 2019-03-19T23:39:46.918Z · score: 7 (6 votes) · EA · GW

Fantastic post, Jeremy! I'm a bit biased, since I had the chance to see earlier drafts, but I really like the generous spirit of this initiative, and it seems like a low-risk, high-potential way to grow the community. It's very kind of you to offer funding to others who want to try their own giveaways.

In fact, I might just try this myself come Giving Season; I've set a reminder in my calendar to think about it on November 15th. Thanks for the idea.

Regarding the survey: Consider changing the wording on question #9:

The bulk of the impact from introducing people to Effective Altruism probably happens over the long term. If you think you might make future changes, or you generally agree with the principles of the book, we'd love to be able to check in with you in a year, to see how things are going.

I'd remove the section in bold. If people are really interested in EA, they'll hopefully give you contact information either way; if they're on the fence, they might feel a bit objectified being referred to as sources of impact, or guilty about donating once and planning not to do so in the future (I can imagine giving $100 to GiveWell, then seeing the survey and losing my warm glow because I haven't had "the bulk of my impact").

This is a highly speculative suggestion, though, and I don't think it makes a big difference either way.

comment by Jeremy (captainjc) · 2019-03-20T14:03:24.981Z · score: 3 (2 votes) · EA · GW

Thanks Aaron. Glad to hear you might try it.

Good point about that survey question. I was really trying to get across the importance of tracking future actions, but I agree that it could come across off-putting. What do you think of this?

Thank you for taking the time to tell us about what you've done. We'd love to hear about what you may still do as well. If you think you might make future donations or changes in line with the principles of Effective Altruism, can we send you a single, short follow-up survey a year from now? If so, please enter your email address below - it will not be shared with anyone, or used for any other purpose.
comment by aarongertler · 2019-03-21T02:02:08.241Z · score: 2 (1 votes) · EA · GW

That version does sound better. One more suggested version:

Thank you for taking the time to share what you've done. Since we also asked about your future plans, could we follow up with one more short survey a year from now, to see what happened?

If that's alright with you, please enter your email address below - it will not be shared with anyone, or used for any other purpose.

I'm hoping this feels a bit less high-pressure than "what you may still do", but you could also remove "to see what happened" to help with that.

comment by Jeremy (captainjc) · 2019-03-21T15:15:17.571Z · score: 1 (1 votes) · EA · GW

I see what you're saying, but I (intentionally) didn't ask about future plans so I'm not sure if that part works. While I definitely don't want it to be high pressure, I do want to somehow emphasize the importance of tracking future actions. I wonder if this strikes a balance.

Thank you for taking the time to share what you have done. In order to help us understand the longer-term impact of the book, would you mind if we followed up with another short survey a year from now? If that's alright, please enter your email address below - it will not be shared with anyone, or used for any other purpose.

I really appreciate you taking the time to back-and-forth with me about this!

comment by aarongertler · 2019-03-21T23:48:35.421Z · score: 3 (2 votes) · EA · GW

I like that wording, and don't have any changes to suggest.

comment by Davidmanheim · 2019-03-24T12:09:58.631Z · score: 5 (3 votes) · EA · GW

Having done a little bit of graduate work on survey design, I'll put on my survey-design hat and offer a bunch of suggestions. If you'd like to discuss further, please feel free to reach out directly on - twitter / @gmail.com / calendly - my username is my full name.

Before getting to the question, a few points on sample design and research ethics:

1) You might want to provide distinct links to the survey for each subset of answerers, i.e. those who you gave the book and are asking directly, or those who each other person has done so for. You may also want to link to a different survey to ask the secondary people who borrowed the book. This allows you to later consider if there are differences in effectiveness.

2) You should clarify that the data will be analyzed and shared, but will first have any personally identifying information removed. (i.e. don't share the raw results with email addresses with ANYONE.)

3) Best-best practice would be to pre-register your analysis plan.

On to the questions! I have a bunch of language and structure nitpicks I'd suggest changing:

re: 1 - "I have't read/finished it yet, but I plan to," I think that's ambiguous - people might have stopped reading, but still be impacted. I'd change to the following options:

  • I read it
  • I haven't begun reading it yet - Please skip to the final 2 questions
  • I just began reading it and plan to finish - Please skip to the final 2 questions
  • I have read a substantial part of the book, but didn't finish it.
  • I do not plan on reading it.

(Also, don't ask for an email address here, instead say "If you have made any changes to your life based on this, please continue. " People are more likely to feel willing to provide extra information as they move later in the survey

Relatedly, re: "2. Since reading the book, have you donated to any effective charities, that you wouldn't have otherwise? If so, please list each charity, with US dollar amount, on its own line."

I'm concerned that people will find the wording or the request pushy / invasive and refuse to answer or stop the survey completely. I'd suggest splitting this into a gentler 3-part version: (Again, people who start complying are more likely to continue filling out more-invasive questions.)

2. Since reading the book, have you donated to any effective charities, that you wouldn't have otherwise? Yes/No

2a. If so, we would be interested in evaluating how effective giving away the book was, and would like to know the total amount you have donated. (# Box)

2b. If you are willing to provide further details please list the charities and US dollar amounts, on separate lines. (Textbox)

For the remaining questions, I'd also recommend refactoring and making the flow better;

3. Have you considered made any other life changes since reading the book? Yes / No

(If not, please skip to question 7.)

3a. If so, did you change or consider changing your: Dietary choices (See Q4) / Career Plans (See Q5) / Other (See Q6)? (Checkboxes)

4. If you changed your Dietary choices, have you become Vegan (4a) / Vegetarian (4a) / Reduced meat consumption (4b) / Other (4c) / Considered this but have not (yet) changed anything (4c)?

4a. How many months ago?

4b. How much less eggs or meat?

4c. What else did you change / consider changing?

5. Have you made or considered making career changes?(Current Q6 options)

5a. Details (text)

6. Other changes (text)

7. Put question 10 here, before asking about the follow up.

Also: Give them the option to provide email addresses instead of asking their friend themselves, if they think it would be less intrusive. Note that the email will be short, and the email addresses will not be kept beyond sending a single email asking them to take the survey. (Note: I ALSO think people are a bit more likely to send the request themselves if given this option as an alternative.)

8. Then, Q9 / If you would be willing to take another follow-up survey, please enter... (Make language include either the "haven't read / finished yet" AND asking for a follow-up - you will know which is which based on the above.)

comment by Jeremy (captainjc) · 2019-03-26T15:41:16.464Z · score: 1 (1 votes) · EA · GW

Thanks David! Great suggestions. I am just back from a trip, but I will dig in more deeply this week and make some revisions and reply to you in more depth - either here or via email. Thanks again!

comment by Jon_Behar · 2019-05-27T21:13:28.149Z · score: 4 (2 votes) · EA · GW

You may want to consider another cheaper and easier way to test a book giveaway, that will have some built in measurement/follow-up processes. As I mentioned on another thread [EA · GW], there’s going to be

an updated 10th anniversary edition of The Life You Can Save coming out in Q4. There will be updated numbers and examples, two new forewords, and increased emphasis on specific calls to action meant for a broad audience (e.g. initially asking people to make a recurring donation vs. a substantial pledge).
The price is also right, as we’ll be able to distribute free copies of the e-book (which will have links so people can take action more easily) and audiobook. The audiobook will have chapters read by celebrity narrators; this isn’t the time or place to list people involved in the project, but they’ll be a great credibility boost.
A lot of EA origin stories start with the first version of TLYCS. We’re about to have a chance to distribute a new and improved version to a much wider audience, and we hope the EA community will help spread it far and wide.
(I work for TLYCS the nonprofit, which is producing and promoting TLYCS the book.)

If you're interested in helping distribute free copies, please let me know!

comment by DavidNash · 2019-03-22T10:43:20.524Z · score: 3 (3 votes) · EA · GW

It's good to see people trying small low cost experiments like this although I suspect there are diminishing returns to how many people will be interested if you try a similar post again but that's also why it's good that you've given a framework for others to copy.

At EA London I haven't given as many books out in the last 12 months as the previous 12 as I've usually only given them to people after a 1-1 meeting or significant interest from someone at an event where it seems more likely that they'll actually read it and want to take away some advice rather than offering to give it to people who are more interested in the idea of a "free book".

comment by Jeremy (captainjc) · 2019-03-26T15:47:41.530Z · score: 1 (1 votes) · EA · GW

Yes, I had considered doing another post, but I guess I was thinking probably not. It would certainly be diminishing returns, yeah. I, and I'm sure many others, wouldn't want to start seeming spammy about it. Maybe after a couple of years.

comment by Davidmanheim · 2019-03-24T12:11:13.118Z · score: 1 (1 votes) · EA · GW

It's an interesting point. Do you know who you've given it to, and would you consider sending them a separate copy of the survey once it's been released, to gauge effectiveness / relative effectiveness?

comment by Jeremy (captainjc) · 2019-03-26T15:44:07.791Z · score: 1 (1 votes) · EA · GW

@Davidmanheim I am not sure if they are still doing it, but I know they were doing a survey. It is linked to in the Google Doc of their summary that I linked to above.

comment by weeatquince · 2019-03-21T18:36:40.772Z · score: 3 (2 votes) · EA · GW

Awesome post.

Suggestion: I have found in person feedback to useful alongside surveys. Suggest making a bit of effort to talk to people in person, especially if it is friends you see anyway, and including this data into a final impact estimate.

comment by Jeremy (captainjc) · 2019-03-21T20:02:11.509Z · score: 1 (1 votes) · EA · GW

Good idea. I will definitely do that.