Editing Festival: Results and Prizes

post by Aaron Gertler (aarongertler) · 2021-05-29T23:16:26.703Z · EA · GW · 7 comments

Contents

  Overall
    good
    bad
  Traffic
  Tagging
  Wiki Edits
  Prizes
  Honoring all contributors
    who helped
None
7 comments

The EA Forum Editing Festival is over! [EA · GW] (And has been, for a bit.)

I’ll use this post to look at how well the festival went, and to award prizes to people who were especially good contributors. (I’ve selected these people in consultation with Pablo Stafforini, who’s been spending more time on the wiki than I have.)

Overall

I’d consider the festival a qualified success.

The good

The bad

I wish we’d seen more participation from more people, but I still wound up with a list of a dozen users whose work I’m happy to reward. (And I appreciate everyone who applied so much as one tag — see the “Honoring all contributors” section at the end.)

Traffic

Between 1 January and 24 February, our tag/wiki pages got ~4,400 total views, with an average view duration of 56 seconds.

Between 1 April and 24 May, those pages got ~16,700 views, with an average duration of 81 seconds. 

To give a specific example: “Longtermism” is our most-visited tag, and went from 118 views (1/1 - 2/24) to 253 views (4/1 - 5/24).

One factor behind this is a test we ran in April to reroute clicks on certain EA Concepts pages to their wiki equivalents (as we might eventually sunset EA Concepts in favor of a set of wiki links). But this contributes only a small fraction of the change; between these two periods, EA Concepts lost ~4,000 pageviews, while the wiki gained ~12,000.

Tagging

As we requested [EA(p) · GW(p)], people tagged every post with 25 karma or more, then almost all posts with 20-24 karma (there are only three untagged posts in that category as I write this).

We also got a lot of strong tag suggestions from users.

Finally, the overall tag rate went up quite a bit during the festival; in the first week of May, we averaged around 100 tags per day, roughly tripling our number from the first week of April. Since the end of the festival, the rate has dropped back to around where it was at the beginning, which isn’t what I’d hoped for — but we did tag a lot of archival material in the meantime. If only new posts are getting tags at this point, each of them should get 3-4 tags on average at current rates, which seems very reasonable.

Wiki Edits

I’ve been happy to see the overall speed of progress on the wiki, but disappointed to see almost all of that progress come from three users so far (Pablo, his assistant Leonardo, and Michael Aird). 

I think the initial collection of wiki content is shaping up well, but I still want us to have a broad set of editors so that new resources, papers, etc., routinely find their way into the appropriate articles. I’ll also note that Pablo, in the course of researching other community wikis, has found that “lack of contributors” appears to be the main reason these wikis fail to thrive. 

Over the next few months, Pablo and I will be looking for ways to incentivize wiki edits from the wider community. A few things that come to mind:

We also have a feature in the pipeline that will allow users to vote on edits, which should be helpful.

Prizes

As promised [EA · GW], we’ll be awarding at least $5000 in prizes to at least a dozen users (in the form of donations to charities listed on EA Funds).

Michael Aird [EA · GW]: $2000

Michael has contributed more words to the wiki than any author aside from Pablo, who has a full-time grant for wiki work. He also applied more than 500 tags over the course of the festival. And his meta-post on tag suggestions [EA · GW] has also been a useful organizational tool. Finally, he’s had many conversations with Pablo and I about the details of the wiki system, and has done much to influence our policies. I don’t know where he finds the time, but I’m grateful to have him as a super-user.

Maria Shakhova [EA · GW]: $750

Maria applied more than 800 tags over the course of the festival, including for a number of older and lower-karma posts that are now less likely to be forgotten. (As I did the research for this post, Maria’s tags gave me some good reading material!)

It’s especially impressive to me that this came from someone who’s written only a single comment during her time on the Forum! I’m glad that the Festival gave Maria this chance to contribute, and that I had the chance to see how much she’s been doing.

Tessa Alexanian [EA · GW]: $750

When she isn’t winning Forum prizes, Tessa has been a prolific participant, with more than 350 tags applied over the course of the festival. She also created many new tags:

Cullen O’Keefe [EA · GW]: $200

Cullen added more than 75 tags and created two new tags — donation matching [? · GW] and EA Giving Tuesday [? · GW].

evelynciara [EA · GW]: $200

evelynciara didn’t just participate actively in the festival (150+ tags, more than almost anyone) — she's also done lots of tagging in the last few weeks. Those recent tags don’t “count” towards this award, but I still love to see that the event helped to make Forum infrastructure into a habit for certain users.

Marisa Jurczyk [EA · GW]: $200

Marisa did a bit of tagging, and also created or heavily updated three useful tags:

Vaidehi Agarwalla [EA · GW]: $200

Vaidehi applied a few tags, but her main contribution came from new tags she created:

David Bernard [EA · GW]: $125

David added more than 75 tags during the festival.

Edo Arad [EA · GW]: $125

As a moderator, Edo winds up applying a fair number of tags (100+ during the festival) — but because he’s a volunteer and his job doesn’t actually require that he tag anything, I still want to reward his work as one of the Forum’s most active taggers.

Finn Hambly [EA · GW]: $125

Finn applied 70+ tags, with a focus on AI-related topics.

Harrison Durland [EA · GW]: $125

Harrison created two tags:

Both have grown on me, especially “competitive debating” — it’s nice to see another activity that seems like a natural pipeline for EA-oriented people.

nil [EA · GW]: $125

nil created two tags:

I thought the conversation around the David Pearce tag was valuable, and I’m glad nil’s work prompted it to happen.

Stefan Schubert [EA · GW]: $125

Stefan added useful content to bibliographies on three [? · GW] different [? · GW] tags [? · GW] — and these were three different articles, not just the same article being forced into multiple tags. This kind of contribution is one I’d love to see more of — if you have a favorite book or paper on an EA topic, make sure it’s on the wiki page for that topic!

Honoring all contributors

As promised, we’ll use this post to remember all those who tagged any post, or added any new wiki content. Thank you all for your contributions!

If you took part and don’t see your name here, please let me know.

Those who helped

7 comments

Comments sorted by top scores.

comment by MichaelA · 2021-05-30T09:57:04.384Z · EA(p) · GW(p)

Thanks for this post :)

One thing I've been wondering about is how much impact will result (indirectly) from the wiki/tagging system, and, relatedly, from various types of individual contributions to it

I'm wondering this both to work out whether I'm spending more time contributing than I really should and to work out whether I should do more/less encouraging of other people to contribute. (This also seems relevant for "targeted solicitation toward Forum users we expect will be strong editors for specific articles" - having a clearer impact assessment would help both in deciding when it's worth people spending their time this way and in convincing them of that.)

Currently I - as you can probably tell - feel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I don't actually have a clear sense of how good a use of people's time this is.

---

I think it'd help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/we could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also [EA · GW].)

---

Here are some other proxy/impact assessment ideas, though they're kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion we've previously had non-publicly, where Pablo mentioned some other ideas I won't mention here.)

  • Look at total or average page views and work out how good or bad those results are
    • Obviously you've already got the page view data, but at least personally I don't know whether that's good or bad or what I wanted to see (except "more is better")
  • Run something like Rethink Priorities' impact survey [EA · GW] or my personal adaptation of that (see also [? · GW]), where either relatively "key" people or just a whole bunch of Forum users are asked how much they've engaged with the wiki/tags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
    • (I work at Rethink Priorities, but am - as usual - writing this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
    • What I'd have in mind as a quick and dirty option is basically just replacing "anything I've written" or "anything by Rethink Priorities on [topic]" with "any wiki entries"
    • It's less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
      • Similar questions, but for "have you found content to read via looking at tag pages or filtering by tags", "do you think you would've found that content anyway", "do you think that that changed your beliefs (relative to what you might've done with that time otherwise)", etc.
      • Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/behaviours, etc.
  • See if the wiki/tags show up in a future EA Survey as a way people learned about or became more engaged with EA?

A caveat I should add: The wiki could be seen more as a product or website than a research output/project. And I'm more used to thinking about how to assess the impact of the latter (though I'm not an expert on that anyway).So maybe my suggestions are a bit off, and maybe it'd be useful to think more about things like:

  • How 80k assess the impact of their services
  • How the Forum and LessWrong assess their impact or performance or whatever
  • How software companies assess how their products are doing (aside from just revenue)
comment by MichaelA · 2021-05-30T10:00:14.484Z · EA(p) · GW(p)

Over the next few months, Pablo and I will be looking for ways to incentivize wiki edits from the wider community. A few things that come to mind:

  • A feature allowing people to see their wiki edits on their own profiles
  • An addition to the Forum Prize. awarding one or more wiki edits each month
  • More targeted solicitation toward Forum users we expect will be strong editors for specific articles

These ideas seem good to me. I think another low-hanging fruit might be some way of displaying quantitative info about how much someone has contributed to this EA wiki, as is done on Wikipedia; see here [EA(p) · GW(p)] for elaboration.

But it might be best to do all of these things after getting more sense of how much impact contributing to the wiki/tags has (see my other comment), since that'll inform how much time it'd actually be good to get people spending contributing to the wiki/tags. 

Replies from: aarongertler
comment by Aaron Gertler (aarongertler) · 2021-05-31T01:19:29.805Z · EA(p) · GW(p)

We definitely want to add wiki contribution data to user profiles (it's a natural accompaniment to the lists of users' posts and comments), though the timeline for that project isn't yet established.

comment by MichaelA · 2021-05-30T09:38:50.861Z · EA(p) · GW(p)

Oh wow, that's a fairly chunky prize! (For readers who don't follow the link to the festival announcement post, it seems worth noting that these prizes are "In the form of donations to an EA Funds-eligible charity of your choice", rather than e.g. deposits to my own bank account)

This seems like a good prompt to revisit and share my rough, interim notes-to-self on where I might donate in 2021 and why.  (I plan to do more thinking and share more details around November, like I did last year [EA · GW].)

  • The EA Long-Term Future Fund
    • See also The Long-Term Future Fund has room for more funding, right now [EA · GW]
    • As of April 2021, I have a positive impression of their fund managers
    • In April 2021, I skimmed through the LTFF's grants to date to see if there were any that I knew had later turned out quite well, and how this compares to what would've happened if I'd donated directly to things I'd heard of and thought were good
      • I felt unsure how to update on the results of that skimming and thinking
      • There were many cases where I do have a positive impression of the person or project that was funded, but where it wasn’t obvious that (a) LTFF was especially early in identifying this as good, or that (b) the LTFF money allowed the project to go much better than it would’ve gone otherwise
        • E.g., HIPE, All-Party Parliamentary Group for Future Generations, stuff Ozzie Gooen has done
        • In these cases, LTFF should get “points” for funding this, but it’s unclear that they’re choosing better funding opportunities than I’d choose if making decisions myself directly
          • (But note that I'm just saying this in relation to me, and things like work tests suggest that I may be a fairly good fit for grantmaking. Analogously, I should defer to others regarding good career pathways in the EU, but some other EAs need not defer much on that.)
      • But there was also at least 1-3 cases where the project seems to have gone really well or to still be early stage but have really good expected impact per dollar spent, and where I wasn’t aware of the project/plan at all when the grant was made
        • A grant to Nick Hollman to support the Legal Priorities Project
        • Pablo’s wiki work might count
          • (Note that I'm currently unsure how to think about the expected impact of this; I'll write another comment about that)
        • Nuno Sempere’s work might count
      • There was a perhaps surprising number of cases where, as of April 2021, I still hadn’t heard of the person or project at all, or where I knew the person but hadn’t seen any outputs of the project yet
        • But it’s hard to say what I should’ve expected here, since I of course shouldn’t expect to know about every nook and cranny of longtermism, and it’s good if LTFF finds good-but-obscure funding opportunities, and all of the grants were within the last couple years
    • I’d really like it if LTFF at least sometimes set or shared proxies of the impact they hope to see, made forecasts, and shared retrospectives
      • I feel the same way about the other EA Funds too
      • See discussion of this here [EA(p) · GW(p)] and here [EA · GW]
      • I will probably write more about this soon
    • I read LTFF's May 2021 payout report [EA · GW] yesterday, and I don't think it was a positive or negative update for me (i.e., still seems basically quite good, but not obviously better than my donating directly, at least after accounting for the "diversity of viewpoints" benefit of me making my own, direct donation choices)
  • Effective Thesis [? · GW]
    • My current, tentative impression is that they're filling a really important niche, and that it's really odd they haven't received funding at a somewhat higher level and with more stability
    • I may think and write more about this in future
    • But I haven't yet tried to look closely at or make quantitative cost-effectiveness models or Fermi estimates or the link
  • GCRI [? · GW]
    • This is one of the places I gave to in 2020, for the reasons explained here [EA(p) · GW(p)], and I still have roughly the views expressed there
  • ALLFED [? · GW]
    • This is one of the places I gave to in 2020, for the reasons explained here [EA(p) · GW(p)], and I still have roughly the views expressed there
  • Funding work on "Creating a central, editable database to help people choose and do research projects", of the kind I'll discuss in an upcoming post
    • This funding could be paying for contractors, for someone to "own" this project, for hosting fees, etc.
  • All-Party Parliamentary Group for Future Generations [? · GW]
    • [I had some notes to self explaining why this stands out to me as one option, but will leave those of this comment. If I end up thinking more about whether to donate to them, I'll say more on this later, probably around November.]

It seems that Effective Thesis and APPGFG aren't available via the EA Funds platform, even though both have been funded by EA Funds before? Can they be added to the platform relatively easily? (That isn't a super big deal just for this particularly $2000 donation - I'm ok just picking between LTFF, GCRI, and ALLFED - but it seems like that'd be good for other donors too.)

Once I find out whether Effective Thesis and APPGFG can be options for me for this $2k, I'll decide on the resulting shortlist.

Replies from: aarongertler, Jonas Vollmer
comment by Aaron Gertler (aarongertler) · 2021-05-30T09:43:03.328Z · EA(p) · GW(p)

(For people who don't follow the link to the festival announcement post, it seems worth noting that these prizes are "In the form of donations to an EA Funds-eligible charity of your choice.")

I've added this detail to the post for clarity. Thanks!

I'll check in with EA Funds about whether they have any plans to add those options.

Replies from: MichaelA
comment by MichaelA · 2021-05-30T09:54:08.540Z · EA(p) · GW(p)

Thanks! (On both parts, but especially the latter.)

comment by Jonas Vollmer · 2021-05-31T10:25:45.738Z · EA(p) · GW(p)

I haven't read this post and comment in full, but here are our criteria for adding charities to EA Funds.