How x-risk projects are different from startups 2019-04-05T07:35:39.513Z · score: 51 (30 votes)
Request for comments: EA Projects evaluation platform 2019-03-20T22:36:32.565Z · score: 32 (30 votes)
OpenPhil: Reflections on 2018 Generalist Research Analyst Recruiting 2019-03-08T02:41:44.804Z · score: 44 (16 votes)
What to do with people? 2019-03-06T11:04:21.556Z · score: 69 (47 votes)
Critique of “Existential Threats” chapter in Enlightenment Now 2018-11-21T10:09:54.552Z · score: 10 (10 votes)
Suggestions for developing national-level effective altruism organizations 2018-10-17T23:35:37.241Z · score: 18 (14 votes)
Why develop national-level effective altruism organizations? 2018-10-17T23:29:44.203Z · score: 26 (18 votes)
Effective Thesis project review 2018-05-31T18:45:22.248Z · score: 25 (24 votes)
Review of CZEA "Intense EA Weekend" retreat 2018-04-05T20:10:04.290Z · score: 25 (25 votes)
Optimal level of hierarchy for effective altruism 2018-03-27T22:32:15.211Z · score: 8 (11 votes)
Introducing Czech Association for Effective Altruism - history 2018-03-12T22:01:49.556Z · score: 21 (21 votes)


Comment by jan_kulveit on Get-Out-Of-Hell-Free Necklace · 2019-07-15T07:45:25.653Z · score: 5 (4 votes) · EA · GW

I've suggested something similar for happiness ( ). If you don't want to introduce the weird asymmetry where negative counts and positive not, what you get out of that could be somewhat surprising - it possibly recovers more "common folk" altruism where helping people who are already quite well off could be good, and if you allow more speculative views on the space on mind-states, you are at risk of recovering something closely resembling some sort of "buddhist utilitarian calculus".

Comment by jan_kulveit on EA Forum 2.0 Initial Announcement · 2019-07-12T22:26:45.635Z · score: 11 (5 votes) · EA · GW

As humans, we are quite sensitive to signs of social approval and disapproval, and we have some 'elephant in the brain' motivation to seek social approval. This can sometimes mess up with epistemics.

The karma represents something like sentiment of people voting on a particular comment, weighted in a particular way. For me, this often did not seemed to be a signal adding any new information - when following the forum closely, usually I would have been able to predict what will get downvoted or upvoted.

What seemed problematic to me was 1. a number of times when I felt hesitation to write something because part of my S1 predicted it will get downvoted. Also I did not wanted to be primed by karma when reading other's comments.

On a community level, overall I think the quality of the karma signal is roughly comparable to facebook likes. If people are making important decisions, evaluating projects, assigning prices... based on it, it seems plausible it's actively harmful.

Comment by jan_kulveit on EA Forum 2.0 Initial Announcement · 2019-07-12T11:10:47.130Z · score: 10 (4 votes) · EA · GW

It's not an instance of complain, but take it as a datapoint: I've switched off the karma display on all comments and my experience improved. The karma system tends to mess up with my S1 processing.

It seems plausible karma is causing harm in some hard to perceive ways. (One specific way is by people updating on karma pattern mistaking them for some voice of the community / ea movement / ... )

Comment by jan_kulveit on Is there an analysis that estimates possible timelines for arrival of easy-to-create pathogens? · 2019-06-15T10:22:35.332Z · score: 19 (7 votes) · EA · GW

I would expect if organizations working in the area have reviews of expected technologies and how they enable individuals to manufacture pathogens, which is likely the background necessary for constructing timelines, they would not publish too specific documents.

Comment by jan_kulveit on What new EA project or org would you like to see created in the next 3 years? · 2019-06-14T23:06:13.481Z · score: 6 (4 votes) · EA · GW

If people think this is generally good idea I would guess CZEA can make it running in few weeks. Most of the work likely comes from curating the content, not from setting up the service

Comment by jan_kulveit on Long-Term Future Fund: April 2019 grant recommendations · 2019-04-09T00:24:20.137Z · score: 5 (4 votes) · EA · GW

To clarify - agree with the benefits of splitting the discussion threads for readability, but I was unenthusiastic about the motivation be voting.

Comment by jan_kulveit on Long-Term Future Fund: April 2019 grant recommendations · 2019-04-09T00:02:29.425Z · score: 7 (7 votes) · EA · GW

I don't think karma/voting system should be given that much attention or should be used as a highly visible feedback on project funding.

Comment by jan_kulveit on Long-Term Future Fund: April 2019 grant recommendations · 2019-04-08T23:16:52.995Z · score: 40 (13 votes) · EA · GW

I don't think anyone should be trying to persuade IMO participants to join the EA community, and I also don't think giving them "much more directly EA content" is a good idea.

I would prefer Math Olympiad winners to think about long-term, think better, and think independently, than to "join the EA community". HPMoR seems ok because it is not a book trying to convince you to join a community, but mostly a book about ways how to think, and a good read.

(If they readers eventually become EAs after reasoning independently, it's likely good; if they for example come to the conclusion there are mayor flaws in EA and it's better to engage with the movement critically, it's also good.)

Comment by jan_kulveit on How x-risk projects are different from startups · 2019-04-06T17:05:37.220Z · score: 30 (10 votes) · EA · GW

I don't think risk of this type is given too much weight now. In my model, considerations like this got at some point in the past rounded of to some over-simplified meme like "do not start projects, they fail and it is dangerous". This is wrong and led to some counterfactual value getting lost.

This was to some extent reaction to the previous mood, which was more like "bring in new people; seed groups; start projects; grow everything". Which was also problematic.

In my view we are looking at something like pendulum swings, where we were somewhere at the extreme position of not many projects started recently, but the momentum is in direction of more projects, and the second derivative is high. So I expect many projects will actually get started. In such situation the important thing is to start good projects, and avoid anti-unicorns.

IMO the risk was maybe given too much weight before, but is given too little weight now, by many people. Just look at many of the recent discussions, where security mindset seem rare, and many want to move fast forward.

Comment by jan_kulveit on How x-risk projects are different from startups · 2019-04-06T16:54:18.392Z · score: 12 (6 votes) · EA · GW

Discussing specific examples seems very tricky - I can probably come up with a list of maybe 10 projects or actions which come with large downside/risks, but I would expect listing them would not be that useful and can cause controversy.

Few hypothetical examples

  • influencing mayor international regulatory organisation in a way leading to creating some sort of "AI safety certification" in a situation where we don’t have the basic research yet, creating false sense of security/fake sense of understanding
  • creating a highly distorted version of effective altruism in a mayor country e.g. by bad public outreach
  • coordinating effective altruism community in a way which leads to increased tension and possibly splits in the community
  • producing and releasing some infohazard research
  • influencing important players in AI or AI safety in a harmful leveraged way, e.g. by bad strategic advice

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-26T00:54:09.963Z · score: 2 (1 votes) · EA · GW

My impression is you have in mind something different than what was intended in the proposal.

What I imagined was 'priming' the argument-mappers with prompts like

  • Imagine this projects fails. How?
  • Imagine this project works, but has some unintended bad consequences. What they are?
  • What would be a strong reason not to associate this project with the EA movement?

(and the opposites). When writing their texts the two people would be communicating and looking at the arguments from both sides.

The hope is this would produce more complete argument map. One way to think about it, is each person is 'responsible' for the pro/con section, trying to make sure it captures as much important considerations as possible.

It seems quite natural for people to think about arguments in this way, with "sides" (sometimes even single authors expose complex arguments in the "dialogue" way).

There are possible benefits - related to why 'debate' style is used in justice

  • It levels the playing field in interesting ways (when compared to public debate on the forum). In the public debate, what "counts" is not just arguments, but also discussion and social skills, status of participants, moods and emotions of the audience, and similar factor. The proposed format would mean both the positives and negatives have "advocates" ideally of "similar debate strength" (anonymous volunteer). This is very different from a public forum discussion, where all kinds of "elephant in the brain" biases may influence participants and bias judgements.
  • It removes some of the social costs and pains associated with project discussions. Idea authors may get discouraged by negative feedback, downvotes/karma, or similar.

Also, just looking at how discussions on the forum look now, it seems in practice it is easy for people to look at things from positive or negative perspectives: certainly I have seen arguments structured like (several different ways how something fails + why is it too costly if it succeeded + speculation what harm it may cause anyway).

Overall: in my words, I'm not sure whether your view is 'in the space of argument-mapping, noting in the vicinity of debate, will work - at least when done by humans and applied to real problems'. Or 'there are options in this space which are bad' - where I agree something like bullet-pointed lists of positives and negatives where the people writing them would not communicate seems bad.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-22T10:02:09.061Z · score: 1 (2 votes) · EA · GW

My impression was based mostly on our conversations several months ago - quoting the notes from that time

lot of the discussion and debate derives from differing assumptions held by the participants regarding the potential for bad/risky projects: Benjamin/Brendon generally point out the lack of data/signal in this area and believe launching an open project platform could provide data to reduce uncertainty, whereas Jan is more conservative and prioritizes creating a rigorous curation and evaluation system for new projects.

I think it is fair to say you expected very low risk from creating an open platform where people would just post projects and seek volunteers and funding, while I expected with minimum curation this creates significant risk (even if the risk is coming from small fraction of projects). Sorry if I rounded off suggestions like "let's make an open platform without careful evaluation and see" and "based on the project ideas lists which existed several years ago the amount of harmful projects seems low" to "worrying about them is premature".

Reading your recent comment, it seems more careful, and pointing out large negative outcomes are more of a problem with x-risk/long-term oriented projects.

In our old discussions I also expressed some doubt about your or ability to evaluate x-risk and similar projects, where your recent post states that projects that impact x-risks by doing something like AI safety research has not yet applied to the EA Angel Group.

I guess part of the disagreement comes from the fact that I have focus on x-risk and the long-term future, and I'm more interested both in improving the project landscape in these areas, and more worried about negative outcomes.

If open platforms or similar evaluation process also accept mitigating x-risk and similar proposals, in my opinion, unfortunately the bar how good/expert driven evaluations you need is higher, and unfortunately signals like "this is a competent team" which VCs would mainly look at are not enough.

Because I would expect the long-term impact will come mainly from long-term, meta-, exploratory or very ambitious projects, I think you can be basically right about low obvious risk of all the projects historically posted on hackpad or proposed to, and still miss the largest term in the EV.

Milan asked this question and I answered it.

Thanks - both of that happened after I posted my comment, and also I still do not see the numbers which would help me estimate the ratio of projects which applied and which got funded. I take as mildly negative signal that someone had to ask, and this info was not included in the post, which solicits project proposals and volunteer work.

In my model it seems possible you have something like chicken-and-egg problem, not getting many great proposals, and the group of unnamed angels not funding many proposals coming via that pipeline.

If this is the case and the actual number of successfully funded projects is low, I think it is necessary to state this clearly before inviting people to work on proposals. My vague impression was we may disagree on this, which seems to indicate some quite deep disagreement about how funders should treat projects.

I'm not entirely sure what your reasons are for having this opinion, or what you even mean

The whole context was, Ryan suggested I should have sought some feedback from you. I actually did that, and your co-founder noted that he will try to write the feedback on this today or tomorrow, on 11th of Mar - which did not happen. I don't think this is large problem, as we had already discussed the topic extensively.

When writing it I was somewhat upset about the mode of conversation where critics do ask whether I tried to coordinate with someone, but just assume I did not. I apologize for the bad way it was written.

Overall my summary is we probably still disagree in many assumptions, we did invest some effort trying to overcome them, it seems difficult for us to reach some consensus, but this should not stop us trying to move forward.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-22T02:30:36.610Z · score: 4 (3 votes) · EA · GW

Summary impressions so far: object-level

  • It seems many would much prefer expediency in median project cases to robustness and safety in rare low frequency possibly large negative impact cases. I do not think this is the right approach, when the intention is also to evaluate long-term oriented, x-risk, meta-, cause-X, or highly ambitious projects.
  • I'm afraid there is some confusion about project failure modes. I'm more worried about projects which would be successful in having a team, working successfully in some sense, changing the world, but achieving large negative impact in the end.
  • I feel sad about the repeated claims the proposal is rigid, costly or large-scale. If something would not work in practice it could be easily changed. Spending something like 5h of time on a project idea which likely was result of much longer deliberation and which may lead to thousands hours of work seems reasonable. Paradoxically, just the discussion about whether the project is costly or not likely already had higher cost than what setting the whole proposed infrastructure for the project + phases 1a,1d,1c would cost.


  • I will .not have time to participate in the discussion in next few days. Thanks for the comments so far.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-22T00:30:37.082Z · score: 10 (3 votes) · EA · GW

Thanks Sundanshu! Sorry for not replying sooner, I was a bit overwhelmed by some of the negative feedback in the comments.

I don't think step 1b. has the same bottleneck as current grant evaluator face, because it is less dependent on good judgement.

With your proposal, I think part of it may work, I would be worried about other parts. With step 2b I would fear nobody would feel responsible for producing the content.

With 3a or any automatic steps like that, what does that lack is some sort of (reasonably) trusted expert judgement. In my view this is actually the most critical step in case of x-risk, long-term, meta-, and similarly difficult to evaluate proposals.


  • I'm sceptical the karma or similar automated system is good for tracking what is actually important here
  • I see some beauty in automation, but I don't see it applied here in the right places

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T22:55:58.368Z · score: 7 (2 votes) · EA · GW

FWIW, part of my motivation for the design, was

1. there may be projects, mostly in long-term, x-risk, meta- and outreach spaces, which are very negative, but not in an obvious way

2. there may be ideas, mostly in long-term and x-risk, which are infohazard

The problem with 1. is most of the EV can be caused by just one project, with large negative impact, where the downside is not obvious to notice.

It seems to me standard startup thinking does not apply here, because startups generally can not go way bellow zero.

I also do not trust arbitrary set of forum users to handle this well.

Overall I believe the very lightweight unstructured processes are trading some gain in speed and convenience in most cases for some decreased robustness in worst cases.

In general I would feel much better if the simple system you want to try would avoid projects in long-term, x-risk, meta-, outreach, localization, and "searching for cause X" areas.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T19:42:52.863Z · score: 12 (4 votes) · EA · GW

It is possible my reading of your post somewhat blended with some other parts of the discussion, which are in my opinion quite uncharitable reading of the proposal. Sorry for that.

Actually from the list, I talked about it and shared the draft with people working on EA grants, EA funds, and Brendon, and historically I had some interactions with BERI. What I learned is people have different priors over existence of bad projects, ratio of good projects, number of projects which should or should not get funded. Also opinions of some of the funders are at odds with opinions of some people I trust more than the funders.

I don't know, but it seems to me you are either a bit underestimating the amount of consultation which went into this, or overestimating how much agreement is there between the stakeholders. Also I'm trying to factor in the interests of the project founders, and overall I'm more concerned whether the impact in the world would be good, and what's good for the whole system.

Despite repeated claims the proposal is very heavy, complex, rigid, etc. I think the proposed project would be in fact quite cheap, lean, and flexible (and would work). I'm also quite flexible in modifying it in any direction which seems consensual.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T18:49:51.172Z · score: 3 (2 votes) · EA · GW
You are missing one major category here: projects which are simply bad because they do have approximately zero impact, but aren't particularly risky. I think this category is the largest of the the four.

I agree that's likely. Please take the first paragraphs more as motivation than precise description of the categories.

Which projects have a chance of working and which don't is often pretty clear to people who have experience evaluating projects quite quickly (which is why Oli suggested 15min for the initial investigation above).

I think we are comparing apples and oranges. As far as the output should be some publicly understandable reasoning behind the judgement, I don't think this is doable in 15m.

It sounds to me a bit like your model of ideas which get proposed is that most of them are pretty valuable. I don't think this is the case.

I don't have strong prior on that.

To do this well they need to have a good mental map of what kind of projects have worked or not worked in the past,...

From a project-management perspective, yes, but with slow and bad feedback loops in long-term, x-risk and meta oriented projects, I don't think it is easy to tell what works and what does not. (Even with projects working in the sense they run smoothly and are producing some visible output.)

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T17:47:42.577Z · score: 3 (2 votes) · EA · GW

I'm not sure if we agree or disagree, possibly we partially agree, partially disagree. In case of negative feedback, I think as a funder, you are in greater risk of people over-updating in the direction "I should stop trying".

I agree friends and social neighbourhood may be too positive (that's why the proposed initial reviews are anonymous, and one of the reviewers is supposed to be negative).

When funders give general opinions on what should or should not get started or how you value or not value things, again, I think you are at greater risk of having too much of an influence on the community. I do not believe the knowledge of the funders is strictly better than the knowledge of grant applicants.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T13:24:31.627Z · score: -2 (9 votes) · EA · GW

On a meta-level

I'm happy to update the proposal to reflect some of the sentiments. Openly, I find some of them quite strange - e.g. it seems, coalescing the steps into one paragraph and assuming all the results (reviews, discussion, "authoritative" summary of the discussion) will just happen may make it look more flexible. Ok, why not.

Also it seems you and Oli seem to be worried that I want to recruit people who are currently not doing some high-impact direct work ... instead of just asking a couple of people around me, which would often mean people already doing impactful volunteer work.

Meta-point is, I'm not sure if you or Oli realize how big part of solving

new EA projects evaluation pipeline

is in consensus-building. Actually I think the landscape of possible ways how to do evaluations looks like in such a way that it is very hard to get consensus on what the "strongest form" is. I'm quite happy to create a bunch of proposals, e.g.

  • with removing final expert evaluation
  • removing initial reviews
  • removing public forum discussions
  • writing an unrealistic assumption that the initial reviews will take 15m instead of hours,
  • suggesting that the volunteers will be my busy friends (whose voluntary work does not count?)
  • emphasising public feedback more, or less
  • giving stronger or weaker voice to existing funders.

I have stronger preference for the platform to happen than for one option in any single of these choices. But what is the next step? After thinking about the landscape for a some time I'm quite skeptical any particular combination of options would not have some large drawback.

On the object level:

Re: funder involvement

Cross-posting from another thread

Another possible point of discussion is whether the evaluation system would work better if it was tied to some source of funding. My general intuition is this would create more complex incentives, but generally I don't know and I'm looking for comments.

I think it much harder to give open feedback if it is closely tied with funding. Feedback from funders can easily have too much influence on people, and should be very careful and nuanced, as it comes from some position of power. I would expect adding financial incentives can easily be detrimental for the process. (For self-referential example, just look on this discussion: do you think the fact that Oli dislikes my proposal and suggest LTF can back something different with $20k will not create at least some unconscious incentives?)

We had some discussion with Brendon, and I think his opinion can be rounded to "there are almost no bad projects, so to worry about them is premature". I disagree with that. Also, given the Brendon's angel group is working, evaluating and funding projects since October, I would be curious what projects were funded, what was the total amount of funding allocated, how many applications they got.

Based on what I know I'm unconvinced that Brendon or BERI should have some outsized influence how evaluations should be done; part of the point of the platform would be to serve broader community.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T04:09:23.379Z · score: 9 (5 votes) · EA · GW

It is very easy to replace this stage with e.g. just two reviews.

Some of the arguments for the contradictory version

  • the point of this stage is not to produce EV estimate, but to map the space of costs, benefits, and considerations
  • it is easier to be biased in a defined way than unbiased
  • it removes part of the problem with social incentives

Some arguments against it are

  • such adversarial setups for truth-seeking are uncommon outside of judicial process
  • it may contribute to unnecessary polarization
  • the splitting may feel unnatural

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T02:07:13.942Z · score: 3 (3 votes) · EA · GW

I don't see why continuous coordination of a team of about 6 people on slack would be very rigid, or why people would have very narrow responsibilities.

For the panel, having some defined meeting and evaluating several projects at once seems time and energy conserving, especially when compared to the same set of people watching the forum often, being manipulated by karma, being in a way forced to reply to many bad comments, etc.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T01:56:58.382Z · score: 3 (2 votes) · EA · GW

On the contrary: on slack, it is relatively easy to see the upper bound of attention spent. On the forum, you should look not on just the time spent to write comments, but also on the time and attention of people not posting. I would be quite interested how much time for example CEA+FHI+GPI employees spend reading the forum, in aggregate (I guess you can technically count this.)

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T01:12:16.424Z · score: 4 (3 votes) · EA · GW

I don't understand why you assume the proposal is intended as something very rigid, where e.g. if we find the proposed project is hard to understand, nobody would ask for clarification, or why you assume the 2-5h is some dogma. The back-and-forth exchange could also add to 2-5h.

With assigning two evaluators to each project you are just assuming the evaluators would have no say in what to work on, which is nowhere in the proposal.

Sorry but can you for a moment imagine also some good interpretation of the proposed schema, instead of just weak-manning every other paragraph?

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T00:52:55.462Z · score: 7 (3 votes) · EA · GW

I would be curious about you model why the open discussion we currently have does not work well - like here, where user nonzerosum proposed a project, the post was heavily downvoted (at some point to negative karma) without substantial discussion of the problems. I don't think the fact that I read the post after three days and wrote some basic critical argument is a good evidence for an individual reviewer and a board is much less likely to notice problems with a proposal than a broad discussion with many people contributing would.

Also when you are making these two claims

Setting up an EA Forum thread with good moderation would take a lot less than 20 hours.


I am pretty excited about someone just trying to create and moderate a good EA Forum thread, and it seems pretty plausible to me that the LTF fund would be open to putting something in the $20k ballpark into incentives for that

at the same time I would guess it probably needs more explanation from you or other LTF managers.

Generally I'm in favour of solutions which are quite likely to work as opposed to solutions which look cheap but are IMO likely worse.

I also don't see how complex discussion on the forum with the high quality reviews you imagine would cost 5 hours. Unless, of course, the time and attention of the people who are posting and commenting on the forum does not count. If this is the case, I strongly disagree. The forum is actually quite costly in terms of time, attention, and also emotional impacts on people trying to participate.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T00:50:37.640Z · score: 2 (1 votes) · EA · GW

With the first part, I'm not sure what would you imagine as the alternative - having access to evaluators google drive so you can count how much time they spent writing? The time estimate is something like an estimate how much it can take for volunteer evaluators - if all you need is in the order of 5m you are either really fast or not explaining your decisions.

I expect much more time of experts will be wasted in forum discussions you propose.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-21T00:07:57.412Z · score: 1 (2 votes) · EA · GW

As I've already explained in the draft, I'm still very confused by what

An individual reviewer and a board is much less likely to notice problems with a proposal than a broad discussion with many people contributing would ...

should imply for the proposal. Do you suggest that steps 1b. 1d. 1e. are useless or harmful, and having just the forum discussion is superior?

The time of evaluators is definitely definitely definitely not free, and if you treat them as free then you end up exactly in the kind of situation that everyone is complaining about. Please respect those people's time.

Generally I think this is quite strange misrepresentation of how I do value people's time and attention. Also I'm not sure if you assume the time people spend arguing on fora is basically free or does not count, because it is unstructured.

From my perspective, having this be in the open makes it a lot easier for me and other funders in the space to evaluate whether the process is going well, whether it is useful, or whether it is actively clogging up the EA funding and evaluation space. Doing this in distinct stages, and with most of the process being opaque, makes it much harder to figure out the costs of this, and the broader impact it has on the EA community, moving the expected value of this into the net-negative.

Generally almost all of the process is open, so I don't see what should be changed. If the complain is the process has stages instead of unstructured discussion, and this makes it less understandable for you, I don't see why.

Comment by jan_kulveit on Request for comments: EA Projects evaluation platform · 2019-03-20T23:26:35.661Z · score: 10 (4 votes) · EA · GW

To make the discussions more useful, I'll try to briefly recapitulate parts of the discussions and conversations I had about this topic in private or via comments in the draft version. (I'm often coalescing several views into more general claim)

There seems to be some disagreement about how rigorous and structured the evaluations should be - you can imagine a scale where on one side you have just unstructured discussion on the forum, and on the opposite side you have "due diligence", multiple evaluators writing detailed reviews, panel of forecasters, and so on.

My argument is: unstructured discussion on the forum is something we already have, and often the feedback project ideas get is just a few bits from voting, plus a few quick comments. Also the prevailing sentiment of comments is sometimes at odds with expert views or models likely used by funders, which may cause some bad surprises. That is too "light". The process proposed here is closer to the "heavy" end of the scale. My reason is it seems easier to tune the "rigour" parameter down than up, and trying it on a small batch has higher learning value.

Another possible point of discussion is whether the evaluation system would work better if it was tied to some source of funding. My general intuition is this would create more complex incentives, but generally I don't know and I'm looking for comments.

Some people expressed uncertainty if there is a need for such system. Some because they believe that there aren't many good project ideas or projects (especially unfunded ones). Others expressed uncertainty if there is a need for such system, because they feel proposed projects are almost all good, there are almost no dangerous project ideas, and even small funders can choose easily. I don't have good data, but I would hope having largely public evaluations could at least help everyone to be better calibrated. Also, when comparing the "EA startup ecosystem" with the normal startup ecosystem, it seems we are often lacking what is provided by lead investors, incubators or mentors.

Comment by jan_kulveit on Announcement: Join the EA Careers Advising Network! · 2019-03-19T21:51:44.064Z · score: 6 (6 votes) · EA · GW

Hi Evan, given that effective altruism is somewhat complex, how do you make sure the career advise given will be good? From the brief text, there does not seem to be

  • any quality control of advisors
  • any quality control of the advice given
  • any coordination with other people doing something similar, like local groups

Overall I like the general idea, but I'm worried about the execution.

What do you imagine as a worst-case failure scenario? I can easily imagine various viral headlines like

  • I applied for EA career advice and the advisor recommended me to donate a kidney! Scary!
  • EA coach made unwelcome sexual advances
  • EA advisor tried to recruit me for his ( dubious investment scheme / crypto startup / project to save the world by using psychedelics a lot / ...)

Comment by jan_kulveit on Sharing my experience on the EA forum · 2019-03-19T13:07:23.786Z · score: 24 (9 votes) · EA · GW

Please try to not take the negative feedback personally. I hope it will not discourage you from contributing in the future.

My best guess what happened with your previous post was that a lot of people either disliked the proposal, or disliked the fact the you seem to set on creating something "hard to abandon" without seeking much input from the community before. Downvoting the post is cheap way to express such feeling.

I agree that if people collectively downvote, in particular strong downvote, without explaining why, the result of the collective action is bad. (On the other hand it is easy to see why: explanations are costly. I explained why I don't like the proposal, but that may mean I will bear more of the social costs of disagreement or conflict.)

Comment by jan_kulveit on Concept: EA Donor List. To enable EAs that are starting new projects to find seed donors, especially for people that aren’t well connected · 2019-03-19T02:20:37.916Z · score: 42 (16 votes) · EA · GW

I'm in favour to improving this coordination problem, but I think this particular solution is a bad idea and should not be started. The main problem is unilateralist's curse. Suppose there is a really bad project which 19 out of 20 altruistically minded funders (and no professional grantmaker) would support. Your design of the structure would make it much more likely that it will get funded.

In general effective altruism has a lot of value in brand and goodwill and epistemic standards/culture (like in billions $). It seems relatively easy to create large negative impact by destroying part of this, which can be "achieved" even by relatively small project with modest funding. Public donor list seems to be literally the worst option for structure if we want to avoid bad projects.

Comment by jan_kulveit on Getting People Excited About More EA Careers: A New Community Building Challenge · 2019-03-10T18:08:25.084Z · score: 13 (6 votes) · EA · GW

To clarify the concern, I'm generally not much more worried about how you use it internally, but about other people using the metric. It was probably not clear from my comment.

I understand it was probably never intended as something which other should use either for guiding their decisions or evaluating their efforts.

Comment by jan_kulveit on Getting People Excited About More EA Careers: A New Community Building Challenge · 2019-03-10T16:05:48.501Z · score: 13 (9 votes) · EA · GW

Ultimately the more you ground the metric in "what some sensible people thing is important and makes sense right now", the more nuance it has, and the more is it tracking reality. The text in my quote is verbatim copy from the page describing the metric from 2015, so I think it's highly relevant for understanding how IASPCs were understood. I agree that 80k career guides as a whole actually has much more nuance, and suggests approach like "figure out what will be needed in future and prepare for that".

The whole accounting still seems wrong: per definition, what's counted is ... caused them to change the career path they intend to pursue, ... ; this is still several steps away from impact: if someone changes their intentions to pursue jobs in EA orgs, it is counted as impact, even if the fraction of the people making such plans who will succeed is low.

For specificity, would you agree that someone who was 2 years away from graduation in 2016, deciding to change career plan to pursuing a job in CEA, would have been counted as impact 10, while someone switching from a plan going to industry to pursuing PhD in econ would have been counted as 1, and someone deciding to stay in, let's say, cognitive neuroscience, would have been counted as 0?

Comment by jan_kulveit on Suggestions for developing national-level effective altruism organizations · 2019-03-10T11:53:19.122Z · score: 3 (2 votes) · EA · GW

Here by hierarchy I mean strictly tree-like flow of information, where the centre collects the inputs, decides, and sends commands. By fully distributed I mean "everybody talking to everybody" (fully connected network) or a random network (random pairs of orgs talking to each other).

Comment by jan_kulveit on Getting People Excited About More EA Careers: A New Community Building Challenge · 2019-03-10T11:08:48.035Z · score: 32 (21 votes) · EA · GW

Part of this is caused by (over)use of a metric called impact-adjusted significant career plan changes. In a way, you get exactly what you optimise for. Quoting from 80k website

A typical plan change scored 10 is someone who, in large part due to us, switched to working at a highly effective organisation like GiveWell, became a major donor (>$100k/year) to effective organisations, or become a major advocate of effective causes.
A typical plan change scored 1 is someone who has taken the Giving What We Can pledge or decided to earn to give in a medium income career. We also award 1s to people who want to work on the most pressing problems and who switch to build better career capital in order to do this, for instance doing quantitative grad studies or pursuing consulting; people who have become much more involved in the effective altruism community in a way that has changed their career, and people who switch into policy or research in pressing problem areas.
A typical plan change scored 0.1 is someone shifting to gain better career capital but where they’re less obviously focused on the most pressing problems, or where they’ve switched into an option that is less obviously higher impact than what they were planning before.

Scoring the options you recommend

  • Skills building in non-EA organisations such as start-ups ... scores either 0.1 or 1, so 10x or 100x less valuable, in comparison to changing plan to work in GiveWell
  • Earning to give ... scores 10x less valuable
  • Speculative options ... 10x -100x less valuable

It's worth to emphasise the metric which is optimised is changing plans. How the difference between where someone actually switched, vs. switched just the plan, is handled, is likely inconsistent across places and orgs.

Taken literally, the best thing for a large number of people under this metric is to switch plan to working for OpenPhil, and consider other options as failure.

Taken literally, the best thing to do for student group community builders is to convince everyone to switch plans in this way, and count that as success.

So it is not a bias. Quite the opposite: it is a very literal interpretation of the objective function, which was explicitly specified several years ago.

The meta-level point people should take from this is:

  • If you are in a position of influence, you should be super-careful before you introduce anything like a quantitative metric into EA culture. EAs love measuring impact, are optimisers, and will Goodhart hard
Comment by jan_kulveit on EA is vetting-constrained · 2019-03-09T02:52:39.605Z · score: 27 (12 votes) · EA · GW

I'm intermittently working on a project to provide more scaleable and higher quality feedback for project proposals for several months. First alpha-stage test should start in a time-horizon of weeks and I'll likely post the draft of the proposal soon.

Very rough reply ... the bottleneck is a combination of both of the factors you mention, but the most constrained part of the system is actually something like the time of senior people with domain expertise and good judgement (as far as we are discussing projects oriented on long-term, meta, AI alignment, and similar). Adding people to the funding organisations would help a bit, but less than you would expect: the problem is, for evaluating e.g. somewhat meta- oriented startup trying to do also something about AI alignment, as a grantmaker, you often do not have the domain experience, and need to ask domain experts, and sometimes macrostrategy experts. (If the proposal is sufficiently ambitious or complex or both, even junior domain experts would be hesitant to endorse it.) Unfortunately, the number of people with final authority is small, their time precious, and they are often very busy with other work.

edit: To gesture toward the solution ... the main thing the proposed system will try to do is "amplify" the precious experts. For some ideas how you can do it see Ozzie's posts, other ideas can be ported from academic peer review, other from anything-via-debate.

[meta: I'm curious, why was this posted anonymously?]

Comment by jan_kulveit on You Have Four Words · 2019-03-07T23:25:53.036Z · score: 5 (3 votes) · EA · GW

I'll probably refer people to this post when trying to explain why you totally need complex networks when you are trying to coordinate about absolutely anything more complicated than what you can express in 4 words.

(Also: One friend pointed toward the fact that the word hierarchy comes from organisation coordinating effort of more than a billion of people across long time horizons)

Comment by jan_kulveit on What to do with people? · 2019-03-07T21:25:38.288Z · score: 4 (2 votes) · EA · GW

My rough understanding:

To some extent the ideas seem to be now "in the water". The maths part is something now more developed under the study of complex networks. Alexander's general ideas about design inspired people to create wikis, patterns in software movement, to some extent objective oriented programming, and extreme programming, and some urbanists... which made me motivated to read more from of him.

(Btw in another response here, I pointed toward Wikipedia as a project with some interesting social technology behind it. So, it's probably worth to note that a lot of the social technology was originally created/thought about at wikis like Meatball and the original WikiWiki by Ward Cunningham who was in turn inspired by Alexander.)

Comment by jan_kulveit on What to do with people? · 2019-03-07T12:32:59.515Z · score: 3 (4 votes) · EA · GW

In my view without all the hierarchy stuff, it is harder to see what to create, start, manage, delegate. I would be significantly more worried about the meme of "just go&do things&manage others" spreading than about the meme "figure out how to grow the structure".

Comment by jan_kulveit on What to do with people? · 2019-03-07T00:56:24.253Z · score: 3 (2 votes) · EA · GW

I like the second link. From a network scientists perspective, one way how to model such structures is by overlapping hierarchical stochastic block models, or generally "community structure". (Alexander's essay predates network science by several decades.)

Which also makes “partonomy” and “mereonomy” possibly problematic labels (because of tree structures).

Comment by jan_kulveit on What to do with people? · 2019-03-06T22:45:57.717Z · score: 10 (6 votes) · EA · GW

Counting datapoints, for some time I worked on "translating" the community structure behind Wikipedia from the en: version to cz:; co-founded Czech Wikimedia chapter; led CZEA.

Wikipedia was created almost entirely by hierarchical volunteer structure, so you can take it as some sort of existential proof. (IMO possibly the most impressive thing about Wikipedia is the social technology behind it, and how much deliberate is it)

I agree that fully volunteer organisations are difficult, but something like 5 volunteers / 1 staffer is much easier. Also what makes large difference is volunteers working physically together vs. distributed online work. Also what is actually important is not so much whether people are getting payed, but how large fraction of attention/time can someone put into coordination. (I.e. a group of 10 volunteers where every member puts in 0.1FTE is much less effective than group of 6 volunteers on 0.1FTE + one coordinator on 0.4FTE)

Overall didn't meant to suggest anything is easy; on the other hand in my opinion it is in general easier to get good in managing volunteers than, for example, to get hired for some of the most competitive jobs in EA orgs.

Comment by jan_kulveit on What to do with people? · 2019-03-06T20:33:22.287Z · score: 4 (3 votes) · EA · GW

I agree some orgs are possibly close to the margin on how fast you can grow, but my view is we are way below that on the movement level.

One reason for that belief is just looking around on the amount of effort which is going in that direction. If you compare how much work and attention is spent on thinking about the structure, in comparison to the amount of work spent collectively on for example "selecting people for jobs", my impression is there is a difference of an order. So while you may be right that more effort would not help, in my view we are "not actually trying" (with some positive exceptions).

Another reason is looking on cheap actions people or orgs can take (as with the coaching knowhow transfer) which are not taken.

Comment by jan_kulveit on What to do with people? · 2019-03-06T17:50:24.808Z · score: 29 (14 votes) · EA · GW

As a whole, I think effective altruism is currently more structurally bottlenecked and network bottlenecked than funding bottlenecked. Improving the structural and networking constrains is higher leverage than adding more money to the system (on average). Which is not say increasing funding is not valuable. I would expect this to depend a lot on individual circumstances.

If you look on the funding bottlenecks, they seem to be mostly the result of the structure of the funding, that of aggregate sum: imagine a counterfactual world in which OpenPhil has $1b more than it has. How much more effective actions you would expect to see in the world?

So also in funding, we need more structures. EtG is highly impactful as far as the giving is smart, with money directed toward alleviating the structural constrains of funding. I don't think this scales "endlessly".

Another point is, from a global perspective, EtG makes more sense in some places than others. For example, a philosophy postdoc in Prague can earn as little as £1100/m. Should such a person drop an academic career, and do EtG? Almost certainly not. What about EAs in India? They have likely very different comparative advantages than EtG.

Comment by jan_kulveit on Effective Thesis project review · 2019-03-06T15:47:52.058Z · score: 4 (2 votes) · EA · GW

In my opinion it would be definitely worth to try! What's possibly not clear is that Effective Thesis is to some extent funding constrained, so in present it would need to get some additional sources of funding to run prizes

Comment by jan_kulveit on Neglected Goals for Local EA Groups · 2019-03-02T14:32:12.514Z · score: 9 (6 votes) · EA · GW

While I agree the current focus is too limited, I would generally advice against emphasising "general advice what to do everywhere" like the suggestions nr. 1 or 2. on global level, because of reasons mostly explained here. One sentence version is people greatly under-appreciate the differences caused by location.

(For example, while donating money may be a good option for someone working in fintech in the US, a philosophy postdoc in Prague can be earning about £1000/m.)

Meta-point is with increasing "distance" from Oxford and the Bay area, effective altruism groups need to do more of their own prioritization, and need to think more in terms of the "actual consequences" and less in terms of proxy metrices like number of etgs.

I agree "measuring projects" is important, but sees it's not that neglected - for example in evaluations in EA community building grants, there is explicit space for this.

I have quite a lot of theory on network effects, in part explicit, but very little time to write some accessible explanations. Also writing is slow and painful process for me. If anyone would be interested in collaborating on this and doing most of the actual writing, I would be happy to share it.

Comment by jan_kulveit on Pre-announcement and call for feedback: Operations Camp 2019 · 2019-02-21T01:49:41.888Z · score: 6 (5 votes) · EA · GW

Few comments

  • It seems to me there is some inherent tension between the goals "helping the camp participants" and "helping the organizations". The later puts more pressure on selection, which may not be that good for learning.
  • In case of the group nr.1, willingness to spend 4-6 weeks on such event is likely a credible signal of dedication to EA, but plausibly may be even slightly anti-correlated with ops talent (I would expect to find some of the best people organising something, and having higher opportunity costs).
  • In case of the group nr.2 and 4 days event the whole setup seems more balanced and fair when considering interests of the participants. Many EA orgs are somewhat special and the job market is less one-sided in the senior roles side.
  • General comment regarding all efforts related to talent: in the whole EA talent ecosystem people should think carefully about creating strange incentive landscapes and possibly moral hazards. I recommend spending a lot of thoughts on that, not necessarily in a public way.
Comment by jan_kulveit on The Need for and Viability of an Effective Altruism Academy · 2019-02-16T18:52:47.340Z · score: 12 (6 votes) · EA · GW

It seems to me the plan is based on several assumptions which actually do not hold

  • Effective altruism wants to grow the number of its members faster. It seems there was something like an attempt to deliberately slow down, limit outreach, international expansion, etc. based on problems with too fast expansion like coordination or diluted knowledge. The academy would likely help with such problems, but generally there is likely not such a strong urge to create new EAs now as you feel.
  • If EA wants to grow faster, there are cheaper ways.
  • There seems to be a persistent misconception in the community about how likely is it to get "initial grants" from OPP.

That said, something like EA Academy is a format which may be wort to explore some time in the future. (Other people thought about the idea before)

Comment by jan_kulveit on Three Biases That Made Me Believe in AI Risk · 2019-02-15T10:55:01.120Z · score: 21 (11 votes) · EA · GW

It's good to see some intelligent criticisms of the argument for doing AI safety research!

Just two short remarks to this post: I generally tend to think in the log scale about probabilities and it's possible I haven't used the a.bcd notation with any probability smaller than 10^-3 (0.1%) for years. So it is hard to see why I should be influenced by the described effect.

With language, your distinction is correlated with the distinction between two of Dennett's levels of abstraction (the design stance and the intentional stance). Claiming something like the design stance is more accurate or better than the intentional stance for analyzing present day systems seems too bold: it's really a different level of description. Would you say the design stance is also more accurate when thinking e.g. about animals?

Obviously, looking on any system with intentional stance comes with ...mentalizing, assuming agency. People likely utilize the different levels of abstraction not really well, but I'm not convinced they systematically over-utilize one. It seems arguable they under-utilize the intentional stance when looking at "emergent agency" in systems like the stock market.

Comment by jan_kulveit on Reflections on doing good with lump sums - the retired person's dilemma · 2019-02-09T14:11:34.636Z · score: 16 (7 votes) · EA · GW

I think the question how much to give over time in such situation is a good one, and hope someone will write a carefully considered answer.

I'd like to push back a bit on this part of the reasong

I am persuaded that there is a case for local charities or those with which I have a personal connection because these cannot be on the radar of the big charity evaluators. And if everyone did this, and all surplus money were channelled to the "best" causes as assessed by self-appointed experts, I'm not convinced the world would be a better place. A bit, or a lot, of anarchy is needed, I think. Especially if the internet tendency to encourage information monopolies kicks in and everyone consults the same oracle.

We are certainly not in a world where everyone would consult effective altruist sources. On the contrary, I think the correct view is that basically everybody gives to local/familiar charities randomly and based on emotional appeal, and just a very tiny fraction of people is influenced by any rational advice at all. If you are considering EA viewpoint, you are an exception.

To put things in scale, the UK based "Dog Trust", just one of many charities in the UK supporting pet wellfare, had an income £106.4m in 2017. In comparison, the Anti-malaria foundation, for many years a top charity in GiveWell lists, had an annual income just $46,8m. Obviously the dogs in the UK are closer to people there than the people AMF is helping.

So to a first approximation, I think you can say that almost nobody gives effectively, and everybody gives randomly to charities based on availability heuristics and similar. Based on this reasoning I would expect if you support more anarchy, you will basically do what everybody is doing anyway, and the impact of that will be small to negligible. I would expect giving to effective charities, EA funds, and EA meta-charities to be better for almost any goal you may have.

Comment by jan_kulveit on List of possible EA meta-charities and projects · 2019-01-15T07:03:36.672Z · score: 6 (4 votes) · EA · GW

Just wanted to add that that while I think many of the listed ideas are in my opinion useful and should eventually turn into projects, some of them are quite sensitive to the way how they are executed, or who executes them, to the extent that creating negative impact is easy.

Also, often, there is some prior work, existing knowledge, etc. in many of the listed directions. That no project is visible in some direction may mean also that someone carefully considered it and decided it is not something which should be started now.

(For example: it's not like nobody thought about EA outreach to different demographics, Muslim, seniors / retirees, other cultures / countries. There is in part public, in part "internal" discussion about this, and the consensus seems to be this is in many cases delicate and should not be rushed.

Or: it's not like EffectiveThesis did not considered or experimented with different intervention points in the academic chain.)

Comment by jan_kulveit on Should donor lottery winners write reports? · 2019-01-14T12:40:50.347Z · score: 2 (1 votes) · EA · GW

I disagree. You should not have as a central example some sort of secret, but trust. Transitivity of trust is limited, and everybody has a unique position in the trust network. Many will have interesting opportunities in their network neighborhoods. (I don't claim to be typical, but still: I can easily list maybe a dozen of such not easily justifiable opportunities where I could send money; even if I'm somewhere on the tail on the distribution, I'd guess typical lottery winner has at leas 1 or 2 such opportunitites)