Limits to Legibility 2022-06-29T17:45:58.461Z
Ways money can make things worse 2022-06-21T15:26:52.855Z
Continuity Assumptions 2022-06-13T21:36:59.560Z
Different forms of capital 2022-04-25T08:05:25.123Z
Case for emergency response teams 2022-04-05T11:08:22.173Z
How we failed 2022-03-23T10:08:06.861Z
What we tried 2022-03-21T15:26:30.067Z
Hinges and crises 2022-03-17T13:43:04.755Z
Experimental longtermism: theory needs data 2022-03-15T10:05:35.294Z
Epistea Summer Experiment (ESE) 2020-01-24T10:51:00.672Z
How x-risk projects are different from startups 2019-04-05T07:35:39.513Z
Request for comments: EA Projects evaluation platform 2019-03-20T22:36:32.565Z
OpenPhil: Reflections on 2018 Generalist Research Analyst Recruiting 2019-03-08T02:41:44.804Z
What to do with people? 2019-03-06T11:04:21.556Z
Critique of “Existential Threats” chapter in Enlightenment Now 2018-11-21T10:09:54.552Z
Suggestions for developing national-level effective altruism organizations 2018-10-17T23:35:37.241Z
Why develop national-level effective altruism organizations? 2018-10-17T23:29:44.203Z
Effective Thesis project review 2018-05-31T18:45:22.248Z
Review of CZEA "Intense EA Weekend" retreat 2018-04-05T20:10:04.290Z
Optimal level of hierarchy for effective altruism 2018-03-27T22:32:15.211Z
Introducing Czech Association for Effective Altruism - history 2018-03-12T22:01:49.556Z


Comment by Jan_Kulveit on Open EA Global · 2022-09-12T16:16:13.609Z · EA · GW

Just flagging that in my view the goal to have just 1 EAGx per region, and make the EAGx regionally focused, with taking very few people from outside the region, is really bad, in my view. Reasons for this are in the effects on network topology, and subsequently on the core/periphery dynamic.

I find the argument about the cost of "flying everyone from around the world to one location" particularly puzzling, because this is not what happens by default: even if you don't try to push events to being regional at all, they naturally are, just because people will choose the event which is more conveniently located closer to them. So it's not like everyone flying everywhere all the time (which may be the experience of the events team, but not of typical participants).

Comment by Jan_Kulveit on A concern about the “evolutionary anchor” of Ajeya Cotra’s report on AI timelines. · 2022-08-21T21:24:55.200Z · EA · GW

Not that important, but...  in terms of what intuitions people have, the split of the computation into neurons/environment  is not a reasonable model of how life works. Simple organisms do a ton of non-neuron-based computations distributed across many cells, and are able to solve pretty complex optimization problems. The neurons/environment  split pushes this into the environment , and this means the environment was sort of complex in a way for which people don't have good ituitions (e.g. instead of mostly thinking about costs of physics simulation, they should include stuff like immunse system simulations).

Comment by Jan_Kulveit on Announcing: the Prague Fall Season · 2022-08-01T07:24:50.293Z · EA · GW

It seems to me that there is some subtle confusion going on here. 

0. It's actually more about the 'Season'.

1. This isn't really "a push to establish a community outside of The Bay or Oxford", as that community already exists in Prague for some time. E.g. Prague had it's coworking space since ca 2017, sooner than almost anywhere else, already has something like ˜15 FTE ppl working on EA/longtermist relevant projects, etc.  I think to some extent what happened over past few years was the existing Prague hub focused too much on 'doing the work' and comparably less on 'promoting the place' or 'writing posts about how it is a hub on EA forum'. So, in the hub dynamics, more than 'establishing something', perhaps you can view this as 'creating common knowledge about something' /  'upgrade'.

2.  I think structure with 'one giant hub' is bad not only for suving physical catastrophe, but mainly because more subtle memetics and social effects, talent-routing, and overall robustness. For example: if the US cultural wars stuff escalated and EA become subject of wrath of one of the sides, it could have large negative effects not only directly due to hostile environment, but also due to secondary reactions of EA, induced opinion polarization, etc. 

3. On practical level, I think the strongest current developments toward  multi-hub network structure are often clearly sensible - for example, not having visible presence on the East Coast was in my view a bug, not a feature. 

Comment by Jan_Kulveit on Why AGI Timeline Research/Discourse Might Be Overrated · 2022-07-03T09:28:07.779Z · EA · GW

I do broadly agree with the direction and the sentiment: on the margin, I'd be typically interested in other forecasts than "year of AGI" much more.

For example: in time where we get "AGI" (according to your definition) ... how large fraction of GDP are AI companies? ... how big is AI as a political topic? ... what does the public think?

Comment by Jan_Kulveit on Community Builders Spend Too Much Time Community Building · 2022-06-29T09:29:59.654Z · EA · GW

Strongly upvoted. 

In my view, part of the problem are feedback loops in the broader EA scene, where focus on "marketing" was broadly rewarded and copied. (Your uni group grew so large so fast! How we can learn what you did and emulate it?) . 

Also - I'm not sure what metrics are now evaluated by central orgs when people ask for grants or grant renewals, but I suspect something like "number of highly engaged EAs produced" is/was prominent, and an optimizer focusing on this metric will tend to converge on marketing, and will try to bring in more (highly engaged) marketers. 

Comment by Jan_Kulveit on Impact markets may incentivize predictably net-negative projects · 2022-06-22T10:38:46.390Z · EA · GW

If the main problem you want to solve is "scaling up grantmaking", there are probably many other ways how to do it other than "impact markets". 

(Roughly, you can amplify any "expert panel of judges" evaluations with judgemental forecasting.)

Comment by Jan_Kulveit on Impact markets may incentivize predictably net-negative projects · 2022-06-22T10:28:58.961Z · EA · GW

Seems a case of "How x-risk projects are different from startups"

Comment by Jan_Kulveit on On Deference and Yudkowsky's AI Risk Estimates · 2022-06-20T03:37:34.712Z · EA · GW

(i.e. most people who are likely to update downwards on Yudkowsky on the basis of this post, seem to me to be generically too trusting, and I am confident I can write a more compelling post about any other central figure in Effective Altruism that would likely cause you to update downwards even more)

My impression is the post is somewhat unfortunate attempt to "patch" the situation in which many generically too trusting people updated a lot on AGI Ruin: A List of Lethalities  and Death with Dignity  and subsequent deference/update cascades. 

In my view the deeper problem here is instead of disagreements about model internals, many of these people do some sort of "averaging conclusions" move, based on signals like seniority, karma, vibes, etc. 

Many of these signals are currently wildly off from truth-tracking, so you get attempts to push the conclusion-updates directly. 


Comment by Jan_Kulveit on What’s the theory of change of “Come to the bay over the summer!”? · 2022-06-08T20:01:20.487Z · EA · GW

I. It might be worth reflecting upon how large part of this seem tied to something like "climbing the EA social ladder".

E.g. just from the first part, emphasis mine

Coming to Berkeley and, e.g., running into someone impressive  at an office space already establishes a certain level of trust since they know you aren’t some random person (you’ve come through all the filters from being a random EA to being at the office space).
If you’re in Berkeley for a while you can also build up more signals that you are worth people’s time. E.g., be involved in EA projects, hang around cool EAs.

Replace "EA" by some other environment with prestige gradients, and you have something like a highly generic social climbing guide. Seek cool kids, hang around them, go to exclusive parties, get good at signalling.

II. This isn't to say this is bad . Climbing the ladder to some extent could be instrumentally useful, or even necessary, for an ability to do some interesting things, sometimes.

III. But note the hidden costs. Climbing the social ladder can trade of against building things. Learning all the Berkeley vibes can trade of against, eg., learning the math actually useful for understanding agency. 

I don't think this has any clear bottom line - I do agree for many people caring about EA topics it's useful to come to the Bay  from time to time. Compared to the original post  I would probably mainly suggest to also consult virtue ethics and think about what sort of person you are changing yourself to, and if you, for example, most want to become "a highly cool and well networked EA" or e.g. "do  things which need to be done", which are different goals.

Comment by Jan_Kulveit on Getting GPT-3 to predict Metaculus questions · 2022-05-07T07:05:30.305Z · EA · GW

Suggested variation, which I'd expect to lead to better results: use raw "completion probabilities" for different answers.

E.g. with prompt "Will Russia invade Ukrainian territory in 2022?" extract completion likelihoods of the next few tokes "Yes" and "No". Normalize

Comment by Jan_Kulveit on Case for emergency response teams · 2022-04-10T15:39:07.007Z · EA · GW

Also the direction of ALERT is generally more on "doing". Doing seems often very different from forecasting, often needs different people - part of the relevant skills is plausibly even anticorrelated.

Comment by Jan_Kulveit on [deleted post] 2022-04-10T15:32:48.591Z

Crisis response is a broader topic. I would probably suggest creating additional tag for Crises response (most of our recent sequence would fit there)

Comment by Jan_Kulveit on "Long-Termism" vs. "Existential Risk" · 2022-04-07T14:04:09.523Z · EA · GW

I don't have a strong preference. There a some aspects in which longerism can be better framing, at least sometimes.

I. In a "longetermist" framework, x-risk reduction is the most important thing to work on for many orders of magnitude of uncertainty about the probability of x-risk in the next e.g. 30 years. (due to the weight of the long term future). Even if AI related x-risk is only 10ˆ-3 in next 30 years, it is still an extremely important problem or the most important one. In a "short-termist" view with, say, a discount rate of 5%, it is not nearly so clear.

The short-termist urgency of x-risk ("you and everyone you know will die") depends on the x-risk probability being actually high, like of order 1 percent, or tens of percents . Arguments why this probability is actually so high are usually brittle pieces of mathematical philosophy (eg many specific individual claims by Eliezer Yudkowsky) or brittle use of proxies with lot of variables obviously missing from the reasoning (eg the report by Ajeya Cotra). Actual disagreements about probabilities are often in fact grounded in black-box intuitions  about esoteric mathematical concepts.  It is relatively easy to come with brittle pieces of philosophy arguing in the opposite direction: why this number is low. In fact my actual, action guiding estimate is not based on an argument conveyable by a few paragraphs, but more on something like "feeling you get after working on this over years". What I can offer other is something like "an argument from testimony", and I don't think it's that great. 

II. Longermism is a positive word, pointing toward the fact that future could be large and nice. X-risk is the opposite. 

Similar: AI safety  vs AI alignment. My guess is the "AI safety" framing is by default more controversial and gets more of a pushback (eg  "safety department" is usually not the most loved part of an organisation, with connotations like "safety people want to prevent us from doing what we want")


Comment by Jan_Kulveit on Off Road: Interviews with EA College Dropouts · 2022-04-05T12:34:13.522Z · EA · GW

Title EA Dropouts  seems a bit confusing, because it can be naturally interpreted as people who dropped out of EA

Comment by Jan_Kulveit on What we tried · 2022-04-05T12:10:12.902Z · EA · GW

I had little influence over the 1st wave, credit goes elsewhere. 

What happened in subsequent waves is  complicated.  One sentence version is Czechia changed minister of health 4 times, only some of them were reasonably oriented, and how much they were interested in external advice differed a lot in time. 

Note that the "death tolls per capita in the world" stats are  misleading, due to differences in reporting. Czechia had average or even slightly lower than average mortality compared to "Eastern Europe" reference class, but much better reporting. For more reliable data, see

Comment by Jan_Kulveit on Awards for the Future Fund’s Project Ideas Competition · 2022-03-25T12:41:00.790Z · EA · GW

Both "EA translation service" and "EA spaces everywhere" seem like ideas which can take good, but also many bad or outright harmful forms. 

A few years ago, I tried to describe how to to establish a robustly good "local effective altruism" in a new country or culture (other than Anglosphere).

The super brief summary is
1. it's not about translating a book, but about “transmission of a tradition of knowledge”
2. what is needed is a highly competent group of people 
3. who can, apart from other things, figure out what the generative question of EA means in the given context (which may be quite different from Oxford or Bay)

Point #2 is the bottleneck. Without it, efforts to have "country/language X EA" will be often unsuccessful or worse. Doing Good Better was translated to e.g. japanese. It is doing  a bit worse on than the english version on, but not  by many orders. Yet you aren't seeing a lot of content and projects from EA Japan.

So: One good version of "translation service" seems to be basically giving functional national EA groups money to get as good translations  as needed.

One bad version is a "centralized project" trying to translate centrally selected content to centrally selected languages by hiring professional contractors to do it.

Similarly EA spaces:

Hiring offices in 100 cities is easy, and you can do that centrally. Running a space with a good culture and gatekeeping is harder, and bottlenecked on "#2".

Good version of the project seems to be  basically giving functional city EA groups money to get good coworking spaces, which is probably doable via CEA group support.

Many bad versions are somewhere close to "centralized megaproject to run EA spaces everywhere".

In other words: it isn't easy to get around the bottleneck 'you need people on the ground'. 

Footnote: there are many variations on this theme. The basic pattern roughly is:

 "notice that successful groups are doing activity X" (e.g. they have a coworking space, or materials in the local language, or they organize events,...).  

the next step, in which this goes wrong, is, "let's start a project P(X) that will try to make activity X happen everywhere".

Also: The chances of bad or explicitly harmful outcomes increase roughly in proportion to the combination of cultural, network and geographic distance from the "centre" from which such activity should be directed. Eg project which will try to run spaces in Stanford from Berkeley seem fine

Comment by Jan_Kulveit on How we failed · 2022-03-24T22:50:51.758Z · EA · GW

Mass-reach posts came later, but sooner than the US mainstream updates

Comment by Jan_Kulveit on How we failed · 2022-03-24T17:19:57.831Z · EA · GW

The practical tradeoff was between what, where and when to publish. The first version of the preprint which is on medrxive contains those estimates. Some version with them could probably be published in a much worse journal than Science, and would have much less impact.

We could have published them separately, but a paper is a lot of work, and it's not clear to me whether, for example, to sacrifice some of the"What we tried"and get this done would have been a good call. 

It is possible to escape from the game in specific cases - in the case of covid, for example, the advisory body we created in the Czech Republic was able to take into account analyses based on "internal quality", especially if it was clear peer review game will take months. If such bodies existed in more/more countries, it would be possible. 
Similarly, it could be done with the help of an ECDC or WHO type institution. 

In general, it's an "inadequate equilibrium" type of problem, I have some thoughts on typical solutions to them, but not in easily shareable written form, at the moment.

Comment by Jan_Kulveit on How we failed · 2022-03-23T14:00:27.103Z · EA · GW

NPI & IFR: thanks, it's now explained in the text.

Re: Rigour

I think much of the problem is due not to our methods being "unrigourous" in any objective sense, but to interdisciplinarity. For example, in the survey case, we used mostly standard methods from a field called "discrete choice modelling" (btw, some EAs should learn it - it's a pretty significant body of knowledge on "how to determine people's utility functions").  

Unfortunately, it's not something commonly found in the field of, for example, "mathematical modeling of infectious diseases". It makes it more difficult for journals to review such a paper, because ideally they would need several different reviewers for different parts of the paper. This is unlikely to happen in practice, so usually the reviewers tend to either evaluate everything according to the conventions of their field, or to be critical and dismissive of things they don't understand.  

Similar thing is going on with use of "forecasting"-based methods. There is published scientific literature on their use, their track record is good, but before the pandemic there was almost no "published literature" on the subject of their use in combination with epidemic modelling (there is now!). 

The second part of the problem is that we were ultimately more interested in "what is actually true" than what "looks rigorous". A paper that contains few pages of equations, lots of complex modeling, and many simulations can look "rigorous" (in the sense of the stylized dialogue). If at the same time, for example, it contains completely and obviously wrong assumptions about the IFR of covid it will still pass many tests of "rigorousness" because it only shows that "under assumptions that do not hold in our world we reach conclusions that are irrelevant to our world" (the implication is true). At the same time, it can have disastrous consequences, if used by policymakers, who assume something like "research tracks reality".

Ex post, we can demonstrate that some of our methods (relying on forecasters) were much closer to reality (e.g. based on serological studies) than a lot of published stuff.

Ex ante, it was clear this will be the case to many people who understand both academic research and forecasting.

Re: Funding

For the record, EpiFor is a project that has ended, and is not seeking any funding.  Also, as noted in the post, we were actually able to get some funding offered: just not in a form which the university was able to accept, etc. 

It's not like there is one funder evaluating whether to fund IHME, or EpidemicForecasting. In my view the problems pointed to here are almost completely unrelated, and I don't want them to get conflated in some way 


Comment by Jan_Kulveit on What we tried · 2022-03-22T08:08:07.078Z · EA · GW

Hi, as the next post in the sequencd is about 'failures' I think it would be more useful after that is published.

Comment by Jan_Kulveit on Hinges and crises · 2022-03-21T14:30:58.527Z · EA · GW

Sorry, but this seems to me to confuse the topic of the post "Experimental Longtermism" and the topic of this post. Note that the posts are independent, and about different topics. 

The table in this post is about timescales of OODA loops (observe–orient–decide–act), not about feedback. For example, in a situation which is unfolding on a timescale of days and weeks, as was the early response to covid, some actions are just too slow to have an impact: for example, writing a book, or funding basic research. The same is true for some decision and observation making processes: the speed of the whole loop matters, and if the decide part needs, for example, to 50 ppl from 5 different organizations to convene and reach a consensus, it won't work.


Comment by Jan_Kulveit on The Future Fund’s Project Ideas Competition · 2022-03-18T12:15:34.901Z · EA · GW

Note that CSER is running a project roughly in this direction.

Comment by Jan_Kulveit on Where would we set up the next EA hubs? · 2022-03-16T14:36:54.030Z · EA · GW

Thanks for sharing!  We plan to announce some new significant effort in Prague in next ~1 month, and also likely will offer some aid to people moving to Prague. If anyone is interested in moving somewhere in Europe, send me a pm. 

Basic reasoning is Prague is pareto-optimal on some combination of 'existing community (already 10-20 FTE people working on longtermist projects, other ~10FTE job openings this year)', 'better quality of living', 'costs to relocate', 'in Europe', 'cultural sanity'. There wasn't much effort going into promoting Prague in past two years, but that's likely to change.

- I don't think any european AI efforts apart from DeepMind are important
- It is something between 'hard' and 'almost impossible' to start a community somewhere without a strong grass-root effort by a local group; top-down reasoning of the style 'it would be good to have a hub in Japan' leads to no practical results 

Comment by Jan_Kulveit on Experimental longtermism: theory needs data · 2022-03-16T09:39:24.460Z · EA · GW

Millions is probably a safe bet/lower bound: majority won't be via direct twitter reads, but via mainstream media using it in their writing. 

With twitter, we have a better overview in the case of our other research on seasonality (still in review!). Altmetric estimate is it was shared with accounts with an upper bound of 13M followers. However, in this case, almost all the shares were due to people retweeting my summary. Per twitter stats, it got 2M actual impressions. Given the fact the NPI research was shared and referenced more, it's probably more >1M  reads just on twitter.

Re: forecasting (or bets). In a broad sense, I do agree. In practice I'm a bit skeptical that a forecasting mindset is that good for generating ideas about "what actions to take". "Successful planning and strategy" is often something like "making a chain of low-probability events happen", which seems distinct, or even at tension with typical forecasting reasoning. Also, empirically, my impression is that forecasting skills can be broadly decomposed into two parts - building good models / aggregates of other peoples models, and converting those models into numbers. For most people, the "improving at converting non-numerical information into numbers" part has initially much better marginal returns (e.g. just do calibration trainings...), but I suspect doesn't do that much for the "model feedback".


Comment by Jan_Kulveit on How big are risks from non-state actors? Base rates for terrorist attacks · 2022-02-16T12:44:57.581Z · EA · GW

Handy reference! Apart from the average rate, it seems also interesting to notice the variance in the table, spread over 4 orders of magnitude. This may point to something like 'global sanity' being an important existential risk factor. 


Comment by Jan_Kulveit on The Cost of Rejection · 2021-10-08T14:03:10.783Z · EA · GW

I mostly agree with the problem statement.

With the proposed solution of giving people feedback - I've historically proposed this on various occasions, and from what I have heard, one reason for not giving feedback on the side of organizations is something like "feedback opens up space for complaints, drama on social media, or even litigation". The problem looks very different from the side of the org: when evaluating hundreds of applications, it is basically certain some errors are made, some credentials misunderstood, experiences not counted as they should, etc. - but even if the error rate is low, some people will rightfully complain, making hiring processes even more costly. Other question is, what is the likelihood of someone from the hundreds of applicants you don't know doing something bad with the feedback - ranging from "taking it too seriously" to "suing the org for discrimination". (Where the problem is more likely to come from the non-EA applicants).

I'm not saying this is the right solution, but it seems like a reasonable consideration.

One practical workaround: if you really want feedback, and ideally know someone in the org, what sometimes works is asking informally +signaling you won't have do anything very unreasonable with the feedback.

Comment by Jan_Kulveit on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-10T14:06:16.574Z · EA · GW

I would guess the 'typical young researcher fallacy' also applies to Hinton  - my impression is he is  basically advising his past self, similarly to Toby. As a consequence,  the advice is likely  sensible for people-much-like-past-Hinton, but  not a good general advice for everyone.

In  ~3 years most people are able to re-train their intuitions a lot (which is part of the point!). This seems particularly dangerous in cases where expertise in the thing you are actually interested in does not exist, but expertise in something somewhat close does -  instead of following your curiosity, you 'substitute the question' with a different question, for which a PhD program exists, or senior researchers exist, or established directions exist. If your initial taste/questions was better than the expert's, you run a risk of overwriting your taste with something less interesting/impactful.

Anecdotal illustrative story:

Arguably, large part of what are now the foundations of quantum information theory / quantum computing could have been discovered much sooner, together with taking more sensible interpretations of quantum mechanics than Copenhagen interpretation seriously. My guess what was happening during multiple decades (!) was many early career researchers were curious what's going on, dissatisfied with the answers, interested in thinking about the topic more... but they were given the advice along the lines 'this is not a good topic for PhDs or even undergrads; don't trust your intuition; problems here are distasteful mix of physics and philosophy; shut up and calculate, that's how a real progress happens' ... and they followed it; acquired a taste telling them that solving difficult scattering amplitudes integrals using advanced calculus techniques is tasty, and thinking  about deep things formulated using tools of high-school algebra is for fools.   (Also if you did run a survey in year 4 of their PhDs, large fraction of quantum physicists would probably endorse the learned  update from thinking about young foolish questions about QM interpretations to the serious and publishable thinking they have learned.)


Comment by Jan_Kulveit on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-10T13:16:21.044Z · EA · GW

Let's start with the third caveat: maybe the real crux is what we think are the best outputs;  what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples - so it's e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth.  As far as I can tell, none of these seems to be following the proposed 'formula for successful early-career research'. 

My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI), and subsequently, the advice should be more applicable to them. I would even say part of what makes a field "established" actually is something like "somewhat clear direction in the space of unknown in which people are trying to push the boundary" and "shared taste in what is good, according to the direction". (The general direction or at least the taste seems to be ~ self-perpetuating once the field is "established", sometimes beyond the point of usefulness). 

In contrast to your experience with AI students in Berkeley, in my experience about ~20% of ESPR students have generally good ideas even while at high school/first year in college, and I would often prefer these people to think about ways in which their teachers, professors or seniors are possibly confused, as opposed to learning that their ideas are now generally bad and they should seek someone senior to tell them what to work on. (Ok - the actual advice would be more complex and nuanced, something like "update on the idea  taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people"). (ESPR is also very selective, although differently.) 

With hypothetical surveys, the conclusion (young researchers should mostly defer to seniors in idea taste) does not seem to follow from estimates like "over 80% of them would think their initial ideas were significantly worse than their later ideas".  Relevant comparison is something like "over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas". Would you guess the answer would still be 80%? 


Comment by Jan_Kulveit on Announcing the launch of EA Impact CoLabs (beta) + request for projects, volunteers and feedback · 2021-09-08T15:12:24.182Z · EA · GW

It's good to see a new enthusiastic team  working on this! My impression, based on working on the problem ~2 years ago is this has good chances to provide value in global health a poverty, animal suffering, or parts of meta- cause areas; in case of x-risk focused projects, something like a 'project platform' seems almost purely bottlenecked by vetting. In the current proposal this seems to mostly depend on "Evaluation Commission"->  as a result,  the most important part for x-risk projects seems judgement of members of this commission and/or it's ability to seek external vetting

Comment by Jan_Kulveit on How to succeed as an early-stage researcher: the “lean startup” approach · 2021-09-08T13:17:45.701Z · EA · GW

In my view this text should come with multiple caveats.

- Beware 'typical young researcher fallacy'. Young researchers are very diverse, and while some of them will benefit from the advice, some of them will not. I do not  believe there is a general 'formula for successful early-career research'. Different people have different styles of doing research, and even different metrics for  what 'successful research' means. While certainly many people would benefit from the advice 'your ideas are bad', some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the"senior researchers". 

- Beware 'generalisation out of training distribution' problems. Compared to some other fields, AI governance as studied by Allan Dafoe is relatively well decomposed into a hierarchy of problems and you can meaningfully scale it by adding junior people and telling them what to do (work on sub-problems senior people consider interesting). This seems more typical for research fields with established paradigms than for fields which are pre-paradigmatic, or fields in need of a change of paradigm. 

- Large part of the described  formula for success seems to be optimised for success in the direction getting attention of senior researchers, writing something well received, or similar. This is highly practical, and likely good for many people in fields like Ai governance; at the same time, it seems the best research outputs by early career researchers in eg AI safety do not follow this generative pattern, and seem to be motivated more by curiosity,  reasoning from first principles, and  ignoring authority opinions.

Comment by Jan_Kulveit on EA Group Organizer Career Paths Outside of EA · 2021-07-21T10:10:28.696Z · EA · GW

Contrary to what seems an implicit premise of this post,  my impression is 

- most EA group organizers  should have this as a side-project, and should not think about "community building" as about their "career path" where they could possibly continue to do it in a company like Salesforce
- the label "community building" is unfortunate for what most of the EA group organizing work should consist of
- most of the tasks in "EA community building" involve skills which are pretty universal a generally useable in most other fields, like "strategizing", "understanding people", "networking" or  "running events"
- for example: in my view, what can an EA group organizer on a research career path get from  organizing an EA group as a side-project are skills like "organizing event", "explaining complex ideas to people" or even "thinking clearly in groups about important topics". Often the benfits of improving/practicing such skills for a research career are similar or larger than e.g. learning a new programming language

There are exceptions to this, such as people who want to work on large groups full time, build national groups, or similar. In my view these projects are often roughly of the scope of founding or leading a startup or a NGO and should be attempted by people who, in general, have a lot of optionality in what to do both before working on an EA group and eventually after it. 

Vint Cerf seems actually more of a counterexample toward "community building and evangelism" as a career objective: anyone who wants to follow this path should note he wrote the TCP protocol internet is still running on first, co-founded one of the entities governing internet later, and worked for Google on community building only after all these experiences. 

Another reason I'm sceptical of the value of this argument is my guess is people who would be convinced by it ("previously I was hesitant about organizing an EA group because the career path seems too narrow and tied to EA, now I see career paths in for-profit world") are people who should mostly not lead or start EA groups. In most cases EA group organizing  involves significant amount of talking to people about careers, and whoever has so limited understanding of the careers to benefit from this advice seems likely to have  non-trivial chance of giving people harmful career advice.

Comment by Jan_Kulveit on How much does performance differ between people? · 2021-04-10T16:09:39.771Z · EA · GW


For different take on very similar topic check  this discussion between me and Ben Pace  (my reasoning was  based on the same Sinatra paper).

For practical purposes, in case of scientists, one of my conclusions was

Translating into the language of digging for gold, the prospectors differ in their speed and ability to extract gold from the deposits (Q). The gold in the deposits actually is randomly distributed. To extract exceptional value, you have to have both high Q and be very lucky. What is encouraging in selecting the talent is the Q seems relatively stable in the career and can be usefully estimated after ~20 publications. I would guess you can predict even with less data, but the correct "formula" would be trying to disentangle interestingness of the problems the person is working on from the interestingness of the results.


For practical purposes, my impression is some EA recruitment efforts could be more often at risk of over-filtering by ex-ante proxies and being bitten by tails coming apart, rather than at risk of not being selective enough.

Also, often the practical optimization question is how much effort you should spend on on how extreme tail of the ex-ante distribution. 


Meta-observation is someone should really recommend more EAs to join the complex systems / complex networks community.  

Most of the findings from this research project seem to be based on research originating in complex networks community, including research directions such as "science of success", and there is more which can be readily used,  "translated" or distilled. 

Comment by Jan_Kulveit on Some thoughts on EA outreach to high schoolers · 2020-09-23T20:03:42.134Z · EA · GW

First EuroSPARC was in 2016. Targeting 16-19 year olds, my prior would be participants should still mostly study, and not work full-time on EA, or only exceptionally.

Long feedback loops are certainly a disadvantage.

Also in the meantime ESPR underwent various changes and actually is not optimising for something like "conversion rate to an EA attractor state".

Comment by Jan_Kulveit on The case of the missing cause prioritisation research · 2020-09-10T10:31:32.630Z · EA · GW

Quick reaction:

I. I did spent a considerable amount of time thinking about prioritisation (broadly understood)

My experience so far is

  • some of the foundations / low hanging sensible fruits were discovered
  • when moving beyond that, I often run into questions which are some sort of "crucial consideration" for prioritisation research, but the research/understanding is often just not there.
  • often work on these "gaps" seems more interesting and tractable than trying to do some sort of "lets try to ignore this gap and move on" move

few examples, where in some cases I got to writing something

  • Nonlinear perception of happiness - if you try to add utility across time-person-moments, it's plausible you should log-transform it (or non-linearly transform it) . sums and exponentiation do not commute, so this is plausibly a crucial consideration for part of utilitarian calculations trying to be based on some sort of empirical observation like "pain in bad"
  • Multi-agent minds and predictive processing - while this is framed as about AI alignment, super-short version of why this is relevant for prioritisation is: theories of human values depend on what mathematical structures you use to represent these values. if your prioritization depnds on your values, this is possible important
  • Another example could be the style of thought explained in Eliezer's "Inadequate Equillibria". While you may not count it as "prioritisation research", I'm happy to argue the content is crucially important for prioritisation work on institutional change or policy work. I spent some time thinking about "how to overcome inadequate equillibria", which leads to topics from game theory, complex systems, etc.

II. My guess is there are more people who work in a similar mode, trying to basically 'build as good world model as you can', dive into problems you run into, and at the end prioritise informally based on such a model. Typically I would expect such model to be in parts implicit / be some sort of multi-model ensemble / ...

While this may not create visible outcomes labeled as prioritisation, I think it's important part of what's happening now

Comment by Jan_Kulveit on 'Existential Risk and Growth' Deep Dive #2 - A Critical Look at Model Conclusions · 2020-08-27T10:41:09.392Z · EA · GW

I posted a short version of this, but I think people found it unhelpful, so I'm trying to post somewhat longer version.

  • I have seen some number of papers and talks broadly in the genre of "academic economy"
  • My intuition based on that is, often they seem to consist of projecting complex reality into a space of single-digit real number dimensions and a bunch of differential equations
  • The culture of the field often signals solving the equations is profound/important, and the how you do the projection "world -> 10d" is less interesting
  • In my view for practical decision making and world-modelling it's usually the opposite: the really hard and potentially profound part is the projection. Solving the maths is in often is some sense easy, at least in comparison to the best maths humans are doing
  • While I overall think the enterprise is worth to pursue, people should in my view have a relatively strong prior that for any conclusions which depends on the "world-> reals" projection there could be many alternatives leading to different conclusions; while I like the effort in this post to dig into how stable the conclusions are, in my view people who do not have cautious intuitions about the space of "academic economy models" could still easily over-update or trust too much the robustness
  • If people are not sure, an easy test could be something like "try to modify the projection in any way, so the conclusions do not hold". At the same time this will usually not lead to an interesting or strong argument, it's just trying some semi-random moves is the model space. But it can lead to a better intuition.
  • I tried to do few tests in a cheap and lazy way (eg what would this model tell me about running at night on a forested slope?) and my intuitions was:
  • I agree with the cautious the work in the paper represents very weak evidence for the conclusions that follow only from the detailed assumptions of the model in the present post. (At the same time it can be an excellent academic economy paper)
  • I'm more worried about other writing about the results, such as linked post on Phil's blog , which in my reading signals more of "these results are robust" than it's safe
  • Harder and more valuable work is to point to something like some of the most significant way in which the projection fails" (aspects of reality you ignored etc.). In this case this was done by Carl Shulman and it's worth discussing further
  • In practice I do have some worries about some meme 'ah, we don't know, but given we don't know, speeding up progress is likely good' (as proved in this good paper) being created in the EA memetic ecosystem. (To be clear I don't think the meme would reflect what Leopold or Ben believe)
Comment by Jan_Kulveit on 'Existential Risk and Growth' Deep Dive #2 - A Critical Look at Model Conclusions · 2020-08-25T09:44:31.855Z · EA · GW

In my view

  • a safe way how to read the paper is as academic economy - the paper says what happens if you solve a particular set of equations
  • while the variable names used in the equations appear to point toward reality, in fact it is almost completely unclear if the model is a reasonable map of at least some aspect of the territory

Overall I think a good check for EAs if they should update based on this result is

  • would you be able to make different set of at first glance reasonable assumptions of the same type, leading to opposite conclusions?

where if the answer is "no", I would suggest people basically should not update.

Comment by Jan_Kulveit on Neglected EA Regions · 2020-02-18T14:05:50.459Z · EA · GW

I'm not sure you've read my posts on this topic? (1,2)

In the language used there, I don't think the groups you propose would help people overcome the minimum recommended resources, but are at the risk of creating the appearance some criteria vaguely in that direction are met.

  • e.g., in my view, the founding group must have a deep understanding of effective altruism, and, essentially, the ability to go through the whole effective altruism prioritization framework, taking into account local specifics to reach conclusions valid at their region. This basically impossible to implement as membership requirement in a fb group
  • or strong link(s) to the core of the community ... this is not fulfilled by someone from the core hanging in many fb groups with otherwise unconnected ppl

Overall, I think sometimes small obstacles - such as having to find EAs from your country in the global FB group or on EA hub and by other means - are a good thing!

Comment by Jan_Kulveit on Neglected EA Regions · 2020-02-18T13:42:00.787Z · EA · GW

FWIW the Why not to rush to translate effective altruism into other languages post was quite influential but is often wrong / misleading / advocating some very strong prior on inaction, in my opinion

Comment by Jan_Kulveit on Neglected EA Regions · 2020-02-17T20:11:18.468Z · EA · GW

I don't think this is actually neglected

  • in my view, bringing effective altruism into new countries/cultures is in initial phases best understood as a strategy/prioritisation research, not as "community building"
    • importance of this increases with increasing distance (cultural / economic / geographical / ...) from places like Oxford or Bay

(more on the topic here)

  • I doubt the people who are plausibly good founders would actually benefit from such groups, and even less from some vague coordination due to facebook groups
    • actually I think on the margin, if there are people who would move forward with the localization efforts if such fb groups exist and other similar people express interest, and would not do that otherwise, their impact could be easily negative
Comment by Jan_Kulveit on AI safety scholarships look worth-funding (if other funding is sane) · 2019-11-26T12:24:57.312Z · EA · GW
  • I don't think it's reasonable to think about FHI DPhil scholarships and even less so RSP as a mainly a funding program. (maybe ~15% of the impact comes from the funding)
  • If I understand the funding landscape correctly, both EA funds and LTFF are potentially able to fund single-digit number of PhDs. Actually has someone approached these funders with a request like "I want to work on safety with Marcus Hutter, and the only thing preventing me is funding"? Maybe I'm too optimistic, but I would expect such requests to have decent chance of success.
Comment by Jan_Kulveit on I'm Buck Shlegeris, I do research and outreach at MIRI, AMA · 2019-11-24T14:30:50.107Z · EA · GW



For example, CAIS and something like "classical superintelligence in a box picture" disagree a lot on the surface level. However, if you look deeper, you will find many similar problems. Simple to explain example: problem of manipulating the operator - which has (in my view) some "hard core" involving both math and philosophy, where you want the AI to somehow communicate with humans in a way which at the same time allows a) the human to learn from the AI if the AI knows something about the world b) the operator's values are not "overwritten" by the AI c) you don't want to prohibit moral progress. In CAIS language this is connected to so called manipulative services.

Or: one of the biggest hits of past year is the mesa-optimisation paper. However, if you are familiar with prior work, you will notice many of the proposed solutions with mesa-optimisers are similar/same solutions as previously proposed for so called 'daemons' or 'misaligned subagents'. This is because the problems partially overlap (the mesa-optimisation framing is more clear and makes a stronger case for "this is what to expect by default"). Also while, for example, on the surface level there is a lot of disagreement between e.g. MIRI researchers, Paul Christiano and Eric Drexler, you will find a "distillation" proposal targeted at the above described problem in Eric's work from 2015, many connected ideas in Paul's work on distillation, and while find it harder to understand Eliezer I think his work also reflects understanding of the problem.


For example: You can ask whether the space of intelligent systems is fundamentally continuous, or not. (I call it "the continuity assumption"). This is connected to many agendas - if the space is fundamentally discontinuous this would cause serious problems to some forms of IDA, debate, interpretability & more.

(An example of discontinuity would be existence of problems which are impossible to meaningfully factorize; there are many more ways how the space could be discontinuous)

There are powerful intuitions going both ways on this.

Comment by Jan_Kulveit on I'm Buck Shlegeris, I do research and outreach at MIRI, AMA · 2019-11-21T12:38:40.575Z · EA · GW

I think the picture is somewhat correct, and we surprisingly should not be too concerned about the dynamic.

My model for this is:

1) there are some hard and somewhat nebulous problems "in the world"

2) people try to formalize them using various intuitions/framings/kinds of math; also using some "very deep priors"

3) the resulting agendas look at the surface level extremely different, and create the impression you have

but actually

4) if you understand multiple agendas deep enough, you get a sense

  • how they are sometimes "reflecting" the same underlying problem
  • if they are based on some "deep priors", how deep it is, and how hard to argue it can be
  • how much they are based on "tastes" and "intuitions" ~ one model how to think about it is people having boxes comparable to policy net in AlphaZero: a mental black-box which spits useful predictions, but is not interpretable in language

Overall, given our current state of knowledge, I think running these multiple efforts in parallel is a better approach with higher chance of success that an idea that we should invest a lot in resolving disagreements/prioritizing, and everyone should work on the "best agenda".

This seems to go against some core EA heuristic ("compare the options, take the best") but actually is more in line with what rational allocation of resources in the face of uncertainty.

Comment by Jan_Kulveit on Update on CEA's EA Grants Program · 2019-11-16T16:35:38.785Z · EA · GW

Re: future of the program & ecosystem influences.

What bad things will happen if the program is just closed

  • for the area overlapping with something "community building-is", CBG will become the sole source of funding, as meta-fund does not fund that. I think at least historically CBG had some problematic influence on global development of effective altruism not because of the direct impact of funding, but because of putting money behind some specific set of advice/evaluation criteria. (To clarify what I mean: I would expect the space would be healthier if exactly the same funding decisions were made, but less specific advice what people should do was associated; the problem is also not necessarily on the program side, but can be thought about as goodharting on the side of grant applicants/grant recipients.)
  • for x-risk, LTFF can become too powerful source of funding for new/small projects. In practice while there are positive impacts of transparency, I would expect some problematic impacts of mainly Oli opinions and advice being associated with a lot of funding. (To clarify: I'm not worried about funding decisions, but about indirect effects of the type "we are paying you so you better listen to us", and people intentionally or unintentionally goodharting on views expressed as grant justification)
  • for various things falling in between the gaps of fund scope, it may be less clear what to do
  • it increases the risks of trying to found something like "EA startups"
  • it can make the case for individual donors funding things stronger

All of that could be somewhat mitigated if rest of the funding ecosystem adapts; e.g. by creating more funds with intentional overlap, or creating others stream of funding going e.g. along geographical structures.

Comment by Jan_Kulveit on Which Community Building Projects Get Funded? · 2019-11-16T15:48:46.887Z · EA · GW

As a side-note: In case of the Bay area, I'd expect some funding-displacement effects. BERI grant-making is strongly correlated with geography and historically BERI funded some things which could be classified as community building. LTFF is also somewhat Bay-centric, and also there seem to be some LTFF grants which could be hypothetically funded by several orgs. Also some things were likely funded informally by local philantrophists.

To make the model more realistic one should note

  • there is some underlying distribution of "worthy things to fund"
  • some of the good projects could be likely funded from multiple sources; all other things being equal, I would expect the funding to come more likely from the nearest source

Comment by Jan_Kulveit on EA Hotel Fundraiser 6: Concrete outputs after 17 months · 2019-11-05T12:26:57.608Z · EA · GW

meta: I considered commenting, but instead I'm just flagging that I find it somewhat hard to have an open discussion about the EA hotel on the EA forum in the fundraising context. The feeling part is

  • there is a lot of emotional investment in EA hotel,
  • it seems if the hotel runs out of runway, for some people it could mean basically loosing their home.

Overall my impression is posting critical comments would be somewhat antisocial, posting just positives or endorsements is against good epistemics, so the personally safest thing to do for many is not to say anything.

At the same time it is blatantly obvious there must be some scepticism about both the project and the outputs: the situation when the hotel seems to be almost out of runway repeats. While eg EA funds collect donations basically in millions $ per year, EA hotel struggles to collect low tens of $.

I think this equilibrium where

  • people are mostly silent but also mostly not supporting the hotel, at least financially
  • the the financial situation of the project is somewhat dire
  • talks with EA Grants and the EA Long Term Future Fund are in progress but the funders are not funding the project yet

is not good for anyone, and has some bad effects for the broader community. I'd be interested in ideas how to move out of this state.

Comment by Jan_Kulveit on Only a few people decide about funding for community builders world-wide · 2019-10-25T13:52:25.269Z · EA · GW

In practice, it's almost never the inly option - e.g. CZEA was able to find some private funding even before CBG existed; several other groups were at least partially professional before CBG. In general it's more like it's better if national-level groups are funded from EA

Comment by Jan_Kulveit on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-10T19:54:54.670Z · EA · GW

The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little. For more understanding why, check this post of Wei Dai and the discussion under it.

(Also: if you follow just LW, your understanding of the field of AI safety is likely somewhat distorted)

With hypothesis 4.&5. I expect at least Oli to have strong bias of being more enthusiastic in funding people who like to interact with LW (all other research qualities being equal), so I'm pretty sure it's not the case

2.&3. is somewhat true at least on average: if we operationalize "private people" as "people who do you meet participating in private research retreats or visiting places like MIRI or FHI", and "online people" as "people posting and commenting on AI safety on LW" than the first group is on average better.

1. is likely true in the sense that best LW contributors are not applying for grants

Comment by Jan_Kulveit on Long-Term Future Fund: August 2019 grant recommendations · 2019-10-08T14:33:14.431Z · EA · GW

In my experience teaching rationality is more tricky than the reference class education, and is an area which is kind of hard to communicate to non-specialists. One of the main reasons seems to be many people have somewhat illusory idea how much they understand the problem.

Comment by Jan_Kulveit on Get-Out-Of-Hell-Free Necklace · 2019-07-15T07:45:25.653Z · EA · GW

I've suggested something similar for happiness ( ). If you don't want to introduce the weird asymmetry where negative counts and positive not, what you get out of that could be somewhat surprising - it possibly recovers more "common folk" altruism where helping people who are already quite well off could be good, and if you allow more speculative views on the space on mind-states, you are at risk of recovering something closely resembling some sort of "buddhist utilitarian calculus".

Comment by Jan_Kulveit on EA Forum 2.0 Initial Announcement · 2019-07-12T22:26:45.635Z · EA · GW

As humans, we are quite sensitive to signs of social approval and disapproval, and we have some 'elephant in the brain' motivation to seek social approval. This can sometimes mess up with epistemics.

The karma represents something like sentiment of people voting on a particular comment, weighted in a particular way. For me, this often did not seemed to be a signal adding any new information - when following the forum closely, usually I would have been able to predict what will get downvoted or upvoted.

What seemed problematic to me was 1. a number of times when I felt hesitation to write something because part of my S1 predicted it will get downvoted. Also I did not wanted to be primed by karma when reading other's comments.

On a community level, overall I think the quality of the karma signal is roughly comparable to facebook likes. If people are making important decisions, evaluating projects, assigning prices... based on it, it seems plausible it's actively harmful.