What are EA's biggest legible achievements in x-risk? 2022-06-14T18:37:10.123Z
Mentorship request on near-term AI and bureaucracies 2022-05-18T19:14:56.808Z
EA frontpages should compare/contrast with other movements 2022-05-01T17:43:32.226Z
Samo Burja on Effective Altruism 2022-02-17T19:20:33.856Z
Institution design for exponential technology 2022-01-02T10:52:00.034Z
Have you considered switching countries to save money? 2021-12-30T12:33:24.151Z
Is EA compatible with technopessimism? 2021-12-25T11:16:47.391Z
Measuring the "apocalyptic residual" 2021-12-18T09:06:11.811Z
The case for studying stylometric deanonymisation as surveillance tech 2021-12-08T12:57:33.680Z
Profit maximisation and obligations on shareholders 2021-12-01T11:32:25.408Z
EA orgs should integrate with The Giving Block for cryptocurrency donations 2021-11-25T13:54:14.875Z
Do any EA orgs accept crypto donations (unbounded) from anonymous parties or smart contracts? 2021-11-22T21:57:51.722Z
Scihub backups for open research 2021-11-20T17:18:39.583Z
Bounty to disclose new x-risks 2021-11-05T12:53:17.043Z
Effects of being able to see effects of donation first-hand 2021-10-29T15:41:01.033Z
Samuel Shadrach's Shortform 2021-10-19T07:45:14.758Z
Mapping out cyberwarfare landscape? 2021-10-16T16:52:00.119Z
EA opportunity: Cryptocurrency and DeFi 2021-10-07T15:58:55.879Z
Mass social media as a tool for change 2021-10-03T14:57:45.362Z


Comment by acylhalide (Samuel Shadrach) on Against longtermism · 2022-08-11T13:05:00.377Z · EA · GW

Yup, no idea what a longtermist in 1960 would've done. I would be keen on reading the report though. I'm also interested in future-looking policy - for instance if nuclear power plants become prevalent in a much larger set of countries than they are today. If they are indeed net positive it does seem useful to establish consensus that that is so! Right now the topic has become politicised but at the same time I feel there isn't good enough technical material for someone to read and go "yes this is obviously net good".

Comment by acylhalide (Samuel Shadrach) on Announcing the Introduction to ML Safety Course · 2022-08-11T07:12:03.055Z · EA · GW

Will strongly endorse this resource, as someone currently doing this course!

Course deals primarily with empirical research directions in safety on modern ML systems. You'll find less theoretical work and less work on non-ML systems, for better or for worse. One advantage of having primarily empirical work is that it is more concrete and there are many research directions that are both tractable and useful to pursue. This material will be better for people who are already ML researchers, including those who are skeptical on the case for x-risk or those who are skeptical we can pursue any research at all to reduce x-risk.

Course material attempts to integrate a lot of different research directions into a coherent framework. It also integrates work on both near-term and long-term risks into a single framework. The course borrows from concepts of risk analysis in general, which I liked.

Comment by acylhalide (Samuel Shadrach) on Against longtermism · 2022-08-11T07:08:23.767Z · EA · GW

re: 1960 AD and nuclear power plants

It is still not obvious in 2022 whether abundant nuclear power plants are net good - namely whether the extra energy we obtain is worth weapons proliferation and other risks. Or atleast it isn't obvious to me, and EA doesn't seem to have consensus on this either (yet).

Just wished to point this out - if you're looking for less controversial examples.

Comment by acylhalide (Samuel Shadrach) on On the Vulnerable World Hypothesis · 2022-08-02T04:29:54.166Z · EA · GW

I'm not sure how much anxiety matters.

Everyone is being surveillance by Google today, all your files, emails, geo locations, your whole daily routine can already be mapped unless you're taking special efforts to prevent this. If there is a significant effect on anxiety it should be measurable as of today.

Comment by acylhalide (Samuel Shadrach) on Interesting vs. Important Work - A Place EA is Prioritizing Poorly · 2022-07-29T12:46:02.136Z · EA · GW

Thanks, this is a fair opinion.

Comment by acylhalide (Samuel Shadrach) on Interesting vs. Important Work - A Place EA is Prioritizing Poorly · 2022-07-28T13:11:05.853Z · EA · GW

For completeness, a third strategy is selecting for people who are (somewhat) resilient to status and monetary incentives, and their internal motivations are less elastic* and they're more willing to build their own inside views and act on them.

If you want to promote this third strategy you may also want to reduce the status and monetary gradients currently present across roles. Academia tries to do this with monetary incentive gradients (relative to corporate), but they fail when it comes to status gradients.

I don't know which strategy works better where, and iM still trying trying figure this out as well, but I felt it was worth mentioning as a strategy.

*I wonder if it's also useful to look at existing communities based on how elastic their members' values and worldviews are, and see what works.

Comment by acylhalide (Samuel Shadrach) on Why EAs are skeptical about AI Safety · 2022-07-28T06:53:01.699Z · EA · GW

Yup agreed this is an open question.

Some stuff I've heard so far:

 - Homo sapiens went from ape-level cognition to massively capable, inventing language and all the vast capabilities it entails, in a very short span of time. See this list of features unique to human language. The fact that we invented all this capability in a short span of time means there weren't a large number of "special sauces" that had to each be evolutionarily selected for, it's one core change that captures most of our capability.

 - One way of defining "general" is just applying one capability obtained from one task, to a different task. Like if you are good at math and symbolically manipulating symbols, and you use this to design a reactor which is an entirely different task from doing math well. Animals don't seem to show much of such transfer. ML systems however are already showing capacity for transfer learning.

 - Lots of tasks require generality in the sense that learning general capabilities is by far the easiest way to git gud at those tasks. Brute forcing your way through 10^24 ways how not to build a reactor is an impractical way to learn how to build a reactor, whereas if you have a core that allows you to generalise stuff you already know, such as math, this works. It's not enough to know math, you also need to know how to generalise your math capability to designing a reactor.

 - Rob Bensinger has an analogy for this previous point where he says that it is weird to teach someone to park their car in the first parking spot or the third parking spot, such that they completely lack the capability to park in any other spot. Successfully teaching for such broad tasks automatically means teaching the agent how to also do all sorts of other tasks.

Comment by acylhalide (Samuel Shadrach) on EA's Culture and Thinking are Severely Limiting its Impact · 2022-07-28T06:26:24.644Z · EA · GW

Strong upvote for asking - what are the tangible actions one can take, and who do you want should take them. This answer is often missing in EA critiques.

Comment by acylhalide (Samuel Shadrach) on GLO, a UBI-generating stablecoin that donates all yields to GiveDirectly · 2022-07-23T08:17:34.867Z · EA · GW

Thank you this makes sense!

Comment by acylhalide (Samuel Shadrach) on GLO, a UBI-generating stablecoin that donates all yields to GiveDirectly · 2022-07-22T16:19:30.529Z · EA · GW

Keen to know if you have worked out the legal risks of offering such a product!

If you have, I also wonder if you'll be able to offer higher risk products to users, such as bonds of lower rating. As a stablecoin protocol, you face significant competition, whereas as a bond provider you may be able to offer a new product.

(This last part is just a random thought I had, I'm aware there's a lot of pros/cons to it)

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-19T10:32:41.889Z · EA · GW

Yup this is fair! I just realised this once I tried formalising.

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-19T10:12:55.784Z · EA · GW

I'll add a math formalisation if it helps!

I've made a major assumption in between though, about cost-effectiveness being strictly decreasing.


Assume you need to donate amount C across i opportunities, x_i is amount for i-th opportunity. Assume you need to donate now, and aren't considering saving for investment purposes or saving for future opportunities..

sum x_i = C

For i-th opportunity, let ΔU_i(x) map amount donated to utility gained. U_i could map scalar to scalar or scalar to a probability distribution over scalars. The delta sign indicates utility of the world after donation minus utility of the world if you burned the money instead. If you assume the second term is zero in your reference frame, then you can omit Δ sign for neatness.

We wish to maximise:

ΔU = sum ΔU_i(x_i)

U = sum U_i(x_i)


Assume there are no gained resources (monetary or otherwise) or information in each grant, besides the immediately obvious 1st order effect of the donation captured by the term U_i(x_i).

Assume U_i(x) is strictly increasing but with strictly decreasing slope.

U_i ' (x) > 0 for all positive x          (more money donated is always good)

U_i '' (x) < 0 for all positive x         (more money donated reduces cost-effectiveness)

Note this is a major assumption that does not always apply.

For instance, say malaria nets cost $2 each and some constant number saves a life. Then U_i(x) appears to be a linear function, which satisfies this. But in practice, donating 2 cents more does not lead to 1/100th of a malaria net. So U_i(x) is actually a function that looks like a staircase making a tiny step every $2. This does not satisfy above conditions. So that condition does not apply. However, we can approximate it as linear as $2 is very small.

We cannot approximate a step function or staircase as linear if it only generates utility in response to very large costs.


Now we can approximate U_i(x)  as linear in a small region around x=x_i

U_i(x) = y_i * x_i

for some constant y_i

y_i is the cost-effectiveness, it is also the slope of the function U_i(x)

We may have a probability distribution over y_i, instead of a known constant y_i.


We can now use a greedy algo (*) to take decisions. Assume we split our large amount C, into tiny amounts c to donate one at a time.

For each c, we have:

Maximise sum y_i * x_i subject to sum x_i = c

The solution is to just allocate x_i = c for whichever y_i is largest, and all other x_i = 0

Once you've done this you can calculate the new cost-effectiveness values and then repeat the process. You'll have to run many iterations of the algo until you donate the whole amount C.

If y_i is indeed a probability distribution, then we have

Maximise E(sum y_i * x_i) = sum x_i * E(y_i) subject to sum x_i = c

Here we can indeed take the expected cost-effectiveness of each opportunity.


(*) We can do this only if all U_i are strictly increasing with strictly decreasing slope I think.

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-19T09:19:37.785Z · EA · GW

Will check!

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-19T09:04:57.803Z · EA · GW

re: last 2 questions, I'm not immediately sure, sorry!

Having better educational content on the nature of the problem we're optimising would be useful.

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-19T09:02:36.670Z · EA · GW

Just checked knapsack problem and yup this seems super useful. 

re: convexity, maybe I am making the wrong assumption that all opportunities will have diminishing cost-effectiveness as more money pours into them. It is possible there are some opportunities that become more cost-effective* once some initial money has been poured in and groundwork has been done.

*Ideally you can factor in all those new opportunities opening up into the "effect" term of the original grant, then you again get diminishing cost-effectiveness I think. But I don't know if that can always be done in practice due to uncertainty.

Comment by acylhalide (Samuel Shadrach) on Probability distributions of Cost-Effectiveness can be misleading · 2022-07-18T18:14:57.598Z · EA · GW

Just curious: Has anyone framed EA giving as a convex optimisation problem yet*?

Theres a number of posts like this on what feel like fairly obvious thing to me, so I wonder if there should be an educational series around it.

*(atleast ignoring knightian uncertainty)

Comment by acylhalide (Samuel Shadrach) on EA opportunity: Cryptocurrency and DeFi · 2022-07-16T05:16:17.746Z · EA · GW

This seems like a fair opinion to have if you look at crypto in isolation, many people are divided, I am as well. If you wish to defend it perhaps you could write a separate post.

From an earning to give perspective IMO still be locally* positive to work in the crypto space - SBF has done more good via EA than harm via crypto CO2 emissions.

I'm personally not sure if messing up the economy is a worthy consideration, I'd be keen to know your thoughts on that. I currently assign a very low probability that BTC market cap will become 2x of gold's market cap for instance. (But sure, even a low probability could in expectation be dangerous.)

Comment by acylhalide (Samuel Shadrach) on Person-affecting intuitions can often be money pumped · 2022-07-16T05:10:48.695Z · EA · GW

Thanks for this comment!

Comment by acylhalide (Samuel Shadrach) on Person-affecting intuitions can often be money pumped · 2022-07-15T14:43:35.061Z · EA · GW

Thanks for the reply!

Sorry, I really tried writing you a reply, even deleted a few I wrote, but I think I should probably spend some time on it myself first so I can present it better.

If I've to really shorten, in general I feel like we don't have that much "free choice" even in simulations, we're anchored to the observations we've actually had, and our creativity is very limited. And different futures can provide us with wildly different observations all unimaginable to people in other futures, and people in 2022. But defending this and other things will require lot more effort on my part. 


Sorry. Thanks for your time anyways!

Comment by acylhalide (Samuel Shadrach) on Punching Utilitarians in the Face · 2022-07-15T03:13:03.266Z · EA · GW

IMO utilitarians (or more generally believers of any ethical theory) should flesh out what it means for a theory to be true some of the time but not at all of the time, essentially what are their meta-ethics. Some people's meta-ethics doesn't lend itself well to accepting a theory that is true some of the time but not all of the time, because they don't want to build stuff purely on intuitions.

Accepting such a theory also means the theory is now dependent on the probability distribution of scenarios you see in a world (what if you lived in a world where pascal mugging were very common), and hence less objective in a sense. Which is fine if you accept it.

Comment by acylhalide (Samuel Shadrach) on Person-affecting intuitions can often be money pumped · 2022-07-14T18:34:38.667Z · EA · GW

This is fair!

So to restate, your claim is that in the absence of such adversaries, moral reasoning processes will in fact all converge to the same place. Even if we're exposed to wildly different experiences/observations/futures, the only thing that determines whether there's convergence or divergence is whether those experiences contain intelligent adversaries or not.

I have some intuitions against this claim too, but I'm not sure how to make my thoughts airtight or present them well. I'll still try!

(If you think anything here is valuable or that I should spend more time trying to present anything here better, do tell! My comment might be a bit rambly right now, sorry.)


Question 1: What precisely about our moral reasoning process make them unlikely to be attacked by "natural" conditions but attackable by an intelligently designed one? If I had to mathematically formally written down every possible future, what makes this distinction between natural and not natural a sharp distinction, and what makes it perfectly 100% set-theoretically overlap with the distinction between futures where our reasoning processes converge and where they don't?


One way to answer this is to point at some deep structures we can see today, in people's moral reasoning processes. If you have any I'd be keen to see them.

Another is to rely on intuitions we have today and trust that in the future we can formalise it. I find this plausible, but if you're claiming this I also don't know how to attach a lot of confidence to just intuitions. Maybe I need to see your intuitions!

Yet another is to claim that sure, in theory there might be "natural" conditions that can attack our reasoning processes, it's just that those natural conditions are super unlikely in practice. This will then shift the question more from theoretical to practical, from theoretically possible futures to futures actually likely in practice.


As a practical matter, I don't know how we can say with high confidence anything about what our natural conditions will look like millions of years from now.

As another practical matter, our world today is full of intelligent adversaries trying to hack each other's moral reasoning processes. Sure, the intelligence differential isn't as big as could maybe be with an AI or digital minds, but there are both intelligence and power differentials.


Speaking of power differentials specifically, Most people are socially punished for various thoughts to the point those thoughts can't be expressed in public; enough such punishment can cause people to end up lying even to themselves. For instance, see hegemonic culture.

Maybe the future will not contain such adversaries or they won't have similar intelligence or power differentials, but it seems non-obvious at first glance.


Question 2: Could natural conditions ever play the equivalent of intelligent adversaries? For instance imagine we had Robin Hanson-style Malthusian competition playing out among digital minds for millions of years. (I'm not sure this is actually likely but imagine it happened.) Assume all the agents competing have very similar but not perfectly identical values and reasoning processes. Feel free to interpret "similar but not identical" however you wish, I'm just trying to somehow capture the intuition that human values and moral reasoning processes atleast at surface-level seem similar but not identical, there are some structures that are identical.

Now if you did run Malthusian competition for millions of years (which seems a dumb algorithm and a natural algorithm), are you confident we wouldn't end up with agents whose values converged somewhere else? (Versus worlds not run by competition)


Actually on second thought, maybe in this example I am conflating "dumb natural algorithms/conditions selecting agents whose values converge differently" with "dumb natural algorithms/conditions selecting observations that can attack the moral reasoning process of the very same agent". Maybe I'll try thinking of an example specifically for the latter.

But also, shouldn't your convergence claim also resist selection processes? For instance if there's imperfect communication between Americans and Chinese, would they both still converge to the same values? If the Chinese died but Americans lived, or if the Americans lived but Chinese died, will that affect where their moral reasoning processes converge long-term?


There is a trivial way in which dumb natural algorithms can find observations and experiences to attack you, which is by selecting for intelligent adversaries who can create those observations.

For instance on the stuff about Malthusian competition and hegemonic culture, competition itself is a dumb natural state not an intelligent adversary. But it can select for intelligent adversaries who will then be able to create experiences to attack your reasoning process. So if a capitalist is able to create a bunch of observations to convince you into thinking that working yourself harder is a moral good ... you could say that this is because of the capitalist being an intelligent adversary to your moral reasoning process. But you could also say it's because a really dumb natural algorithm of capitalistic competition selected this specific capitalist to be in a position to attack your reasoning process in the first place.


But if you are looking specifically for ways in which natural conditions can convince people of things without creating intelligent adversaries at all, ... my intuitions are weaker here. I can still try writing about this if you feel it'll be useful, because even here I feel different futures may get us to converge to different values.

Comment by acylhalide (Samuel Shadrach) on Person-affecting intuitions can often be money pumped · 2022-07-13T16:37:40.338Z · EA · GW

I mean something like: If I'm exposed to persuasive sequence of words A, I'll become strongly convinced of one set of values, and if I'm exposed to a different persuasive sequence of words B, I'd become strongly convinced of a different set of values.

Instead of words, it could also be observations or experiences. Which I'm assuming is part of "learning and thinking" as intended here. (And with digital wireheading  for instance, we might be able to generate a lot of such experiences to subject each other to.)

It isn't obvious to me that different sets of experiences in different futures will still cause us to converge to the same "future-proof" values. 


And maybe that's cause humans are faulty reasoners, and ideal reasoners can in fact do better. But if I had to self-modify into an ideal reasoner I'm not sure what a "moral reasoning process" even looks like. Our best (or atleast good) formal model for an ideal reasoner is one that just happens to know its unchangeable utility function from birth, not one that reasons about what values it should have, in any meaningful sense.

And if you cannot formalise what these moral reasoning processes could look like, even in theory, I also find it easier to believe any such process can be attacked. (Probably such security mindset.)


Keen on your thoughts!

Comment by acylhalide (Samuel Shadrach) on Person-affecting intuitions can often be money pumped · 2022-07-10T15:45:42.084Z · EA · GW

If I may ask, why do you believe there exist any future-proof ethics?

I kinda suspect no ethics are future-proof in this sense, hence had to ask.

Comment by acylhalide (Samuel Shadrach) on Why AGI Timeline Research/Discourse Might Be Overrated · 2022-07-08T05:39:21.123Z · EA · GW

I don't agree that it's easier to rally people around a narrow cause; on the contrary, single minded focus on AI would drive away all but a small fraction of potential supporters, and have an evaporative cooling effect on the current community too.

I see. This is probably somewhere we might disagree then. But if you don't have time it is okay, this whole scenario anyway seems hypothetical so may not be super high value to discuss.

Comment by acylhalide (Samuel Shadrach) on Why AGI Timeline Research/Discourse Might Be Overrated · 2022-07-04T05:57:38.669Z · EA · GW

Thanks for replying!

Rest of my comment is conditional on EA folks having consensus for >50% by 2040. I personally don't actually believe this yet, but let's imagine a world where you and I did. This discussion would still be useful if we are one day year X and we knew >50% odds by year X + 20.

And there are other causes where we can have a positive effect without directly competing for resources

I do feel it competes for resources though. Attention in general is a finite resource. 

 - People learning of EA have a finite attention span and can be exposed to only so many ideas. If AI risk is by far the most important idea to convey, then all (or most) content should focus on AI risk.

 - All EA orgs and senior members have finite time and attention to spare. An org can maybe get distracted if it has multiple things to look at, and if its senior members need to keep themselves updated on multiple topics if they are to lead the org effectively. 

 - It is easier to rally people around a narrower cause - be it inside EA orgs or the general public looking to support us. Cause neutrality doesn't come naturally - it is easier to get people to accept the importance of one single cause.

because not all participants and funders are willing or able to work on AI.

I feel like community building is something more people can do, even if it's not their ideal role in terms of personal fit. Especially if we expand notion of community building beyond just AI researchers and make it a mass movement. Even without funding, a lot of people could continue in their current careers while they spread the movement. Or they could go on strike etc, like mass movements sometimes do.

But yes you're right some people can't do that, and there may be other forms of good they can do in those 20 years. Maybe I'm just biased because I'm fairly longtermist myself, and saving a few more lives in those 20 years feels not very important compared to something that could potentially be the most important thing mankind has ever done. But I'm aware not everyone shares my intuitions.

Working on other issues gives us better engagement with policy makers and the public, improves the credibility of the movement, and provides more opportunities to get feedback on what does or doesn't work for maximizing impact.

Not sure what works with policymakers!

Agree that testing multiple approaches to community building among the public is useful, I'm not sure why you need multiple cause areas for that.

Becoming insular or obsessive about AI would be alienating to many potential allies and make it harder to support good epistemic norms. 

I'm not sure on the good epistemic norms thing. Like yes people will want to be strategic rather than honest on some matters. But the basic fact that "yes we care about AI a loooot more than other things because it's happening so soon" is something we'll be more honest about, if we chose to focus on only that.

I agree it will alienate people, but I'm also not sure how much that matters. Where what "matters" too again gets measured primarily in terms of impact on AI risk.


Keen on your thoughts, if you have time. 

Comment by acylhalide (Samuel Shadrach) on Why AGI Timeline Research/Discourse Might Be Overrated · 2022-07-03T16:12:45.972Z · EA · GW

Single data point but I think my life trajectory could look different if I believed >50% odds of AGI by 2040, versus >50% odds by 2100. And I can't be the only one who feels this way.

On the 2100 timeline I can imagine trusting other people to do a lot of the necessary groundwork to build the field globally till it reaches peak size. On the 2040 timeline every single month matters and it's not obvious the field will grow to peak size at all. So I'd feel much more compelled to do field building myself. And so would others even if they have poor personal fit for it. On the 2040 timeline it could make sense* for EA to essentially repurpose the whole movement solely to AI risk, as funding, thinking or even talking about anything else (be it other GCRs or global health etc) would be a distraction from what's most important.

*If someone disagrees with me please let me know. This comment was a bit speculative.

P.s. Im only critiquing the "timelines don't matter to people's decisions" point, not the "we shouldn't invest more into improving timelines" one.

Comment by acylhalide (Samuel Shadrach) on Things that limit my ambition in EA · 2022-06-29T10:49:31.018Z · EA · GW

These are real factors and major reasons people are not ambitious.

Some level of financial stability is prudent. Beyond that you just have to  care about the goal more than securing complete stability. Lots of people don't stay at early-stage startups because they can get good enough pay at later stage ones.

Burnout is real and there are tips you will find (on this site as well as elsewhere). Many people don't reduce this odds of burnout to zero (I don't know if they even can), but you can reduce it by some amount.

Imposter syndrome and aversion to rejection are things that you need to IMO tackle early on. It is possible to have near-zero amounts of these traits. How to get there (if you aren't already there) is a different question, again you will find some tips here on this site and elsewhere.

And I would emphasise on the fear of rejection in particular, as most ambitious tasks require networking with other people a lot. As a founder your primary role is being able to attract other people and selling them into doing things you want done. This typically involves a lot of rejection.

Comment by acylhalide (Samuel Shadrach) on Will faster economic growth make us happier? The relevance of the Easterlin Paradox to Progress Studies · 2022-06-27T06:46:23.217Z · EA · GW

Appreciate your attempts to steer rather than mutiny! Hope they work out well for you.


Comment by acylhalide (Samuel Shadrach) on On Deference and Yudkowsky's AI Risk Estimates · 2022-06-23T07:06:24.705Z · EA · GW

Not sure why this is on EAF rather than LW or maybe AF, but anyway

One obvious answer is LW community and mods tend to defer to yudkowksy more than EAF connunity.

(This doesn't argue whether the deferrence is good or bad, but this difference is a fact about reality I think)

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-20T03:55:47.927Z · EA · GW

I understand, thanks again

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-19T18:42:02.404Z · EA · GW

Thanks for replying. I'm not sure this satisfies the criteria of "legible" as I was imagining it, since I buy most of the AI risk arguments and still feel poorly equipped to evaluate how important this was. But I do not have sufficient ML knowledge; perhaps it was legible to people with sufficient knowledge.

P.S. If it doesn't take too much of your time I would love to know if there's any discussion on why this was significant for x-risk, say, on the alignment forum. I found the paper and OpenAI blogpost but couldn't find discussion. (If it will take time I totally understand, I will try finding it myself.)

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-19T18:22:37.457Z · EA · GW

Thanks for the reply. This makes a lot of sense! I will look into Bridgewater.

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-19T16:43:23.882Z · EA · GW

Oh okay I see. Thank you for responding!

You have shifted my opinion a bit, it's no longer obvious to me where exactly the line between private and public information should be drawn here.

I can see why releasing information about personal incompetence for instance might be unusual in some cultures; I'm not sure why you can't build a culture where releasing such information is accepted.

You're right OpenPhil not being a grant recommendation service changes things a bit.

I think I would still be keen on seeing more research (or posts or comments etc) that tries to figure out what the optimal level of openness is. (Not just OP but all large EA orgs in general.) One of the core reasons EA is as successful as it is, is its commitment to higher epistemic standards. I tentatively feel you need a lot of openness for good epistemics, and we should start with a prior that more openness is good, and try figuring out what are the real things it trades off against.

I'd be keen if there exists such research or if you think it would be useful if it existed!

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-19T07:22:46.657Z · EA · GW

However, it would likely be very awkward and unprofessional to actually release this information publicly.

IMO this is a norm that should urgently change. AFAIK 80k hours and CEA have admitted mistakes and changes in cause prioritisation before, including the whole global health to longtermist shift.

Willingness to admit mistakes is table stakes for good epistemics.

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-17T18:37:28.786Z · EA · GW

Interesting link, thanks for sharing! I can think of counterpoints to that assumption too but won't discuss here as its offtopic. (I mean I agree it is a real effect, its strength can vary I think.)

Comment by acylhalide (Samuel Shadrach) on Are too many young, highly-engaged longtermist EAs doing movement-building? · 2022-06-17T18:20:19.862Z · EA · GW

As an undergrad who has considered doing movement-building despite probably having more fit for direct research, you're basically right.

The biggest reason for not doing direct research though is shortage of mentorship. Expecting someone to write their own research agendas, apply for funding and work solo straight after undergrad is a tall order. There is no clear feedback loop and there is less clearly visible scope for upskilling, which as someone junior you intrinsically tend to consider valuable. It is easier to take the lower risk path which is apply for PhD even if only in EA-adjacent or non-EA areas and universities.

I'm sure people  who take the latter route could eventually go on to do useful EA work, but maybe once they reach that stage they don't fit your "young" bracket anymore. (This is a guess on my part, I could easily be wrong.) Which means when you look at young people actively signalling their EA-ness you may not be seeing these people. 

More reasons to do movement building are building career capital inside of the EA space to increase odds of being hired by an EA org later, and networking with EA researchers so you can learn from them, so they agree to mentor you etc.

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-17T15:50:02.123Z · EA · GW

I see. That statement of yours brought to my mind questions like:

 - can we assume (most) criminals are rational actors? is there evidence?

 - how do you compare say, years in prison versus benefits from getting away with a theft in an EV calculation?

 - how do you consider personal circumstances? (a poor thief gets more utility than a rich thief)

 - does the punitive justice system currently do any calculations of this sort?


I don't think any of this discussion is essential to the piece, it's just that that line caught my eye as smuggling in a whole perspective.

Comment by acylhalide (Samuel Shadrach) on What is the right ratio between mentorship and direct work for senior EAs? · 2022-06-17T14:09:03.487Z · EA · GW

Not a complete answer, but some subquestions this question invokes for me:

 - How much difference does lack of mentorship have on junior researchers? Are they likely to quit EA research altogether and do something else, are they likely to take more years to reach same capability level, are they likely to simply fail to reach the same level of capability despite trying?

 - Split above question by more minimal mentorship and deeper mentorship. I have a feeling deeper mentoring relationships can be very valuable (I could be wrong).

 - Mentorship has compounding benefits for EA and this gives it priority to be done sooner rather than later. How compounding or "foundational" is the direct research work of the senior researcher? To make this more concrete, how much is lost if the senior researcher does their own research 10 years later (because they want to do mentoring first)?

If someone postpones work on directly reducing nuclear risk, the loss is 10 years of humanity's exposure to the nuclear risk delta caused by that research piece. Whereas if the senior researcher is working to create valuable new research agendas that can eventually absorb a large number of researchers, and they don't believe anyone else could create this agenda at around the same time, it seems important that this gets done sooner rather than later. Civilisational refuges seems like an example of such a work, so does some (not all) alignment research.

Comment by acylhalide (Samuel Shadrach) on Steering AI to care for animals, and soon · 2022-06-17T13:43:21.804Z · EA · GW

If digital minds have moral status comparable to biological ones, will this matter as much?

a) Digital minds are more matter- and energy-efficient, so we can have much more of them than biological minds.

b) If we deploy transformative AI that is aligned, we likely can eventually get digital minds if we want. (Atleast under naturalist worldviews)

c) Digital minds optimised for any metric probably do not look identical to either human or animal minds. Examples of metrics include any function of capability, energy-efficiency, hedonic capacity or neural correlates of consciousness (if we find such a thing).

Comment by acylhalide (Samuel Shadrach) on A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform · 2022-06-17T13:19:45.095Z · EA · GW

Criminals should be punished in proportion to an estimate of the harm they have caused, times a factor to account for a less than 100% chance of getting caught, to ensure that crimes are not worth it in expectation.


Where is this perspective coming from? I googled this and it seems to be referring to rational choice theory in criminology. How well supported is this theory?

Comment by acylhalide (Samuel Shadrach) on The totalitarian implications of Effective Altruism · 2022-06-15T18:13:57.258Z · EA · GW

. My concern however is that this is not EA, or at least not EA as embodied by its fundamental principles as explored in my piece. 


And I think your last para is mostly valid too.

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-15T18:11:55.266Z · EA · GW

Thanks this definitely seems helpful!

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-15T18:11:29.297Z · EA · GW

No problem!

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-15T03:41:59.488Z · EA · GW

Thanks for this anecdote!

Given the scarcity of such successes, I think people here would be interested in hearing a longer form version of this. Just wished to suggest!

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-15T03:41:06.931Z · EA · GW

Thanks for your reply! I'll see if I can convince people using this.

(Also very small point but: the pdf title says "Insert your title here" when viewed on chrome atleast)

Comment by acylhalide (Samuel Shadrach) on What are EA's biggest legible achievements in x-risk? · 2022-06-14T21:06:30.765Z · EA · GW

Thanks for your reply!

This makes sense. Linked post at the end was useful and new for me.

Comment by acylhalide (Samuel Shadrach) on Catholic theologians and priests on artificial intelligence · 2022-06-14T20:21:12.006Z · EA · GW

Would recommend adding something attention catching or interesting from the special issue if you can. Will increase the likelihood someone decides its worth their time investment to go through it.

Comment by acylhalide (Samuel Shadrach) on The totalitarian implications of Effective Altruism · 2022-06-14T18:26:54.307Z · EA · GW

Nice article.

One mental move you can make to avoid this totalisation is to frame "doing the most good" not as the terminal goal of your life, but as an instrumental goal to your terminal goal which is "be happy/satisfied/content". Doing good is obviously one of the things that brings you satisfaction (if it didn't you obviously would not do it), but it isn't the only thing.

Accepting this frame risks ending up not doing good at all, cause there's plenty of people who are happy without doing much good for others (atleast as measured by an EA lens). Which may be something you're okay with, or something you wish to defend against. But for some people I think doing good is important enough to their own life satisfaction that I don't think losing the goal of doing good is a concern.

(Personally I think the whole terminal goal / instrumental goal abstraction breaks down a bit when discussing humans, but a lot of EAs are clearly utilitarians who think in those terms so I wished to provide some thoughts in a frame they could parse.)

Comment by acylhalide (Samuel Shadrach) on AGI Ruin: A List of Lethalities · 2022-06-12T16:36:10.383Z · EA · GW

Interesting suggestion, although my first reaction is it feels a bit like handing things over to Moloch. Like, I would rather replace a bad judge of what is infohazardous content with a good judge, than lose our ability to keep any infohazards private at all.

There's also a similar discussion on having centralized versus decentralized grantmaking for longterm future stuff. People pointed out unilateralists curse as a reason to keep it centralised.

I am normally super in favour of more decentralisation and pluralism of thought, but having seen a bunch of infohazardous stuff and potential dangerous-if-executed plans here and there I'm no longer sure.

P.s. Maybe one needs to decouple stuff like "low quality", "not a topic of interest in this particular forum", "not worth funding" etc from "infohazardous" and "dangerous to execute".

Like maybe we can decentralize decision making on former without doing the same on latter.

Comment by acylhalide (Samuel Shadrach) on AGI Ruin: A List of Lethalities · 2022-06-11T14:29:55.071Z · EA · GW

Just as a datapoint:

I had once explicitly posted on LW asking what pivotal act you could take by querying a misaligned oracle AI, with the assumption that you want to leak as little bits of information about the world as possible to the AI. Reasoning being if it had lesser data it would have lesser ability to break out of its box even if it failed to answer your query.

LW promptly downvoted the question heavily so I assumed it's taboo.