Posts

Summary of Stuart Russell's new book, "Human Compatible" 2019-10-19T19:56:52.174Z · score: 28 (12 votes)
Alignment Newsletter One Year Retrospective 2019-04-10T07:00:34.021Z · score: 61 (23 votes)
Thoughts on the "Meta Trap" 2016-12-20T21:36:39.498Z · score: 8 (12 votes)
EA Berkeley Spring 2016 Retrospective 2016-09-11T06:37:02.183Z · score: 6 (6 votes)
EAGxBerkeley 2016 Retrospective 2016-09-11T06:27:16.316Z · score: 10 (6 votes)

Comments

Comment by rohinmshah on I'm Buck Shlegeris, I do research and outreach at MIRI, AMA · 2019-11-20T19:09:25.842Z · score: 19 (9 votes) · EA · GW
I suspect that things like the Alignment Newsletter are causing AI safety researchers to understand and engage with each other's work more; this seems good.

This is the goal, but it's unclear that it's having much of an effect. I feel like I relatively often have conversations with AI safety researchers where I mention something I highlighted in the newsletter, and the other person hasn't heard of it, or has a very superficial / wrong understanding of it (one that I think would be corrected by reading just the summary in the newsletter).

This is very anecdotal; even if there are times when I talk to people and they do know the paper that I'm talking about because of the newsletter, I probably wouldn't notice / learn that fact.

(In contrast, junior researchers are often more informed than I would expect, at least about the landscape, even if not the underlying reasons / arguments.)

Comment by rohinmshah on AGI safety and losing electricity/industry resilience cost-effectiveness · 2019-11-18T00:51:23.359Z · score: 1 (1 votes) · EA · GW

Thanks!

Comment by rohinmshah on AGI safety and losing electricity/industry resilience cost-effectiveness · 2019-11-18T00:49:52.891Z · score: 6 (4 votes) · EA · GW

I mostly meant phrasing it as "the model result", the "99-100%" is fine if it's clear that it's from a model and not your considered belief.

Comment by rohinmshah on AGI safety and losing electricity/industry resilience cost-effectiveness · 2019-11-17T17:37:20.602Z · score: 5 (3 votes) · EA · GW

I would have liked to see the models and graphs (presumably the most important part of the paper), but the images don't load and the links to the models don't work:

Table 1 shows the key input parameters for Model 1 (largely Denkenberger and conference poll of effective altruists)(D. Denkenberger, Cotton-Barrat, Dewey, & Li, 2019a) and Model 2 (D. Denkenberger, Cotton-Barratt, Dewey, & Li, 2019) (Sandberg inputs)(3).

Also:

However, it can be said with 99%-100% confidence that funding interventions for losing industry now is more cost effective than additional funding for AGI safety beyond the expected $3 billion.

If you don't actually mean such confidence (which I assume you don't because 1. it's crazy and 2. you mention model uncertainty elsewhere), can you please not say it?

Comment by rohinmshah on A conversation with Rohin Shah · 2019-11-14T01:11:42.987Z · score: 2 (2 votes) · EA · GW
perhaps babies develop a sense of "hierarchy" which then gets applied to language, explaining how children learn languages so fast.

Though if we are to believe this paper at face value (I haven't evaluated it), babies start learning in the womb. (The paper claims that the biases depend on which language is spoken around the pregnant mother, which suggests that it must be learned, rather than being "built-in".)

Comment by rohinmshah on On AI Weapons · 2019-11-14T01:09:33.801Z · score: 2 (2 votes) · EA · GW

Ah, somehow I missed that, thanks!

Comment by rohinmshah on On AI Weapons · 2019-11-13T20:41:00.352Z · score: 1 (1 votes) · EA · GW

While I'm broadly uncertain about the overall effects of LAWs within the categories you've identified, and it seems plausible that LAWs are more likely to be good given those particular consequences, one major consideration for me against LAWs is that it plausibly would differentially benefit small misaligned groups such as terrorists. This is the main point of the Slaughterbots video. I don't know how big this effect is, especially since I don't know how much terrorism there is or how competent terrorists are; I'm just claiming that it is plausibly big enough to make a ban on LAWs desirable.

Comment by rohinmshah on A conversation with Rohin Shah · 2019-11-13T02:28:05.613Z · score: 9 (4 votes) · EA · GW
(Not sure how much of this Shah already knows.)

Not much, sadly. I don't actually intend to learn about it in the near future, because I don't think timelines are particularly decision-relevant to me (though they are to others, especially funders). Thanks for the links!

Tooby and Cosmides are big advocates for the "massive modularity" view--a huge amount of human cognition takes place in specialized, task-tailored modules rather than on one big, domain-general "computer".

On my view, babies would learn a huge amount about the structure of the world simply by interacting with it (pushing over an object can in principle teach you a lot about objects, causality, intuitive physics, etc), and this leads to general patterns that we later call "inductive biases" for more complex tasks. For example, hierarchy is a very useful way to understand basically any environment we are ever in; perhaps babies develop a sense of "hierarchy" which then gets applied to language, explaining how children learn languages so fast.

From the Wikipedia page you linked, challenges to a "rationality" based view:

1. Evolutionary theories using the idea of numerous domain-specific adaptions have produced testable predictions that have been empirically confirmed; the theory of domain-general rational thought has produced no such predictions or confirmations.

I wish they said what these predictions were. I'm not going to chase down this reference.

2. The rapidity of responses such as jealousy due to infidelity indicates a domain-specific dedicated module rather than a general, deliberate, rational calculation of consequences.

This is a good point; in general emotions are probably not learned, for the most part. I'm not sure what's going on there.

3. Reactions may occur instinctively (consistent with innate knowledge) even if a person has not learned such knowledge.

I agree that reflexes are "built-in" and not learned; reflexes are also pretty different from e.g. language. Obviously not everything our bodies do is "learned", reflexes, breathing, digestion, etc. all fall into the "built-in" category. I don't think this says much about what leads humans to be good at chess, language, plumbing, soccer, gardening, etc, which is what I'm more interested in.

It seems likely to me that you might need the equivalent of reflexes, breathing, digestion, etc. if you want to design a fully autonomous agent that learns without any human support whatsoever, but we will probably instead design an agent that (initially) depends on us to keep the electricity flowing, to fix any wiring issues, to keep up the Internet connection, etc. (In contrast, human parents can't ensure that the child keeps breathing, so you need an automatic, built-in system for that.)

Comment by rohinmshah on Does 80,000 Hours focus too much on AI risk? · 2019-11-03T17:14:45.567Z · score: 49 (16 votes) · EA · GW
Top AI safety researchers are now saying that they expect AI to be safe by default, without further intervention from EA. See here and here.

Two points:

  • "Probably safe by default" doesn't mean "we shouldn't work on it". My estimate of 90% that you quote still leaves a 10% chance of catastrophe, which is worth reducing. (Though the 10% is very non-robust.) It also is my opinion before updating on other people's views.
  • Those posts were published because AI Impacts was looking to have conversations with people who had safe-by-default views, so there's a strong selection bias. If you looked for people with doom-by-default views, you could find them.
Comment by rohinmshah on Publication of Stuart Russell’s new book on AI safety - reviews needed · 2019-10-24T19:40:33.094Z · score: 3 (2 votes) · EA · GW

Amusingly, I use my own Amazon account so infrequently that they refuse to let me write a review. I didn't think about GoodReads, I might do that.

Comment by rohinmshah on Summary of Stuart Russell's new book, "Human Compatible" · 2019-10-21T21:52:57.636Z · score: 7 (2 votes) · EA · GW

I also added a bunch of comments with some other less polished thoughts on the book on the Alignment Forum version of this post.

Comment by rohinmshah on Why do you reject negative utilitarianism? · 2019-10-18T17:10:51.805Z · score: 2 (2 votes) · EA · GW

Yes, that's correct.

Comment by rohinmshah on Why were people skeptical about RAISE? · 2019-09-04T22:03:47.475Z · score: 3 (3 votes) · EA · GW

Mathematical knowledge would be knowing that the Pythagoras theorem states that , mathematical thinking would be the ability to prove that theorem from first principles.

The way I use the phrase, mathematical thinking doesn't only encompass proofs. It would also count as "mathematical reasoning" if you figure out that means are affected by outliers more than medians are, even if you don't write down any formulas, equations, or proofs.

Comment by rohinmshah on Why were people skeptical about RAISE? · 2019-09-04T16:22:53.326Z · score: 15 (6 votes) · EA · GW

Depends what you call the "goal".

If you mean "make it easier for new people to get up to speed", I'm all for that goal. That goal encompasses a significant chunk of the value of the Alignment Newsletter.

If you mean "create courses that allow new people to get the required mathematical maturity", I'm less excited. Such courses already exist, and while mathematical thinking is extremely useful, mathematical knowledge mostly isn't. (Mathematical knowledge is more useful for MIRI-style work, but I'd guess it's still not that useful.)

Comment by rohinmshah on Debrief: "cash prizes for the best arguments against psychedelics" · 2019-07-17T05:13:39.082Z · score: 3 (2 votes) · EA · GW

I don't know of any such stats, but I also don't know much about CFAR.

Comment by rohinmshah on How Europe might matter for AI governance · 2019-07-17T05:12:50.902Z · score: 4 (3 votes) · EA · GW

I was excluding governance papers, because it seems like the relevant question is "will AI development happen in Europe or elsewhere", and governance papers provide ~no evidence for or against that.

Comment by rohinmshah on How Europe might matter for AI governance · 2019-07-15T20:10:38.757Z · score: 23 (9 votes) · EA · GW

My lived experience is that most of the papers I care about (even excluding safety-related papers) come from the US. There are lots of reasons that both of these could be true, but for the sake of improving AGI-related governance, I think my lived experience is a much better measure of the thing we actually care about (which is something like "which region does good AGI-related thinking").

Comment by rohinmshah on Debrief: "cash prizes for the best arguments against psychedelics" · 2019-07-15T17:11:25.099Z · score: 5 (4 votes) · EA · GW
From my current read, psychedelics have a stronger evidence base than rationality training programs

I agree if for CFAR you are looking at the metric of how rational their alumni are. If you instead look at CFAR as a funnel for people working on AI risk, the "evidence base" seems clearer. (Similarly to how we can be quite confident that 80K is having an impact, despite there not being any RCTs of 80K's "intervention".)

Comment by rohinmshah on Charity Vouchers [public policy idea] · 2019-07-11T04:51:05.090Z · score: 2 (2 votes) · EA · GW

Sorry, I'm claiming government is supposed to spend money to achieve outcomes the public wants. (That felt self-evident to me, but maybe you disagree with it?) Given that, it's weird to say that it is better to give the money to the public than to let the government spend it.

I think the claim "philanthropic spending can do more good than typical government spending" usually works because we agree with the philanthropist's values more so than "government's values". But I wouldn't expect that "public's values" would be better than "government's values", and I do expect that "government's competence" would be better than "public's competence".

Comment by rohinmshah on Charity Vouchers [public policy idea] · 2019-07-10T19:19:18.493Z · score: 2 (2 votes) · EA · GW

Not necessarily disagreeing, but I wanted to point out that this relies on a perhaps-controversial claim:

Claim: Even though government is supposed to spend money to achieve outcomes the public wants, it is better to give the money to the public so that they can achieve outcomes that they want.

Comment by rohinmshah on Please May I Have Reading Suggestions on Consistency in Ethical Frameworks · 2019-07-08T17:24:22.521Z · score: 1 (1 votes) · EA · GW

To me, the most relevant of these impossibility theorems is the Arrhenius paradox (relevant to population ethics). Unfortunately, I don't know of any good public explanation of it.

Comment by rohinmshah on Not getting carried away with reducing extinction risk? · 2019-06-04T20:37:11.612Z · score: 3 (2 votes) · EA · GW

Even with the astronomical waste argument, which is the most extreme version of this argument, at some point you have astronomical numbers of people living, and the rest of the future isn't tremendously large in comparison, and so focusing on flourishing at that point makes more sense. Of course, this would be quite far in the future.

In practice, I expect the bar comes well before that point, because if everyone is focusing on x-risks, it will become harder and harder to reduce x-risks further, while staying equally as easy to focus on flourishing.

Note that in practice many more people in the world focus on flourishing than on x-risks, so maybe the few long-term focused people might end up always prioritizing x-risks because everyone else picks the low-hanging fruit in flourishing. But that's different from saying "it's never important to work on animal suffering", it's saying "someone else will fix animal suffering, and so I should do the other important thing of reducing x-risk".

Comment by rohinmshah on Not getting carried away with reducing extinction risk? · 2019-06-02T16:41:47.082Z · score: 14 (6 votes) · EA · GW

I'm pretty sure all the people you're thinking about won't make claims any stronger than "All of EA's resources should currently be focused on reducing extinction risks". Once extinction risks are sufficiently small, I would expect them to switch to focusing on flourishing.

Comment by rohinmshah on Cash prizes for the best arguments against psychedelics being an EA cause area · 2019-05-24T21:43:59.047Z · score: 3 (2 votes) · EA · GW
The random chance argument is harder to make if the studies have large effect sizes. If the true effect is 0, it's unlikely we'll observe a large effect by chance.

This is exactly what p-values are designed for, so you are probably better off looking at p-values rather than effect size if that's the scenario you're trying to avoid.

I suppose you could imagine that p-values are always going to be just around 0.05, and that for a real and large effect size people use a smaller sample because that's all that's necessary to get p < 0.05, but this feels less likely to me. I would expect that with a real, large effect you very quickly get p < 0.01, and researchers would in fact do that.

(I don't necessarily disagree with the rest of your comment, I'm more unsure on the other points.)

Comment by rohinmshah on [Link] Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-04-26T16:26:27.734Z · score: 4 (3 votes) · EA · GW

See also my summary and Richard Ngo's comments.

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-14T17:25:53.506Z · score: 1 (1 votes) · EA · GW

Yeah, I've been doing this occasionally (though that started recently).

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-11T02:00:51.632Z · score: 3 (2 votes) · EA · GW
From my present vantage, the AI alignment newsletter is becoming a pretty prominent clearinghouse for academic AI alignment research updates. (I wouldn't be surprised if it were the primary source of such for a sizable portion of newsletter subscribers.)
To the extent that's true, the amplification effects seem possibly strong.

I agree that's true and that the amplification effects for AI safety researchers are strong; it's much less strong of an amplification effect for any other category. My current model is that info hazards are most worrisome when they spread outside the AI safety community.

On confidentiality, the downsides of the newsletter failing to preserve confidentiality seem sufficiently small that I'm not worried (if you ignore info hazards). Failures of confidentiality seem bad in that they harm your reputation and make it less likely that people are willing to talk to you -- it's similar to the reason you wouldn't break a promise even if superficially the consequences of the thing you're doing seem slightly negative. But in the case of the newsletter, we would amplify someone else's failure to preserve confidentiality, which shouldn't reflect all that poorly on us. (Obviously if we knew that the information was supposed to be confidential we wouldn't publish it.)

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-11T01:50:41.511Z · score: 3 (2 votes) · EA · GW
This was in response to "the growing amount of AI safety research."

Yeah, I think I phrased that question poorly. The question is both "should all of it be summarized" and "if yes, how can that be done".

Presumably as there is more research, it takes more time to read & assess the forthcoming literature to figure out what's important / worth including in the newsletter.

I feel relatively capable of that -- I think I can figure out for any given reading whether I want to include it in ~5 minutes or so with relatively high accuracy. It's actually reading and summarizing it that takes time.

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T22:56:11.250Z · score: 3 (2 votes) · EA · GW
Interesting to think about what governance the newsletter should have in place re: info hazards, confidentiality, etc.

Currently we only write about public documents, so I don't think these concerns arise. I suppose you could imagine that someone writes about something they shouldn't have and we amplify it, but I suspect this is a rare case and one that should be up to my discretion.

What did you guys do for GPT-2?

Not sure what specifically you're asking about here. You can see the relevant newsletter here.

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T22:50:27.888Z · score: 1 (1 votes) · EA · GW
My intuition is that this would be a good time to formalize the structure of the newsletter somewhat, especially given that there are multiple contributors & you are starting to function more as an editor.

Certainly more systems are being put into place, which is kind of like "formalizing the structure". Creating an organization feels like a high fixed cost for not much benefit -- what do you think the main benefits would be? (Maybe this is combined with paying content writers and editors, in which case an organization might make more sense?)

Plausibly it's fine to keep it as an informal research product, but I'd guess that "AI alignment newsletter editor" could basically be (or soon become) a full-time job.

If I were to make this my full-time job, the newsletter would approximately double in length (assuming I found enough content to cover), and I'd expect that people wouldn't read most of it. (People already don't read all of it, I'm pretty sure.) What do you think would be the value of more time put into the newsletter?

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T22:44:25.779Z · score: 1 (1 votes) · EA · GW
My first guess is that there's significant value in someone maintaining an open, exhaustive database of AIS research.

Yeah, I agree. But there's also significant value in doing more AIS research, and I suspect that on the current margin for a full-time researcher (such as myself) it's better to do more AIS research compared to writing summaries of everything.

Note that I do intend to keep adding all of the links to the database, it's the summaries that won't keep up.

It is plausible to me that an org with a safety team (e.g. DeepMind/OpenAI) is already doing this in-house, or planning to do so.

I'm 95% confident that no one is already doing this, and if they were seriously planning to do so I'd expect they would check in with me first. (I do know multiple people at all of these orgs.)

More broadly, these labs might have some good systems in place for maintaining databases of new research in areas with a much higher volume than AIS, so could potentially share some best-practices.

You know, that would make sense as a thing to exist, but I suspect it does not. Regardless that's a good idea, I should make sure to check.

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T07:04:28.487Z · score: 1 (1 votes) · EA · GW

Comment thread for the question: What is the value of the newsletter for you?

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T07:04:11.529Z · score: 1 (1 votes) · EA · GW

Comment thread for the question: What is the value of the newsletter for other people?

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T07:03:55.320Z · score: 1 (1 votes) · EA · GW

Comment thread for the question: How should I deal with the growing amount of AI safety research?

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T07:03:37.958Z · score: 1 (1 votes) · EA · GW

Comment thread for the question: What can I do to get more feedback on the newsletter on an ongoing basis (rather than having to survey people at fixed times)?

Comment by rohinmshah on Alignment Newsletter One Year Retrospective · 2019-04-10T07:03:21.129Z · score: 1 (1 votes) · EA · GW

Comment thread for the question: Am I underestimating the risk of causing information cascades? Regardless, how can I mitigate this risk?

Comment by rohinmshah on Long-Term Future Fund: April 2019 grant recommendations · 2019-04-09T16:15:42.475Z · score: 9 (3 votes) · EA · GW

Is this different from having more people on a single granting body?

Possibly with more people on a single granting body, everyone talks to each other more and so can all get stuck thinking the same thing, whereas they would have come up with more / different considerations had they been separate. But this would suggest that granting bodies would benefit from splitting into halves, going over grants individually, and then merging at the end. Would you endorse that suggestion?

Comment by rohinmshah on On AI and Compute · 2019-04-04T21:26:49.232Z · score: 1 (1 votes) · EA · GW

Mostly agree with all of this; some nitpicks:

My understanding (and I think everyone else's) of AI capabilities is largely shaped by how impressive the results of major papers intuitively seem.

I claim that this is not how I think about AI capabilities, and it is not how many AI researchers think about AI capabilities. For a particularly extreme example, the Go-explore paper out of Uber had a very nominally impressive result on Montezuma's Revenge, but much of the AI community didn't find it compelling because of the assumptions that their algorithm used.

I'm not sure I fully understand how the metric would work. For the Atari example, it seems clear to me that we could easily reach it without making a generalizable AI system, or vice versa.

Tbc, I definitely did not intend for that to be an actual metric.

But let's say that we could come up with a relevant metric. Then I'd agree with Garfinkel, as long as people in the community had known roughly the current state of AI in relation to it and the rate of advance toward it before the release of "AI and Compute".

I would say that I have a set of intuitions and impressions that function as a very weak prediction of what AI will look like in the future, along the lines of that sort of metric. I trust timelines based on extrapolation of progress using these intuitions more than timelines based solely on compute.To the extent that you hear timeline estimates from people like me who do this sort of "progress extrapolation" who also did not know about how compute has been scaling, you would want to lengthen their timeline estimates. I'm not sure how timeline predictions break down on this axis.

Comment by rohinmshah on On AI and Compute · 2019-04-04T15:58:43.167Z · score: 9 (4 votes) · EA · GW
DeepMind certainly seems to be saying that AlphaZero is better at searching a more limited set of promising moves than Stockfish, a traditional chess engine (unfortunately they don’t compare it to earlier versions of AlphaGo on this metric).

Only at test time. AlphaZero has much more experience gained from its training phase. (Stockfish has no training phase, though you could think of all of the human domain knowledge encoded in it as a form of "training".)

AlphaZero went from a bundle of blank learning algorithms to stronger than the best human chess players in history...in less than two hours.

Humans are extremely poorly optimized for playing chess.

I don’t agree with Garfinkel that OpenAI’s analysis should make us more pessimistic about human-level AI timelines. While it makes sense to revise our estimate of AI algorithms downward, it doesn’t follow that we should do the same for our estimate of overall progress in AI. By cortical neuron count, systems like AlphaZero are at about the same level as a blackbird (albeit one that lives for 18 years),[7] so there’s a clear case for future advances being more impressive than current ones as we approach the human level.

Sounds like you are using a model where (our understanding of) current capabilities and rates of progress of AI are not very relevant for determining future capabilities, because we don't know the absolute quantitative capability corresponding to "human-level AI". Instead, you model it primarily on the absolute amount of compute needed.

Suppose you did know the absolute capability corresponding to "human-level AI", e.g. you can say something like "once we are able to solve Atari benchmarks using only 10k samples from the environment, we will have human-level AI", and you found that metric much more persuasive than the compute used by a human brain. Would you then agree with Garfinkel's point?

Comment by rohinmshah on Is any EA organization using or considering using Buterin et al.'s mechanism for matching funds? · 2019-04-04T15:34:42.666Z · score: 1 (1 votes) · EA · GW
In my understanding, coordination/collusion can be limited by keeping donations anonymous.

It's not hard for an individual to prove that they donated by other means, e.g. screenshots and bank statements.

(See the first two paragraphs on page 16 in the paper for an example.)

Right after that, the authors say:

There is a broader point here. If perfect harmonization of interests is possible, Capitalism leads to optimal outcomes. LR is intended to overcome such lack of harmonization and falls prey to manipulation when it wrongly assumes harmonization is difficult

With donations it is particularly easy to harmonize interests: if I'm planning to allocate 2 votes to MIRI and you're planning to allocate 2 votes to AMF, we can instead have each of us allocate 1 vote each to MIRI and AMF and we both benefit. Yes, we have to build trust that neither of us would defect by actually putting both of our votes to our preferred charity; but this seems doable in practice: even in the hardest case of vote trading (where there are laws attempting to enforce anonymity and inability to prove your vote) there seems to have been some success.

Comment by rohinmshah on Is any EA organization using or considering using Buterin et al.'s mechanism for matching funds? · 2019-04-03T15:06:56.827Z · score: 9 (4 votes) · EA · GW

Sorry, I meant "collusion" in the sense that it is used in the game theory literature, where it's basically equivalent to "coordination in a way not modeled by the game theory", and doesn't carry the illegal/deceitful connotation it does in English. See e.g. here, which is explicitly talking about this problem for Glen Weyl's proposal.

The overall point is, if donors can coordinate, as they obviously can in the real world, then the optimal provisioning of goods theorem no longer holds. The example with MIRI showcased this effect. I'm not saying that anyone did anything wrong in that example.

Comment by rohinmshah on Is any EA organization using or considering using Buterin et al.'s mechanism for matching funds? · 2019-04-03T00:02:42.696Z · score: -4 (2 votes) · EA · GW

Yes.

Comment by rohinmshah on Is any EA organization using or considering using Buterin et al.'s mechanism for matching funds? · 2019-04-02T15:52:44.348Z · score: 1 (1 votes) · EA · GW

The main issue with the mechanism seems to be collusion between donors. As Aaron mentioned, MIRI took part in such a fundraiser. I claim that it was so successful for them precisely because MIRI supporters were able to coordinate well relative to the supporters of the other charities -- there were a bunch of posts about how supporting this fundraiser was effectively a 50x donation multiplier or something like that.

Comment by rohinmshah on Unsolicited Career Advice · 2019-03-04T18:32:32.669Z · score: 20 (9 votes) · EA · GW

I ran the EA Berkeley group and later the UWashington group, and even this estimate seems high to me (but it would be within my 90% confidence bound, whereas 2000 is definitely not in it).

Comment by rohinmshah on Why do you reject negative utilitarianism? · 2019-02-12T18:31:58.991Z · score: 26 (13 votes) · EA · GW
Therefore, it is a straw man argument that NUs don’t value life or positive states, because NUs value them instrumentally, which may translate into substantial practical efforts to protect them (compared even with someone who claims to be terminally motivated by them).

By my understanding, a universe with no conscious experiences is the best possible universe by ANU (though there are other equally good universes as well). Would you agree with that?

If so, that's a strong reason for me to reject it. I want my ethical theory to say that a universe with positive conscious experiences is strictly better than one with no conscious experiences.

Comment by rohinmshah on What are some lists of open questions in effective altruism? · 2019-02-06T20:20:16.172Z · score: 3 (3 votes) · EA · GW

I was going to post a few lists that hadn't already been posted, but this one had all of them already :)

Comment by rohinmshah on Disentangling arguments for the importance of AI safety · 2019-01-23T17:42:55.145Z · score: 2 (2 votes) · EA · GW

I think 4, 5 and 6 are all valid even if you take the CAIS view. Could you explain how you think those depend on the AGI being an independent agent?

Plausibly 2 and 3 also apply to CAIS, though those are more ambiguous.

Comment by rohinmshah on Altruistic Motivations · 2019-01-05T16:06:05.061Z · score: 2 (2 votes) · EA · GW

Actually, my summary of that post initially dropped the obligation frame because of these reasons :P (Not intentionally, since I try to have objective summaries, but I basically ignored the obligation point while reading and so forgot to put it in the summary.)

I do think the opportunity frame is much more reasonable in that setting, because "human safety problems" are something that you might have been resigned to in the past, and AI design is a surprising option that might let us fix them, so it really does sound like good news. On the other hand, the surprising part about effective altruism is "people are dying for such preventable reasons that we can stop it for thousands of dollars", which is bad news that it's really hard to be excited by.

Comment by rohinmshah on 2018 AI Alignment Literature Review and Charity Comparison · 2019-01-04T20:40:01.569Z · score: 8 (4 votes) · EA · GW

Not sure. A few hypotheses:

  • Arxiv sanity has become better at predicting what I care about as I've given it more data. I don't think this is the whole story because the absolute number of papers I see on Twitter has gone down.
  • I did create my Twitter account primarily for academic stuff, but it's possible that over time Twitter has learned to show me non-academic stuff that is more attention-grabbing or controversial, despite me trying not to click on those sorts of things.
  • Academics are promoting their papers less on Twitter.
Comment by rohinmshah on 2018 AI Alignment Literature Review and Charity Comparison · 2018-12-31T14:33:22.493Z · score: 7 (3 votes) · EA · GW

Not the OP, but the Alignment Newsletter (which I write) should help for technical AI safety. I source from newsletters, blogs, Arxiv Sanity and Twitter (though Twitter is becoming more useless over time). I'd imagine you could do the same for other fields as well.