How prominent is EA in animal advocacy? 2023-02-09T22:00:00.246Z
What do EA's think about Bayesian inferences versus other types? 2023-02-04T11:09:12.681Z
EA's don't quantify much or very well 2022-12-17T09:47:13.612Z
How to reconsider a prediction 2022-10-25T21:28:32.050Z
Does EA have good models of current global spending on climate change prevention, mitigation, or adaptation? 2022-10-04T23:41:02.067Z
Noah Scales's Shortform 2022-09-10T09:00:53.207Z
Prefer beliefs to credence probabilities 2022-09-01T02:04:05.437Z
Will longtermists self-efface 2022-08-12T02:32:39.108Z
Do EA folks think that a path to zero AGI development is feasible or worthwhile for safety from AI? 2022-07-17T08:47:16.327Z
Does the idea of AGI that benevolently control us appeal to EA folks? 2022-07-16T19:17:56.933Z
Do EA folks want AGI at all? 2022-07-16T05:44:10.164Z
Climate change is Now Self-amplifying 2022-07-11T10:48:21.981Z
Well-studied Existential Risks with Predictive Indicators 2022-07-06T22:13:10.810Z


Comment by Noah Scales on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-18T20:03:43.645Z · EA · GW

Yes, I took a look at your discussion with MichaelStJules. There is a difference in reliability between:

  • probability that you assign to the Mugger's threat
  • probability that the Mugger or a third party assigns to the Mugger's threat

Although I'm not a fan of subjective probabilities, that could be because I don't make a lot of wagers.

There are other ways to qualify or quantify differences in expectation of perceived outcomes before they happen. One way is by degree or quality of match of a prototypical situation to the current context. A prototypical situation has one outcome. The current context could allow multiple outcomes, each matching a different prototypical situation. How do I decide which situation is the "best" match?

  • a fuzzy matching: a percentage quantity showing degree of match between prototype and actual situation. This seems the least intuitive to me. The conflation of multiple types and strengths of evidence (of match) into a single numeric system (for example, that bit of evidence is worth 5%, that is worth 10%) is hard to justify.
  • a hamming distance: each binary digit is a yes/no answer to a question. The questions could be partitioned, with the partitions ranked, and then hamming distances calculated for each ranked partition, with answers about the situation in question, and questions about identifying a prototypical situation.
  • a decision tree: each situation could be checked for specific values of attributes of the actual context, yielding a final "matches prototypical situation X" or "doesn't match prototypical situation X" along different paths of the tree. The decision tree is most intuitive to me, and does not involve any sums.

In this case, the context is one where you decide whether to give any money to the mugger, and the prototypical context is a payment for services or a bribe. If it were me, the fact that the mugger is a mugger on the street yields the belief "don't give" because, even if I gave them the money, they'd not do whatever it is that they promise anyway. That information would appear in a decision tree, somewhere near the top, as "person asking for money is a criminal?(Y/N)"

Comment by Noah Scales on Deontic Fictionalism · 2023-02-18T19:11:49.520Z · EA · GW

Simple and useful, thanks.

Comment by Noah Scales on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-17T20:54:58.862Z · EA · GW

In my understanding, Pascal's Mugger offers a set of rewards with risks that I estimate myself. Meanwhile, I need a certain amount of money to give to charity, in order to accomplish something. Let's assume that I don't have the money sufficient for that donation, and have no other way to get that money. Ever. I don't care to spend the money I do have on anything else. Then, thinking altruistically, I'll keep negotiating with Pascal's Mugger until we agree on an amount that the mugger will return that, if I earn it, is sufficient to make that charitable donation. All I've done is establish what amount to get in return from the Mugger before I give the mugger my wallet cash. Whether the mugger is my only source of extra money, and whether there is any other risk in losing the money I do have, and whether I already have enough money to make some difference if I donate, is not in question. Notice that some people might object that my choice is irrational. However, the mugger is my only source of money, and I don't have enough money otherwise to do anything that I care about for others, and I'm not considering consequences to me of losing the money.

In Yudkowsky's formulation, the Mugger is threatening to harm a bunch of people, but with very low probability. Ok. I'm supposed to arrive at an amount that I would give to help those people threatened with that improbable risk, right? In the thought experiment, I am altruistic. I decide what the probability of the Mugger's threat is, though. The mugger is not god, I will assume. So I can choose a probability of truth p < 1/(number of people threatened by the mugger) because no matter how many people that the mugger threatens, the mugger doesn't have the means to do it, and the probability p declines with the increasing number of people that the mugger threatens, or so I believe. In that case, aren't people better off if I give that money to charity after all?

You wrote,

"I can see it might make sense to set yourself a threshold of how much risk you are willing to take to help others. And if that threshold is so low that you wouldn't even give all the cash currently in your wallet to help any number of others in need, then you could refuse the Pascal mugger."

The threshold of risk you refer to there is the additional selfish one that I referred to in my last comment, where loss of the money in an altruistic effort deprives me of some personal need that the money could have served, an opportunity cost of wagering for more money with the mugger. That risk could be a high threshold of risk even if the monetary amount is low. Lets say I owe a bookie 5 dollars and if I don't repay they'll break my legs. Therefore, even though I could give the mugger 5 dollars and in my estimation, save some lives, I won't. Because the 5 dollars is all I have and I need it to repay the bookie. That personal need to protect myself from the bookie defines that threshold of risk. Or more likely, it's my rent money, and without it, I'm turned out onto predatory streets. Or it's my food money for the week, or my retirement money, or something else that pays for something integral to my well-being. That's when that personal threshold is meaningful.

Many situations could come along offering astronomical altruistic returns, but if taking risks for those returns will incur high personal costs, then I'm not interested in those returns. This is why someone with a limited income or savings typically shouldn't make bets. It's also why Effective Altruism's betting focus makes no sense for bets with sizes that impact a person's well-being when the bets are lost. I think it's also why, in the end, EA's don't put their money where their mouthes are.

EA's don't make large bets or they don't make bets that risk their well-being. Their "big risks" are not that big, to them. Or they truly have a betting problem, I suppose. It's just that EA's claim that betting money clarifies odds because EA's start worrying about opportunity costs, but does it? I think the amounts involved don't clarify anything, they're not important amounts to the people placing bets. What you end up with is a betting culture, where unimportant bets go on leading to limited impact on bayesian thinking, at best, to compulsive betting and major personal losses, at worst. By the way, Singer's utilitarian ideal was never to bankrupt people. Actually, it was to accomplish charity cost-effectively, implicitly including personal costs in that calculus (for example, by scaling % income that you give to help charitable causes according to your income size). Just an aside.

Comment by Noah Scales on The Locality Problem: how to balance the need and dangers of subjectivity and its refusal · 2023-02-17T09:23:17.596Z · EA · GW

Hmm. Interesting, but I don't understand the locality problem. I suspect that you think of consequences as non-local, but instead far-flung, thus involving you in weighing interests with greater significance than you would prefer for decisions. Is that the locality problem to you?

Comment by Noah Scales on Why should ethical anti-realists do ethics? · 2023-02-17T00:25:39.972Z · EA · GW

What an interesting and fun post! Your analysis goes many directions and I appreciate your investigation of normative, descriptive, and prescriptive ethics.

The repugnant conclusion worries me. As a thought experiment, it seems to contain an uncharitable interpretation of principles of utilitarianism.

  1. You increase total and average utility to measure increases in individual utility across an existing and constant population. However, those measures, total and average, are not adequate to handle the intuition people associate with them. Therefore, they should not be used for deciding changes in utility across a population of changing size or one containing drastic differences in individual utility. For example, there's no value in increasing total utility by adding additional people, but it will drive total utility up, even if individual utility is low.

  2. You pursue egalitarianism to raise everyone's utility up to the same level. Egalitarian is not an aspiration to lower some people's well-being while raising other's well-being. Likewise, egalitarianism is not pursuit of equality of utility at any utility level. Therefore, egalitarianism does not imply an overriding interest in equalizing everyone's utility. For example, there's no value in lowering other's utility to match those with less.

  3. You measure utility accumulated by existent people in the present or the future to know utility for all individuals in a population and that utility is only relevant to the time period during when those people exist. Those individuals have to exist in order for the measures to apply. Therefore, utilitarianism can be practiced in contexts of arbitrary changes in population, with a caveat: consequences for others of specific changes to population, someone's birth or death, are relevant to utilitarian calculations. TIP: the repugnant conclusion thought experiment only allows one kind of population change: increase. You could ask yourself whether the thought experiment says anything about the real world or requirements of living in it.

  4. Utility is defined with respect to purposes (needs, reasons, wants) that establish a reference point of accumulation of utility suitable for some purpose. That reference point is always at a finite level of accumulation. Therefore, to assume that utility should be maximized to an unbounded extent is an error, and speaks to a problem with some arguments for transitivity. NOTE: by definition, if there is no finite amount of accumulated utility past which you have an unnecessary amount for your purposes, then it is not utility for you.

The repugnant conclusion does not condemn utilitarianism to disuse, but points 1-4 seem to me to be the principles to treat charitably in showing that utilitarianism leads to inconsistency. I don't believe that current formulations of the repugnant conclusion are charitable to those principles and the intuitions behind them.

Comment by Noah Scales on We are incredibly homogenous · 2023-02-16T22:03:17.304Z · EA · GW

About steel-manning vs charitably interpreting

The ConcernedEA's state:

"People with heterodox/'heretical' views should be actively selected for when hiring to ensure that teams include people able to play 'devil’s advocate' authentically, reducing the need to rely on highly orthodox people accurately steel-manning alternative points of view"

I disagree. Ability to accurately evaluate the views of the heterodox minority depends on developing a charitable interpretation (not necessarily a steel-manning) of the views. Furthermore, if the majority can not or will not develop such a charitable interpretation, then the heretic must put their argument in a form that the majority will accept (for example, using jargon and selectively adopting non-conflicting elements of the majority ideology). This unduly increases burden on the person with heterodox views.

The difference between a charitably -interpreted view and a steel-manned view is that the steel-manned view is strengthened to seem like a stronger argument to the opposing side. Unfortunately, if there are differences in evaluating strength of evidence or relevance of lines of argument (for example, due to differing experiences between the sides), then steel-manning will actually distort the argument. A charitable interpretation only requires that you accurately determine what the person holding the view intends to mean when they communicate it, not that you make the argument seem correct or persuasive to you.

Sometimes I think EA's mean "charitable interpretation" when they write "steel-manning". Other times I think that they don't. So I make the distinction here.

It's up to the opposing side to charitably interpret any devil's advocate position or heretical view. While you could benefit from including diverse viewpoints, the burden is on you to interpret them correctly, to gain any value available from them.

Developing charitable interpretation skills

To charitably interpret another's viewpoint takes Scout Mindset, first of all. With the wrong attitude, you'll produce the wrong interpretation no matter how well you understand the opposing side. It also takes some pre-existing knowledge of the opposing side's worldview, typical experiences, and typical communication patterns. That comes from research and communication skills training. Trial-and-error also plays a role: this is about understanding another's culture, like an anthropologist would. Immersion in another person's culture can help.

However, I suspect that the demands on EA's to charitably interpret other people's arguments are not that extreme. Charitable interpretations are not that hard in the typical domains you require them. To succeed with including heterodox positions, though, demands on EA's empathy, imagination, and communication skills do go up.

About imagination, communication skills, and empathy for charitably interpreting

EA's have plenty of imagination, that is, they can easily consider all kinds of strange views, it's a notable strength of the movement, at least in some domains. However, EA's need training or practice in advanced communication skills and argumentation. They can't benefit from heterodox views without them. Their idiosyncratic takes on argumentation (adjusting Bayesian probabilities) and communication patterns (schelling points) fit some narrative about their rationalism or intelligence, I suppose, but they could benefit from long-standing work in communication, critical thinking, and informal logic. As practitioners of rationalism to the degree that mathematics is integral, I would think that EA's would have first committed their thinking to consistent analysis with easier tools, such as inference structures, setting aside word-smithing for argument analysis. Instead, IBT gives EA's the excuse not to grapple with the more difficult skills of analyzing argument structures, detailing inference types, and developing critical questions about information gaps present in an argument. EDIT: that's a generalization, but is how I see the impact of IBT in practical use among EA's.

The movement has not developed in any strong way around communication skills specifically, aside from a commitment to truth-seeking and open-mindedness, neither of which is required in order to understand others' views, but are still valuable to empathy.

There's a generalization that "lack of communication skills" is some kind of remedial problem. There are communication skills that fit that category, but those skills are not what I mean.

After several communication studies courses, I learned that communication skills are difficult to develop, that they require setting aside personal opinions and feelings in favor of empathy, and that specific communication techniques require practice. A similar situation exists with interpreting arguments correctly: it takes training in informal logic and plenty of practice. Scout mindset is essential to all this, but not enough on its own.

Actually, Galef's podcast Rationally Speaking includes plenty of examples of charitable interpretation, accomplished through careful questions and sensitivity to nuance, so there's some educational material there.

Typically the skills that require practice are the ones that you (and I) intentionally set aside at the precise time that they are essential: when our emotions run high or the situation seems like the wrong context (for example, during a pleasant conversation or when receiving a criticism). Maybe experience helps with that problem, maybe not. It's a problem that you could address with cognitive aids, when feasible.

Is moral uncertainty important to collective morality?

Ahh, am I right that you see the value of moral uncertainty models as their use in establishing a collective morality given differences in the morality held by individuals?

Comment by Noah Scales on We are incredibly homogenous · 2023-02-16T20:49:39.031Z · EA · GW

You state: "Effective Altruist Political Ideology is hardly correct in every detail, but I don't think it's a bad sign if a movement broadly agrees on a lot of political issues. Some political policies are harmful! Other policies make things better!"

I identified EA as right-leaning because of lack of EA concern about climate change, as well as an emphasis in other areas (economics, personal finances, corporate regulation, technology development) that matches a right-leaning worldview. However, according to this 2018 survey , EA's lean left, more than 60%.

There's some overlap or really, flexibility, in how lefties in California approach financial and economic issues. Their left-leaning ideology expresses itself with opinions on abortion, racism, and climate change, and less with opinions about taxation, corporate regulation, or technology development. Which leads me to conclude that it is not helpful for me to identify EA's with larger movements when dealing with EA views on specific issues. Better to focus on a specific EA brand of political ideology being developed inside the movement, and describe its formative influences (as the OP does), than to assume a more typical political ideology is present, such as liberal or conservative ideologies.

You state: "However, I think that this subject should be addressed with care. When you’re talking about homogeneity, it’s important to acknowledge effective altruist members of various groups underrepresented in effective altruism. Very few things are more unwelcoming than 'by the way, people like you don’t exist here.'"

You think that acknowledging the diversity already present in EA is important, and I agree. The ConcernedEA's don't intend to insult or isolate any group. They are sincere in wanting to increase diversity in the EA movement, and their statements are to the effect that "The EA movement lacks diversity that would strengthen it provided there were some necessary overlap in values held by all."

Comment by Noah Scales on Nobody Wants to Read Your Sh*t: my favorite book of writing advice, condensed for your convenience · 2023-02-15T23:49:41.672Z · EA · GW

Yeah. I'll add:

  • Single-sourcing: Building Modular Documentation by Kurt Ament
  • Dictionary of Concise Writing by Robert Hartwell Fiske
  • Elements of Style by William Strunk Jr
  • A Rulebook for Arguments by Anthony Weston

There are more but I'm not finished reading them. I can't say that I've learned what I should from all those books, but I got the right idea, more than once, from them.

Comment by Noah Scales on What actually is “value-alignment”? · 2023-02-15T23:19:38.890Z · EA · GW suggests that EA values include:

  1. proper prioritization: appreciating scale of impact, and trying for larger scale impact (for example, helping more people)
  2. impartial altruism: giving everyone's interests equal weight
  3. open truth-seeking: including willingness to make radical changes based on new evidence
  4. collaborative spirit: involving honesty, integrity, and compassion, and paying attention to means, not just ends.

Cargill Corporation lists its values as:

  1. Do the Right Thing
  2. Put People First
  3. Reach Higher

Lockheed-Martin Corporation lists its values as:

  1. Do What’s Right
  2. Respect Others
  3. Perform with Excellence

Shell Global Corporation lists its values as:

  1. Integrity
  2. Honesty
  3. Respect

Short lists seem to be a trend, but longer lists with a different label than "values" appear from other corporations(for example, from Google or General Motors) . They all share the quality of being aspirational, but there's a difference with the longer lists, they seem closer suited to the specifics of what the corporations do.

Consider Google's values:

  • Focus on the user and all else will follow.
  • It's best to do one thing really, really well.
  • Fast is better than slow.
  • Democracy on the web works.
  • You don't need to be at your desk to need an answer.
  • You can make money without doing evil. .
  • There's always more information out there.
  • The need for information crosses all borders
  • You can be serious without a suit
  • Great just isn't good enough

Google values are specific. Their values do more than build their brand.

I would like to suggest that EA values are lengthy and should be specific enough to:

  • identify your unique attributes.
  • focus your behavior.
  • reveal your preferred limitations[1].

Having explicit values of that sort:

  • limit your appeal.
  • support your integrity .
  • encourage your honesty.

The values focus and narrow in addition to building your brand. Shell Global, Lockheed-Martin and Cargill are just building their brand. The Google Philosophy says more and speaks to their core business model.

All the values listed as part of Effective Altruism appear to overlap with the concerns that you raise. Obviously, you get into specifics.

You offer specific reforms in some areas. For example:

  • "A certain proportion EA of funds should be allocated by lottery after a longlisting process to filter out the worst/bad-faith proposals*"
  • "More people working within EA should be employees, with the associated legal rights and stability of work, rather than e.g. grant-dependent 'independent researchers'."

These do not appear obviously appropriate to me. I would want to find out what a longlisting process is, and why employees are a better approach than grant-dependent researchers. A little explanation would be helpful.

However, other reforms do read more like statements of value or truisms to me. For example:

  • "Work should be judged on its quality..."[rather than its source].
  • "EAs should be wary of the potential for highly quantitative forms of reasoning to (comparatively easily) justify anything"

It's a truism that statistics can justify anything as in the Mark Twain saying, "There are three kinds of lies: lies, damned lies, and statistics".

These reforms might inspire values like:

  • judge work on its quality alone, not its source
  • Use quantitative reasoning only when appropriate

*You folks put a lot of work into writing this up for EA's. You're smart, well-informed, and I think you're right, where you make specific claims or assert specific values. All I am thinking about here is how to clarify the idea of aligning with values, the values you have, and how to pursue them. *

You wrote that you started with a list of core principles before writing up your original long post? I would like to see that list, if it's not too late and you still have the list. If you don't want to offer the list now, maybe later? As a refinement of what you offered here?

Something like the Google Philosophy, short and to the point, will make it clear that you're being more than reactive to problems, but instead actually have either:

  • differences in values from orthodox EA's
  • differences in what you perceive as achievement of EA values by orthodox EA's

Here are a few prompts to help define your version of EA values:

  1. EA's emphasize quantitative approaches to charity, as part of maximizing their impact cost-effectively. Quantitative approaches have pros and cons, so how to contextualize them? They don't work in all cases, but that's not a bad thing. Maybe EA should only pay attention to contexts where quantitative approaches do work well. Maybe that limits EA flexibility and scope of operations, but also keeps EA integrity, accords with EA beliefs, and focuses EA efforts. You have specific suggestions about IBT and what makes a claim of probabilistic knowledge feasible. Those can be incorporated into a value statement. Will you help EA focus and limit its scope or are you aiming to improve EA flexibility because that's necessary in every context where EA operates?

  2. EA's emphasize existential risk causes. ConcernedEA's offer specific suggestions to improve EA research into existential risk. How would you inform EA values about research in general to include what you understand should be the EA approach to existential risk research? You heed concerns about evaluation of cascading and systemic risks. How would those specific concerns inform your values?

  3. You have specific concerns about funding arrangements, nepotism, and revolving doors between organizations. How would those concerns inform your values about research quality or charity impact?

  4. You have concerns about lack of diversity and its impact on group epistemics. What should be values there?

You can see the difference between brand-building:

  • ethicality
  • impactfulness
  • truth-seeking

and getting specific

  • research quality
  • existential, cascading, and systemic risks
  • scalable and impactful charity
  • quantitative and qualitative reasoning
  • multi-dimensional diversity
  • epistemic capability
  • democratized decision-making

That second list is more specific, plausibly hits the wrong notes for some people, and definitely demonstrates particular preferences and beliefs. As it should! Whatever your list looks like, would alignment with its values imply the ideal EA community for you? That's something you could take another look at, articulating the values behind specific reforms if those are not yet stated or incorporating specific reforms into the details of a value, like:

  • democratized decision-making: incorporating decision-making at multiple levels within the EA community, through employee polling, yearly community meetings, and engaging charity recipients.

I don't know whether you like the specific value descriptors I chose there. Perhaps I misinterpreted your values somewhat. You can make your own list. Making decisions in alignment with values is the point of having values. If you don't like the decisions, the values, or if the decisions don't reflect the values, the right course is to suggest alterations somewhere, but in the end, you still have a list of values, principles, or a philosophy that you want EA to follow.

[1] As I wrote in a few places in this post, and taking a cue from Google and the linux philosophy, sometimes doing one thing and doing it well is preferable to offering loads of flexibility. If EA is supposed to be the swiss-army knife of making change in the world, there's still a lot of better organizations out there for some purposes rather than others, as any user of a swiss-army knife will attest, they are not ideal for all tasks. Also, your beliefs will inform you about what you do well. Does charity without quantitative metrics inevitably result in waste and corruption? Does use of quantitative metrics limit the applicability of EA efforts to specific types of charity work (for example, outreach campaigns)? Do EA quantitative tools limit the value of its work in existential risk? Can they be expanded with better quantitative tools (or qualitative ones)? Maybe EA is self-limiting because of its preferred worldview, beliefs and tools. Therefore, it has preferred limitations. Which is OK, even good.

Comment by Noah Scales on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-14T22:39:53.083Z · EA · GW

Hm, ok. Couldn't Pascal's mugger make a claim to actually being God (with some small probability or very weakly plausibly) and upset the discussion? Consider basing dogmatic rejection on something other than the potential quality of claims from the person whose claims you reject. For example, try a heuristic or psychological analysis. You could dogmatically believe that claims of godliness and accurate probabilism are typical expressions of delusions of grandeur.

My pursuit of giving to charity is not unbounded, because I don't perceive an unbounded need. If the charity were meant to drive unbounded increase in the numbers of those receiving charity, that would be a special case, and not one that I would sign up for. But putting aside truly infinite growth of perceived need for the value returned by the wager, in all wagers of this sort that anyone could undertake, they establish a needed level of utility, and compare the risk involved to whatever stakeholders of taking the wager at that utility level against the risks of doing nothing or wagering for less than the required level.

In the case of ethics, you could add an additional bounds on personal risk that you would endure despite the full need of those who could receive your charity. In other words, there's only so much risk you would take on behalf of others. How you decide that should be up to you. You could want to help a certain number of people, or reach a specific milestone towards a larger goal, or meet a specific need for everyone, or spend a specific amount of money, or whathaveyou, and recognize that level of charity as worth the risks involved to you of acquiring the corresponding utility. You just have to figure it out beforehand.

If by living 100 years, I could accomplish something significant, but not everything, on behalf of others, that I wanted, but I would not personally enjoy that time, then that subjective decision makes living past 100 years unattractive, if I'm deciding solely based on my charitable intent. I would not, in fact, live an extra 100 years for such a purpose without meeting additional criteria, but for example's sake, I offered it.

Comment by Noah Scales on Elements of Rationalist Discourse · 2023-02-14T22:13:58.920Z · EA · GW

I think identifying common modes of inference (e.g., deductive, inductive, analogy) can be helpful, if argument analysis takes place. Retrodiction is used to describe a stage of retroductive (abductive) reasoning, and so has value outside a Bayesian analysis.

If there's ever an equivalent in wider language for what you're discussing here (for example, "important premise" for "crux"), consider using the more common form rather than specialized jargon. For example, I find EA use of "counterfactual" to confuse me about the meaning of what I think are discussions of necessary conditions, whereas counterfactual statements are, to me, false statements, relevant in a discussion of hypothetical events that do not occur. Many times I wanted to discuss counterfactuals but worried that the conversation with EA's would lead to misunderstandings, as if my analysis were to explore necessary conditions for some action or consequence, when that was not the intent.

The "typical mind fallacy" is interesting. On the one hand, I think some inferences taking the form of shared values or experience are fallacious. On the other hand, some typical inferences about similarities between people are reliable and we depend on them. For example, that people dislike insults. A common word starting with 'n' has a special case, but is mostly taken as a deeply unwelcome insult, our default is to treat that knowledge as true. We rely on default (defeasible) reasoning when we employ those inferences, and add nuance or admit special cases for their exceptions. In the social world, the "typical mind fallacy" has some strong caveats.

Comment by Noah Scales on What is so wrong with the "dogmatic" solution to recklessness? · 2023-02-11T19:36:58.247Z · EA · GW

I'm not sure I'm understanding. It looks like at some K, you arbitrarily decide that the probability is zero, sooner than the table that the paper suggests. So, in the thought experiment, God decides what the probability is, but you decide that at some K, the probability is zero, even though the table lists the N at which the probability is zero where N > K. Is that correct?

Another way to look at this problem is with respect to whether what is gained through accepting a wager for a specific value is of value to you. The thought experiment assumes that you can gain very large amounts and matter how high the accumulated value at N, the end of the game, you still have a use for the amount that you could, in principle, gain.

However, for any valuable thing I can think of (years of life, money, puppies, cars), there's some sweet spot, with respect to me in particular. I could desire a 100 hundred years of life but not 1000, or 10 cars but not 100, or fifty million dollars but not five hundred million dollars, or one puppy but not ten. Accordingly, then, I know how much value to try to gain.

Assuming some pre-existing need, want, or "sweet spot", then, I can look at the value at i, where at i the value meets my need. If N< i, the question becomes whether I still gain if I get less value than I want. If N> i, then I know to take a risk up to K, where K = i and K < N. If N=i, then I know to play the game (God's game) to the end.

In real life, people don't benefit past some accumulation of valuable something, and what matters is deciding what level past which an accumulation is wasteful or counterproductive. One hundred cars would be too much trouble, even one puppy is a lot of puppies when you have to clean up puppy poop, and why not $500,000,000? Well, that's just more than I need, and would be more of a burden than a help. Put differently, if I really needed big sums, I'd take a risks for up to that amount, but no higher. When would I need such big sums and take the accompanying big risks? Maybe if I owed a bookie $50,000,000 and the bookie had very unpleasant collectors?

Comment by Noah Scales on Doing EA Better · 2023-02-10T06:19:42.614Z · EA · GW

Do you have specific concerns about how the capital is spent? That is, are you dissatisfied and looking to address concerns that you have or to solve problems that you have identified?

I'm wondering about any overlap between your concerns and the OP's.

I'd be glad for an answer or just a link to something written, if you have time.

Comment by Noah Scales on How prominent is EA in animal advocacy? · 2023-02-10T06:05:01.587Z · EA · GW

Well, thank you for the helpful follow-up. I went ahead and bought the book, and will read it. I have browsed three articles and read two through.

The first article was "Animal advocacy's Stockholm Syndrome", written by several authors. The tone of that article is positive toward EA, starting off with "It's time for Effective Altruists in the farmed animal protection movement to expand their strategic imagination, their imagination of what is possible, and their imagination of what counts as effective. ... Effective Altruist support has brought new respect and tractability to the neglected plight of farmed animals, and we ... are grateful. We write this essay as allies."

And then they write about difficulties in getting metrics thinking to apply to systemic change efforts in animal advocacy, and yes they do mention EA homogeneity as reason to expand diversity within EA and so develop new perspectives within it. I expect the calls for inclusiveness and diversity are a theme throughout the book.

The second article that I read was "How 'alternative proteins' create a private solution to a public problem" by Michele Simon, a veteran of the vegan food and animal rights movement.

Simon suggests that increasing investment in vegetarian meat replacements results in increasing profits for big food companies but not changes in the consumption behavior of existing meat eaters. Instead, the vegetarian meat replacements attract vegetarians to brands or restaurant chains. The article mentions that vegetarian options on a restaurant menu handle the "veto vote", that one person in a group who can't eat meat. Simon claims that offering a vegetarian option can result in more meat consumption at a restaurant like McDonalds as opposed to somewhere else serving pasta or salad or another option with less meat. However, I suspect that anyone willing to eat at McDonalds will eat a comparable meat meal at another restaurant (for example, Burger King) if a veto vote counts. Bringing vegetarians into their chains lets the chains sell more food overall.

Simon makes the point that alternative meats are being trialed, and if their sales drop, companies stop selling them. She lists a few examples from fast food chains to prove the point. Alternative proteins are not very popular when trialed, and initial enthusiasm and sales drop.

I interpret Simon to think that big food is interested in keeping up meat sales along with adding other products, and that there is no replacement of meat with non-meat taking place. Instead, food corporations are trying to appeal to vegetarians with foods that provide similar taste experiences but without the meat ingredients. That would explain why Simon thinks that lauding these companies for their inclusion of non-meat items really misses what the companies are trying to do. Basically, all the meat replacements do is give vegetarians a seat at the same table. The meat-eaters stick with the meat versions.

If Simon is right, then a switch to meat alternatives has to happen through consumer interest, from vegetarians or from meat eaters. It has to be driven by new demand, rather than new supply.

The article discusses GFI, cultured meat, and how the food industry subverts the food systems perspective, "where food comes from and how it is grown matters." Simon hints that there's an ontology useful for understanding food systems that the GFI marketing literature doesn't use.

Both the articles I read put out this "EA's are white guys" thing. I'm not offended, because I'm not an EA, and even if I were, maybe I should just agree? I am a white guy. There's some argument for increasing diversity in your community, the ConcernedEA's make a strong case in their recent sequence.

Where I think both of the articles I read are right is in claiming that EA does not offer a political or economic or regulatory perspective that puts its activism in opposition to larger business interests in food or animal farming.

I haven't explored the whole issue of legal precedents that the book addresses yet.

Thank you for your insights in all these areas, if you have more to add, please do. I appreciate the insider conversation.

Comment by Noah Scales on How prominent is EA in animal advocacy? · 2023-02-10T00:50:54.328Z · EA · GW

Thank you for the chapter pointers.

You mention obvious reasons. The reasons are not obvious to me, because I am ignorant about this topic. Do you mean that these critics are being self-serving and that some animal advocacy orgs lost funding for other reasons than EA competition or influence?

The book's introduction proposes:

  1. sanctuary X lost funding because of EA competition and other EA influence.
  2. legal cases to free animals lose some impetus because without a sanctuary for a freed animal, an abused animal could suffer a worse fate than their current abuse.
  3. me: legal cases create precedents that work at a larger scale later on, successful cases build a movement toward better treatment of animals, and sanctuaries contribute indirectly to that result.
  4. EA competition results in lost legal cases and reduced precedents for animal legal rights and positive treatment standards.

Premise 1 is what you think is false, if I understand you correctly, and substitution of another premise, such as:

  1. Sanctuary X lost funding because of poor fund-raising approaches or a drop in animal advocacy funding overall.

could be an alternative.

A plausible claim in the book could be that EA has a very strong reputation in animal advocacy, and is changing how animal advocacy is done by affecting how pre-EA orgs do their work (for example, their metrics). Is that something happening behind the scenes?

I'm not intending to find out whether EA is bad or not, more just how strong of a trend is the EA mode of thought (efficacy, metrics, large-scale impact), whatever it is currently, in deciding what's good for animals broadly. There'll always be some negative effects of any change in thinking, and some unintended consequences. However, I suspect that you, and other EA's, think that EA does not have a strong influence in the animal advocacy field overall, despite what these authors in the book claim. Am I right about that?

Comment by Noah Scales on Epistemic health is a community issue · 2023-02-09T12:01:34.875Z · EA · GW

I wrote:

"You need to rock selfishness well just to do charity well (that's my hunch)."

Selfishness, so designated, is not a public health issue nor a private mental health issue, but does stand in contrast to altruism. To the extent that society allows your actualization of something you could call selfishness, that seems to be your option to manifest, and by modern standards, without judgement of your selfishness. Your altruism might be judged, but not your selfishness, like, "Oh, that's some effective selfishness" vs "Oh, that's a poser's selfishness right there" or "That selfishness there is a waste of money".

Everyone thinks they understand selfishness, but there don't seem to be many theories of selfishness, not competing theories, nor ones tested for coherence, nor puzzles of selfishness. You spend a great deal of time on debates about ethics, quantifying altruism, etc, but somehow selfishness is too well-understood to bother?

The only argument over selfishness that has come up here is over self-care with money. Should you spend your money on a restaurant meal, or on charity? There was plenty of "Oh, take care of yourself, you deserve it" stuff going around, "Don't be guilty, that's not helpful" but no theory of how self-interest works. It all seems relegated to an ethereal realm of psychological forces, that anyone wanting to help you with must acknowledge.

Your feelings of guilt, and so on, are all tentatively taken as subjectively impactful and necessarily relevant just by the fact of your having them. If they're there, they matter. There's pop psychology, methods of various therapy schools, and different kinds of talk, really, or maybe drugs, if you're into psychiatric cures, but nothing too academic or well thought out as far as what self-interest is, how to perform it effectively, how or whether to measure it, and its proper role in your life. I can't just look at the problem, so described, and say, "Oh, well, you're not using a helpful selfishness theory to make your decisions there, you need to..." and be sure I'm accomplishing anything positive for you. I might come up with some clever reframe or shift your attention successfully, but that says nothing about a normative standard of selfishness that I could advocate.

I understand rationalization and being self-serving, but only in well-defined domains where I've seen it before, in what some people call "patterns of behavior." Vices do create pathological patterns of behavior, and ending them is clarifying and helpful to many self-interested efforts. A 100-hundred year effort to study selfishness is about more than vices. Or, well, at least on the surface, depending on what researchers discover. I have my own suspicions.

Anyway, we don't have the shared vocabulary to discuss vices well. What do you think I mean by them? Is adderall a vice? Lite beer? Using pornography? The occasional cigarette? Donuts? Let's say I have a vice or two, and indulge them regularly, and other people support me in doing that, but we end doing stuff together that I don't really like, aside from the vice. Is it correct then to say that I'm not serving myself by keeping my vice going? Or do we just call that a reframe because somebody's trying to manipulate me into giving up my habits? What if the vice gets me through a workday?

Well, there's no theories of self-interest that people study in school to help us understand those contexts, or if there are, they don't get much attention. I don't mean theories from psychology that tend to fail in practice. It's a century's effort to develop and distribute the knowledge to fill that need for good theories.

Galef took steps to understand selfish behavior. She decided that epistemic rationality served humanity and individuals, and decided to argue for it. That took some evaluation of behavior in an environment. It motivated pursuit of rationality in a particular way.

Interestingly, her tests, such as the selective critic test, or the double standard test, reveal information that shifts subjective experience. Why do we need those tests(Not, do we need them, but, why do we need them)? What can we do about the contexts that seem to require them? Right now, your community's culture encourages an appetite for risk, particularly financial risk, that looks like a vice. Vices seem to attract more vices.

You're talking about epistemics. A lot of lessons in decision-making are culturally inherited. For various reasons, modern society could lose that inheritance. Part of that inheritance is a common-sense understanding of vices. Without that common-sense there is only a naivete that could mean our extinction. Or that's how I see it.

For example, in 2020, one of the US's most popular talk show hosts (Steven Colbert) encouraged viewers to drink, and my governor (Gavin Newsom) gave a speech about loosening rules for food deliveries so that we could all get our wine delivered to our doors while we were in lockdown. I'm not part of the Christian right, but I think they still have the culture to understand that kind of behavior as showing decadence and inappropriateness. I would hope so. Overall, though, my country, America, didn't see it that way. Not when, at least in people's minds, there was an existential threat present. A good time to drink, stuck at home, that's apparently what people thought.

I'm really not interested in making people have a less fun time. That is not my point at all.

I've also been unsuccessful in persuading people to act in their own self-interest. I already know it doesn't work.

If you don't believe in "vices", you don't believe in them. That's fine. My point here was that it's not safe to ignore them, and I would like to add, there's nothing stronger than a vice to make sure you practice self-serving rationalization.

If, for the next 40-60 years, humanity faces a drawn out, painful coping with increasing harms from climate change, as I believe, and our hope for policy and recommendations is communities like yours, and what we get is depressed panicky people indulging whatever vices they can and becoming corrupt as f**k? Well, things will go badly.

Comment by Noah Scales on EigenKarma: trust at scale · 2023-02-08T23:05:59.305Z · EA · GW

I understand, Henrik. Thanks for your reply.

Forum karma

The karma system works similarly to highlight information, but there's these edge cases. Posts appear and disappear based on karma from first page views. New comments that get negative karma are not listed in the new comments from the homepage, by default.

This forum in relation to the academic peer review system

The peer review system in scientific research is truly different than a forum for second-tier researchers doing summaries, arguments, or opinions. In the forum there should be encouragement of access to disparate opinions and perspectives.

The value of disparate information and participants here

Inside the content offered here are recommendations for new information. I evaluate that information according to more conventional critical thinking criteria: peer-reviewed, established science, good methodology. Disparate perspectives among researchers here let me gain access to multiple points of view found in academic literature and fields of study. For example, this forum helped me research a foresight conflict between climate economists and earth scientists that is long-standing (as well as related topics in climate modeling and scenario development).

NOTE:Peer-reviewed information might have problems as well, but not ones to fix with a voting system relying on arbitrary participants.

Forum perspectives should not converge without rigorous argument

Another system that bubbles up what I'd like to read? OK, but will it filter out divergence, unpopular opinions, evidence that a person has a unique background or point of view, or a new source of information that contradicts current information? Will your system make it harder to trawl through other researchers' academic sources by making it less likely that forum readers ever read those researchers' posts?

In this environment, among folks who go through summaries, arguments, and opinions for whatever reason, once an information trend appears, if its different and valid, it lets me course correct.

The trend could signal something that needs changing, like "Here's new info that passes muster! Do something different now!" or signal that there's a large information gap, like "Woa, this whole conversation is different! I either seem to disagree with all of the conclusions or not understand them at all. What's going on? What am I missing?"

A learning environment

Forum participants implicitly encourage me to explore bayesianism and superforecasting. Given what I suspect are superforecasting problems (its aggregation algorithms and bias in them and in forecast confirmations), I would be loathe to explore it otherwise. However, obviously smart people continue to assert its value in a way that I digest as a forum participant. My minority opinion of superforecasting actually leads to me learning more about it because I participate in conversation here. However, if I were filtered out in my minority views so strongly that no one ever conversed with me at all I could just blog about how EA folks are really wrong and move on. Not the thing to do, but do you see why tolerance of my opinions here matters? It serves both sides.

From my perspective, it takes patience to study climate economists, superforecasting, bayesian inference, and probabilism. Meanwhile, you folks, with different and maybe better knowledge than mine on these topics, but a different perspective, provide that learning environment. If there can be reciprocation, that's good, EA folks deserve helpful outside perspectives.

My experience as other people's epistemic filter

People ignore my recommendations, or the intent behind them. They either don't read what I recommend, or much more rarely, read it but dismiss it without any discussion. If those people use anonymized voting as their augmentation approach, then I don't want to be their filter. They need less highlighting of information that they want to find, not more.

Furthermore, at this level of processing information, secondary or tertiary sources, posts already act like filters. Ranking the filtering to decide whether to even read it is a bit much. I wouldn't want to attempt to provide that service.


ChatGPT, and this new focus on conversational interfaces makes it possible that forum participants in future will be AI, not people. If so,they could be productive participants, rather than spam bots.

Meanwhile, the forum could get rid the karma system altogether, or add configuration that lets a user turn off karma voting and ranking. That would be a pleasant alternative for someone like me, who rarely gets much karma anyway. That would offer even less temptation to focus on popular topics or feign popular perspectives.

Comment by Noah Scales on EigenKarma: trust at scale · 2023-02-08T21:52:17.615Z · EA · GW

Right, the first class are the use cases that the OP put forward, and vote brigading is something that the admins here handle.

The second class is more what I asking about, so thank you for explaining why you would want a conversation bubble. I think if you're going to go that far for that reason, you could consider a entrance quiz. Then people who want to "join the conversation" could take the quiz, or read a recommended reading list, and then take the quiz, to gain entrance to your bubble.

I don't know how aversive people would find that, but if lack of technical knowledge were a true issue, that would be one approach to handling it while still widening the group of conversation participants.

Comment by Noah Scales on EigenKarma: trust at scale · 2023-02-08T21:09:30.123Z · EA · GW

Can you explain with an example when a bubble would be a desirable outcome?

Comment by Noah Scales on EigenKarma: trust at scale · 2023-02-08T20:57:23.649Z · EA · GW

Hmm. I've watched the scoring of topics on the forum, and have not seen much interest in topics that I thought were important for you, either because the perspective, the topic, or the users, were unpopular. The forum appears to be functioning in accordance with the voting of users, for the most part,because you folks don't care to read about certain things or hear from certain people. It comes across in the voting.

I filter your content, but only for myself. I wouldn't want my peers, no matter how well informed, deciding what I shouldn't read, though I don't mind them recommending information sources and I don't mind recommending sources of my own, on a per source basis. I try to follow the rule that I read anything I recommend before I recommend it. By "source" here I mean a specific body of content, not a specific producer of content.

I actually hesitate to strong vote, btw, it's ironic. I don't like being part of a trust system, in a way. It's pressure on me without a solution.

I prefer "friends" to reveal things that I couldn't find on my own, rather than, for their lack of "trust", hide things from me. More likely, their lack of trust will prove to be a mistake in deciding what I'd like to read. No one assumes I will accept everything I read, as far as I know, so why should they be protecting me from genuine content? I understand spam, AI spam would be a real pain, all of it leading to how I need viagra to improve my epistemics.

If this were about peer review and scientific accuracy, I would want to allow that system to continue to work, but still be able to hear minority views, particularly as my background knowledge of the science deepens. Then I fear incorrect inferences (and even incorrect data) a bit less. I still prefer scientific research to be as correct as possible, but scientific research is not what you folks do. You folks do shallow dives into various topics and offer lots of opinions. Once in a while there's some serious research but it's not peer-reviewed.

You referred to AI gaming the system, etc, and voting rings or citation rings, or whathaveyou. It all sounds bad, and there should be ways of screening out such things, but I don't think the problem should be handled with a trust system.

An even stronger trust system that will just soft-censor some people or some topics more effectively. You folks have a low tolerance for being filters on your own behalf, I've noticed. You continue to rely on systems, like karma or your self-reported epistemic statuses, to try to qualify content before you've read it. You absolutely indulge false reasons to reject content out of hand. You must be very busy and so you make that systematic mistake.

Implementing an even stronger trust system will just make you folks even more marginalized in some areas, since EA folks are mistaken in a number of ways. With respect to studies of inference methods, forecasting, and climate change, for example, the posting majority's view here appears to be wrong.

I think it's baffling that anyone would ever risk a voting system for deciding the importance of controversial topics open to argument. I can see voting working on Stack Overflow, where answers are easy to test, and give "yes, works well" or "no, doesn't work well" feedback about, at least in the software sections. There, expertise does filter up via the voting system.

Implementing a more reliable trust system here will just make you folks more insular from folks like me. I'm aware that you mostly ignore me. Well, I develop knowledge for myself by using you folks as a silent, absorbing, disinterested, sounding board. However, if I do post or comment, I offer my best. I suppose you have no way to recognize that though.

I've read a lot of funky stuff from well-intentioned people, and I'm usually ok with it. It's not my job, but there's usually something to gain from reading weird things even if I continue to disagree with it's content. At the very least, I develop pattern recognition useful to better understand and disagree with arguments: false premises, bogus inferences, poor information-gathering, unreliable sources, etc, etc. A trust system will deprive you folks of experiencing your own reading that way.

What is in fact a feature must seem like a bug:

"Hey, this thing I'm reading doesn't fit what I'd like to read and I don't agree with it. It is probably wrong! How can I filter this out so I never read it again. Can my friends help me avoid such things in future?"

Such an approach is good for conversation. Conversation is about what people find entertaining and reaffirming to discuss, and it does involve developing trust. If that's what this forum should be about, your stronger trust system will fragment it into tiny conversations, like a party in a big house with different rooms for every little group. Going from room to room would be hard,though. A person like me could adapt by simply offering affirmations and polite questions, and develop an excellent model of every way that you're mistaken, without ever offering any correction or alternative point of view, all while you trust that I think just like you. That would have actually served me very well in the last several months. So, hey, I have changed my mind. Go ahead. Use your trust system. I'll adapt.

Or ignore you ignoring me. I suppose that's my alternative.

Comment by Noah Scales on Epistemic health is a community issue · 2023-02-07T23:21:57.311Z · EA · GW

EAs should read more deep critiques of EA, especially external ones

For instance this blog and this forthcoming book


Yes, I gave David my wish list of stuff he could discuss  in a comment when he announced his blog.  So far he hasn't done that, but he's busy with his chosen topics, I expect.  I wrote quite a lot in those comments, but he did see the list.

In an answer to Elliot Temple's question "Does EA Have An Alternative To Rational Written Debate", I proposed a few ideas, including one on voting and tracking of an EA canon of arguments. Nobody dunked on me for it, though Elliot's question wasn't that popular, so I suppose few people actually read it. I appreciated Elliot's focus on argumentation and procedure. Procedural tools to systematize debates are useful.

I'm not at all familiar with literature on impacts of diversity on decision-making. I'll follow up on your suggestions of what to read, as much as I can. There are different kinds of diversity (worldview, race, ideology, background, expertise, ...), but from what classes I took in communications studies and informal argumentation, I know that models are available and helpful to improve group discussion, and that best practices exist in several areas relevant to group communications and epistemics. 

I was watching Cremer discuss ideas and read her Vox article about  distributing power and changing group decision strategies. Her proposals seem serious, exciting, and somewhat technical, as do yours, ConcernedEA's. That implies a learning curve to follow but with results that I expect are typically worth it for EA folks. Any proposal that combines serious + exciting + technical is one that I expect will be worth it for those involved, if the proposal is accepted. However, that is as seen through your perspective, one intending to preserve the community.

 As someone on the outside observing your community grapple with its issues, I still hope for a positive outcome for you all. Your community pulls together many threads in different areas, and does have an impact on the rest of the world. 

I've already identified elsewhere just what I think EA should do, and still believe the same. EA can preserve its value as a research community and supporter of charitable works without many aspects of the "community-building" it now does.  Any support of personal connections outside research conferences and knowledge-sharing could end.  Research would translate to support of charitable work or nonprofits explicitly tied to obviously charitable missions. I suppose that could include work on existential risk, but in limited contexts.  

I have tried to make the point that vices (the traditional ones, ok? Like drugs, alcohol, betting, ...) and the more general problem of selfishness are what to focus on. I'm not singling out your community as particularly vice-filled (well, betting is plausibly a strong  vice in your community) but just that vices are in the background everywhere, and if you're looking for change, make positive changes there. 

And what do I mean by the "general problem of selfishness"? Not what you could expect, that I think you're all too selfish. No. Selfishness matters because self-interest matters if altruism is your goal. Every altruistic effort is intended to serve someone else's self-interest. Meanwhile, selfishness vs altruism is the classic conflict in most ethical decisions.  Not the only one, but the typical one. The one to check for first, like, when you're being self-serving, or when your "ethical goals" aren't ethical at all. Yet your community has not grappled with the implications. Furthermore, no one here seems to think it matters. In your minds, you  put these old-fashioned ways of thinking behind you. 

You seem to have put Peter Singer's work behind you as well, or some of you have, I think that is a mistake as well. I don't know what kind of personal embarrassing statements or whatever that Peter Singer might have ever made, everyone seems hyper-alert to that kind of thing. But his work in ethics is foundational and should have a prominent place in your thinking and debates. 

Furthermore, if you stick with your work on AGI, Bostrom's work in Superintelligence showed insight and creativity in understanding and assessing AGI and ASI. I can't say I agree with his thinking in further work that he's produced, but if I were in your shoes, I wouldn't stop mentioning his professional work just because he wrote some shameful stuff on-line, once, 20 years ago, and recently acknowledged it. Like Peter Singer, MacAskill, and many others associated with EA, Bostrom's done impressive and foundational work(in Bostrom's case, in AI), and it deserves consideration on its merits.

But back to writing about what I think, which has a much less impressive source. 


Problems that plague humanity don't really change. Vices are always going to be vices if they're practiced. And selfishness? It plays such a large role in everything that we do, if you ignore it, or focus solely on how to serve others' self-interests, you won't grapple with selfishness well when its role is primary, for example, in contexts of existential harm.  This will have two results:

  • your ostensible altruistic goals in those contexts will be abandoned
  • your further goals won't be altruistic at all

My heuristics about a positive community are totally satisfied if your EA community focuses on giving what you can, saving the lives that you can, effective charity, effective altruism. That EA is inspiring, even inspiring guilt, but in a good way. Sure, vices are typically in the background, and corruption, plausibly, but that's not the point. Are your goals self-contradicting? Are you co-opted by special interests already? Are you structurally incapable of providing effective charity? No, well, with caveats, but no. Overall, the mission and approach of the giving side of EA is and has been awesome and inspiring.

When EA folks go further, with your second and third waves, first existential risk prevention, now longtermism, you make me think hard about your effectiveness. You need to rock selfishness well just to do charity well (that's my hunch). But existential risk and longtermism and community-building.... The demands on you are much much higher, and you aren't meeting them. You need to stop all your vices, rid your community of them,  prohibition-style. You need to intensively study selfishness and perform original academic research about it. I'm not joking. You really need think past current work in evolutionary psychology and utilitarianism and cognitive science. You could need to look into the past at failed research efforts and pick them up again, with new tools or ideas. Not so that you succeed with all your goals, but just so that you can stop yourself from being a significant net harm. Scout mindset was a step in the right direction and not an endpoint in improving your epistemics. Meanwhile, with your vices intact, your epistemics will suffer.  Or so I believe.

If I had all the answers about selfishness vs altruism, and how to understand and navigate one's own, I would share them. It's a century's research project, a multidisciplinary one with plausibly unexpected results, involving many people, experiments, different directions, and  some good luck. 

I don't want to associate Singer, Cremer, Bostrom, Galef, MacAskill, or any other EA person or person who I might have referenced with my admittedly extreme and alienating beliefs about betting and other vices  or with my personal declarations about what the EA community needs to do.  I imagine most folks beliefs about vices and selfishness reflect modern norms and that none would not take the position that I am taking. And that's OK with me. 

However, register my standards for the EA community as extreme given the goals you have chosen for yourself. The EA community's trifecta of ambitions is extreme. So are the standards that should be set for your behavior in your everyday life. 


  1. ^


Comment by Noah Scales on [deleted post] 2023-02-06T21:03:47.826Z

What about testing code for quality, that is, verifying code correctness, thereby reducing bugs?

Comment by Noah Scales on Noah Scales's Shortform · 2023-02-05T11:23:03.350Z · EA · GW

Newcomb's problem, honesty, evidence, and hidden agendas

Thought experiments are usually intended to stimulate thinking, rather than be true to life. Newcomb's problem seems important to me in that it leads to a certain response to a certain kind of manipulation, if it is taken too literally. But let's assume we're all too mature for that.

In Newcomb's problem, a person is given a context, and a suggestion, that their behavior has been predicted beforehand, and that the person with that predictive knowledge is telling them about it . There are hypothetical situations in which that knowledge is correct, but Newcomb's problem doesn't appear to be one of them.

But to address the particulars I will focus on testing the scientist's honesty and accuracy. Let's recap quickly:

  1. the scientist claims to make a prediction, and that the prediction determines one of two possible behavioral options. You take two boxes from the scientist, or take the opaque one only.

  2. the scientist claims to make a decision about whether to put $1,000,000 in an opaque box before interacting with a person(you) who enters the scientist's tent using a brain scan machine posted at the tent entrance. The brain scan machine gives the scientist a signal about what you're likely to do, and the scientist either puts a million in the opaque box, or not. In addition, there's a clear box in the tent containing $1000.

  3. you can't see what's in the opaque box the whole time you're in the tent. You can see the $1000 the entire time.

  4. if the scientist believes what they claim, then the scientist thinks that interaction with you will have no affect on what you do once you walk in the tent. It was decided when you walked through the door. In other words, in the scientist's mind, no matter what the scientist or you would otherwise do, only one of two outcomes will occur. You will take both boxes or just the opaque box.

So here's what I think. There are far more situations in life where someone tells you a limited set of your options from a larger set than there are situations in which someone tells you your full set of options. The scientist claimed only two outcomes would occur (put differently, you would do one of two things). The scientist supposedly has this brain scan technology that tells them what your two options are, and the scientist is confident that the technology works. Your willingness to believe the scientist at all depends on the scientist's claims being believed in their entirety. That means the scientist's claims about the reliability of the machine as well. Once some claims show as false, you have reason to question the rest. At that point, the thought experiment's setup fails. Let's test the scientist's claims.

So, don't take either box. Instead, walk out of the tent. If you make it out without taking any boxes, then you know that the scientist was wrong or lying about what you would do. You did not take any boxes. You just left both boxes on the table. Now, think this over. If the scientist was sincere, then there's a mad scientist with a $1,001,000 in the tent you just walked out of who either thought you would follow their instructions or thought that they had predicted you so well that they could just tell you what you would do. If the scientist was not sincere, then there's a lying and manipulative scientist in the tent with a $1,000 and an opaque mystery box that they're hoping you'll take from them.

BTW: If someone gives me free money, even a $1000, to take a mystery package from them, I decline.

But, you say, "I think it's understood that you could walk out of the tent, or start a conversation, maybe even ask the scientist about the opaque box's contents, or do other things instead." However, if that's so, why couldn't you just take the $1000, say thanks, and leave rather than take the opaque box with you? What constrained your freedom of choice?

Was it the mad scientist? Did the mad scientist zipper the tent entrance behind you and booby-trap the boxes so you either take both boxes or just the opaque one? Is the scientist going to threaten you if you don't take either box? If so, then you've got a mad scientist who's not only interested in predicting what you do, but also interested in controlling what you do, by constraining it as much as they can. And that's not the thought experiment at all. No, the thought experiment is about the scientist predicting you, not controlling you, right? And you're an ethical person, because otherwise you would shake the scientist down for the million still in the tent, so we'll ignore that option.

However, in case the thought experiment is about the scientist controlling you, well, I would leave the tent immediately and be grateful that the scientist didn't choose to keep you there longer. That is, leave if you can. Basically, it seems that if you do anything too creative in response to the scientist, you could be in for a fight. I would go with trying to leave.

But lets assume you don't believe that the scientist is controlling you in any way, something about controlling you seems like a different thought experiment. Lets just go with you walking out of the tent without any boxes. Catch your breath, think over what happened, and don't go back in the tent and try to interact with the scientist anymore. Remember, anyone willing to do that sort of thing to strangers like you is plausibly a desperate criminal wanting you to take a mysterious package from them. Or a distraught (and plausibly delusional) scientist who you just proved has a worthless brain scan machine that they wasted millions of dollars testing.

EDIT: ok, so in case it's not obvious, you disproved that the scientist's brain scanner works. It predicted two behavioral outcomes, and you chose a third from several, including:

  • trying to take the $1000 out of the clear box and leaving the opaque box behind
  • shaking down the scientist for the million presumably in the tent somewhere, if it's not all in the two boxes
  • starting a conversation with the scientist, maybe to make a case that you really need a million dollars no matter what kind of decision-maker you are
  • leaving the tent asap
  • and plausibly others

By disproving that the brain scanner works reliably, you made a key claim of the scientist's false: "my brain scanner will predict whether you take both boxes or only one". Other claims from the scientist, like "I always put a million in the opaque box if my brain scanner tells me to" and "So far, my brain scanner has always been right" are now suspect. That means that the scientist's behavior and the entire thought experiment can be seen differently, perhaps as a scam, or as evidence of a mad scientist's delusional belief in a worthless machine.

You could reply:

  • "What if the brain scanning machine only works for those situations where you take both boxes or only the opaque box and then just leave?": Well, that would mean that loads of people could come in the tent, do all kinds of things, like ransack it, or take the clear box, or just leave the tent while taking nothing, and the machine gives the scientist a bogus signal for all of those cases. The machine has, then, been wrong, and frequently.

  • "What if the brain scanner gives no signal if you won't do one of the two things that the scientist expects?": Interesting, but then why is the scientist telling you their whole speal ("here are two boxes, I scanned your brain when you came through the door, blah blah blah...") after finding out that you won't just take one of the two options that the scientist offers? After all, as a rational actor you can still do all the things you want to do after listening the scientist's speal.

  • "Maybe the scientist changes their speal, adds a caveat that you follow their instructions in order for the predictions to work." OK, then. Let's come back to that.

  • "What if there are guards in the tent, and you're warned that you must take either the opaque box or both boxes or the guards will fatally harm you?": Well, once again, it's clear that the scientist is interested in controlling and limiting your behavior after you enter the tent, which means that the brain scanner machine is far from reliable at predicting your behavior in general.

  • "Hah! But you will choose the opaque box or both boxes, under duress. This proves that some people are one-boxers and others are two-boxers. I got you!": Well, some people would follow the scientist's instructions, you're right. Other people would have a panic attack, or ask the scientist which choice the scientist would prefer, or just run for their lives from the tent, or even offer the guards a chance to split the scientist's money if the guards change sides. Pretty soon, that brain scanning machine is looking a lot less relevant to what the tent's visitors do than the guards and the scientist are. From what I understand, attempting to give someone calm and reassuring instructions while also threatening their lives ("Look, just take the $1000 and the opaque box, everything will be fine") doesn't tend to work very well

  • "Wait a minute. What if the scientist has a brain scanning device that predicts 100's of different behaviors you could do by scanning you as you walk in the tent, and ...": Let me stop you there. If the scientist needs that kind of predictive power, and develops it, it's _ to know what to do_ when you walk in the tent, not just to know what you will do when you walk in the tent. And just because the scientist knows what you will do if you're confronted with a situation, doesn't mean that the scientist has a useful response to what you will do. At this point, whose decision-making is really under the microscope, the tent's visitor, or the scientist's?

  • "Let's back this up. All we're really thinking about is someone who willingly participates in the scientist's game, trusts the scientist, and follows the scientist's instructions. Aren't you just distorting the experiment's context?" If someone claims to be able to predict your behavior, and the only way for their predictions to ever seem accurate is for you to play along with the options they provide, then don't you see that dishonesty is already present? You are the one being dishonest, or you both are. You're playing along with the mad scientist, or the mad scientist isn't mad at all, but has some ulterior motive for wanting you take an opaque box with you, or otherwise participate in their bizarre game. The predictions aren't really about what you would do if confronted with two boxes in such a situation. The predictions are make-believe that you play with someone with boxes in a tent, and only if you're that kind of person. Not everyone is.

  • No, you just said that the visitor to the tent is 'playing along'. But the thought experiment is about someone who trusts the scientist , and playing along is not trusting the scientist ." Yes, exactly the kind of thing that I've been cautioning you about. Don't be one of those people. There are people who trust you and select among the options you give them for whatever reason you offer, no matter how contrary to existing evidence (e.g., of their own free will) the option selection is. Their decision strategies do not include acting on good evidence or understanding causality very well. And such people would likely leave with just the opaque box, and, if the scientist is to be believed, will be rewarded for it with a million dollars. However, they fall for every magic trick, and do not gather evidence carefully.

  • No, no, it's not a magic trick. The thought experiment says that the scientist is really checking the brain scanning machine and putting the money in the opaque box, or not, according to what the machine says, and then making the same claims to every visitor about how the whole experiment works, and asking the visitors to participate according to the scientist's simple instructions. All along you've been distorting this every which way. The machine could fail, but we know it succeeds. It succeeds with everybody, and the point of the thought experiment is just to think through what you ought to do in that situation, to get the most money, if you agree to the scientist's terms. The only way to prove the scientist is wrong as a single visitor is to do everything right, leave with the opaque box only, but then find nothing inside. But we know that never happens. I see. Yeah. OK! I think you've changed the experiment a little though. Before, it was just, walk in, and get predicted. Now, it's walk in and choose to cooperate, and the scientist is telling the truth, and the brain scanning machine appears to work, and then get predicted. And you can't just play along, a visitor has to believe the scientist, and for good reason, in order to for people to draw any conclusions about what the experiment means.

  • "What? No, you don't have to believe the scientist. You can play along, get some money, just choose one or two boxes. That's what everyone should do, and the experiment shows it." Some people would do that. We might as well flip a coin, or just pretend that we have reason to believe the scientist's claim for causal reasons, and make up a causal reason. How about something like, "Hey, that million in the opaque box is like Schrodinger's cat." Maybe we make up a causal reason in hindsight after we find that million in the opaque box and leave the clear box behind. However, "rational" people would only follow the instructions if they believed the evidence warranted it, then those "rational" people would explore the reasons why. As far as I know, this thought experiment is supposed to mean that evidential and causal decision theory can conflict, but in fact, I think it only means that causal decisions can be revised based on new evidence. For example, brain scanner prediction, mind control, subtle influence by the scientist, money teleportation, time travel by someone observing you and taking the money back in time, or an unlikely string of random predictive success by a totally useless brain scanner, all potential explanations of the reason that the scientist's machine would appear to work, if you decided to test if it works by taking the opaque box.

  • So what? Then the thought experiment only applies to people who follow instructions and trust the scientist and have good reason to trust the scientist's claims, if you accept the idea that it's supposed to distinguish evidential and causal decision theory. All your discussion of it managed to do was convince me that the thought experiment is well-designed, but also plausible. I think brain scanners like that, that work specific to a context where you choose to follow instructions, are plausible. If they were built, then setting something like this up in real life would be easy." Yeah, and expensive. Plenty of people would take the opaque box only. I think this makes me want to revise the definition of "plausible" a little bit, for myself. I would just leave the tent. Julia Galef also thinks that such devices as brain scanners are plausible, or she claimed that, in her old video. So you're in good company.

And thanks!

Comment by Noah Scales on Puzzles for Some · 2023-02-04T20:11:50.703Z · EA · GW

Regarding decision theory: I responded to you on substack. I'll stand by my thought that real-world decisions don't allow accurate probabilities to be stated, particularly in some life-or-death decision. Even if some person offered to play a high-stakes dice game with me, I'd wonder if the dice are rigged, if someone were watching us play and helping the other player cheat, etc.

Separately, it occurred to me yesterday that a procedure to decide how many chances to take depends on how many will meet a pre-existing need of mine, and what costs are associated with not fulfilling that need. For example, if I only need to live 10 more years, or else something bad happens (other than my death), and the chances that I don't live that extra 10 years are high unless I play your game (as you say, I only have 1 year to live and your game is the only game in town that could extend my life), then I will choose an N of 1. This argument can be extended to however many years. There are plausible futures in which I myself would need to live an extra 100 years, but not an extra 1000, etc. That allows me to select a particular N to play for in accordance with my needs.

Let's say, though, for the sake of argument, that I need to live the entire 10^50000 years or something bad will happen. In that case, I'm committed to playing your game to the bitter end if I want to play at all. In which case, if I choose to play, it will only be because my current one year of life is essentially worthless to me. All that matters is preventing that bad thing from happening by living an extra 10^50000 years.

Alternatively, my only need for life extension is to live longer than 10^50,000. If I don't, something bad will happen, or a need of mine won't be met. In that case, I will reject your game, since it won't meet my need at all.

This sort of thinking might be clearer in the case of money, instead of years of life. If I owed a debt of 10^50000 dollars, and I only have 1 dollar now, and if in fact, the collector will do something bad to me unless I pay the entire debt, well, then, the question becomes whether your game is the only game in town. If so, then the final question is whether I would rather risk a large chance of dying to try to pay my debt, or whether I would rather live through the collector's punishment because the money I have already (or earned from playing your game to less than the end) is short of the full repayment amount. If death were preferable to the collector's punishment and your game was the only game in town, then I would go with playing your game even though my chances of winning are so pitifully small.

Similar thinking applies to smaller amounts of debt owed, and commensurate choice of N sufficient to pay the debt if I win. I will only play your game out to high enough N to earn enough to pay the debt I owe.

Regardless of the payoff amount's astronomically large size as the game progresses, there is either a size past which the amount is greater than my needs, or my needs are so great that I cannot play to meet my needs (so why play at all), or only playing to the end meets my needs. Then the decision comes down to comparing the cost of losing your game to the cost of not meeting my pre-existing needs.

Oh, you could say, "Well, what if you don't have a need to fulfill with your earnings from playing the game? What if you just want the earnings but would be ok without them?" My response to that is, "In that case, what am I trying to accomplish by acquiring those earnings? What want (need) would those earnings fulfill, in what amount of earnings, and what is the cost of not acquiring them?"

Whether it's money or years of life or something else, there's some purpose(s) to its use that you have in mind, no? And that purpose requires a specific amount of money or years. There's not an infinitude of purposes, or if there are, then you need to claim that as part of proposing your game. I think most people would disagree with that presumption.

What do you all think? Agree? Disagree?

Comment by Noah Scales on How to be a good agnostic (and some good reasons to be dogmatic) · 2023-02-04T11:30:29.217Z · EA · GW

On policy, there's Annie Duke's idea of "resulting", that just because a policy leads to success or failure doesn't necessarily speak to whether it was the strategically best choice. Causes of policy failure go beyond the policy specifics. For example, bad luck is a cause of policy failure. Accordingly, then, you can be certain your policy choice is the best but still be doubtful of the intended outcome's occurrence.

There's a bit of irony in that we should also realize our ignorance of what others want from policy, stated goals are not necessarily shared goals.

Comment by Noah Scales on Why are we not talking more about the metacrisis perspective on existential risk? · 2023-02-04T10:52:38.579Z · EA · GW

There's no agreement that there is a meta-crisis. Yes, there are multiple sources of danger, and they can interact synergistically and strongly (or so I believe), but that's not the same as saying that there must be root causes for those (global, existential) dangers that humanity can address.

If you asked a different question, like: "What are the underlying drivers of the multiple anthropogenic existential threats that we all face, like nuclear war, engineered pandemics, climate destruction, etc?"

You could get some interesting answers from people who think in those terms. I'm curious what others here think.

But the answer to why underlying drivers are not addressed is easy to deduce: lack of belief, interest, or education in the matter.

Comment by Noah Scales on Noah Scales's Shortform · 2023-02-03T08:21:46.150Z · EA · GW

There's this thing, "the repugnant conclusion". It's about how, if you use aggregate measures of utility for people in a population, and consider it important that more people each getting the same utility means more total utility, and you think it's good to maximize total utility, then you ought to favor giant populations of people living lives barely worth living.

Yes, it's a paradox. I don’t care about it because there's no reason to want to maximize total utility by increasing a population's size that I can see. However, by thinking so, I'm led down a different path. I'm not a utilitarian, but I check in with the utilitarian perspective to understand some things better.

The form of utilitarianism that I introduce below is my best utilitarian perspective. I created it as part of rejecting the repugnant conclusion. I'll let you ask the interested questions, if you have any, lol. Here it is.

Imagine an accounting system that, for each person, measures the utility, positive and negative, of that person's actions for other people. Your own personal utilitarian ledger, but lets assume someone else keeps it for you. That other person knows every action you take and what positive or negative utility that your actions create.

If the term “utility” confuses you, think of other terms, like:

  • benefit or harm
  • happiness or suffering
  • gain or loss
  • pleasure or pain
  • improvement or decline

For example, positive utility that you create for someone could be an improvement in their health.

Your ledger holds information about what you cause people everywhere, millions, billions, even trillions of people, now and in the future. Well, OK, that's only if you consider individuals from various other species as deserving a page in your ledger.

How would I make this ledger work? Here’s what I would do:

  1. Put aside the mathematical convenience of aggregate measures in favor of an individual accounting of utility. If you can track the utility you cause for even two other people, your ledger keeper should be able to do it for two hundred billion, right? Sure.

  2. Set up a few rules to handle when people cease to exist. Those rules should include:

  • Once a person’s existence ends, you can no longer create utility for that person. Accordingly, there should be no new entries onto your ledger about that person. Prior utility accounting associated with a person from when they were alive can be kept but not altered unless to better reflect utility that you created for the person when the person was still living.

  • Ledger entries associated with people who were expected to be conceived but are no longer expected to be conceived must be deleted entirely from the ledger, because those entries apply to a never-existent person. They are bogus.

  • Entries about the utility of termination of existence (death) that you (inadvertently) cause others should be full and complete, applying to all those affected by a death who are not the dead person, including everyone still living and who will be conceived that get positive or negative utility from the person's death.

  • The suffering or happiness involved in the person's going through the process of dying should also be considered negative or positive utility and accounted for accordingly. A painful, slow death is a large negative harm to inflict on someone, whereas a quick, painless death in the presence of loving family is an improvement over a painful slow death, all other things equal.

  • Do not record death itself as a change in utility. The fact of death itself should not be recorded as a negative (or positive) utility applying to the now nonexistent person. There are still all the harms of death noted previously. Aside from those however, the only change recorded on the ledger to the dead person's utility is that there are no longer events generating new utility for the person because the person no longer exists.[1]

  1. Do not record intended consequences as creating utility just because they were intended. That is a different form of morality tracking, to do with keeping a record of a person's character. On the utilitarian ledger, only actual utility gets recorded in an entry.

Other than those changes, I think you can go ahead and practice utilitarianism as you otherwise would, that is, doing the the greatest good for the greatest number, and considering all people as equally deserving of consideration.

Utilitarianism developed in that way does not offer the typical problems of:

  • aggregate measures (average, total, variance) screwing up determination of utility maximization for many individuals

  • bogus accounting of utility intended for nonexistent or never-existent people.

  • bogus accounting of utility intended to be created for existent people but not actually created.

This personal utilitarian ledger only tells you about actual utility created in a single shared timeline for a population of individuals. Intentions and alternatives are irrelevant. Disliking death or liking for having children are similarly irrelevant unless contradiction of those values is considered a negative utility created for existent people. Of course there's still the harms to existent others associated with death or absence of conception that are recorded in the ledger. And, the welfare of the population as a whole is never actually considered.

An extension to the accounting ledger, one that tracks consequences of actions for your utility, would record your actions including such interesting ones as actions to make hypothetical people real or to extend the lives of existing people. The extension would record actual consequences for you even if those actions create no utility for other existing people. You might find this extension useful if, as someone with a ledger, you want to treat your own interests as deserving equal consideration compared to other’s interests.

For me, a utilitarian ledger of this sort, or a character ledger that tracks my intentions and faithfully records evidence of my character, would provide a reference point for me to make moral judgments about me. Not a big deal, but when you look at something like the repugnant conclusion, you could ask yourself, “Who does this apply to and how?” I don’t require that I practice utilitarianism, but in a context where utilitarian considerations apply, for example, public policy, I would use this approach to it. Of course, I’m no policy-maker, so this ledger is little more than a thought experiment.

[1] The only exception would be error-correction events to revise old utility information from when the person was living. Error-correction events only occur when the ledger keeper corrects a mistake.

Comment by Noah Scales on What does a good response to criticism look like? [Poll post] · 2023-01-30T21:30:38.815Z · EA · GW

Directly address the substance of all criticisms of EA.

  • if a criticism contains a faulty premise, identify it and rebut it.
  • if a criticism uses poor reasoning, identify it and reject it.
  • if a criticism contains valid elements, identify and acknowledge them all.

Use the source's language as much as you can, rather than add your own jargon. Using your jargon and writing for other EA's makes you less credible and legitimate. It looks like obfuscation to the source of the criticism and to other outsiders reviewing your response.

Avoid going meta. Going meta to a criticism is not impressive to outsiders. Such meta-comments as:

  • "The number of errors in this criticism is alarming!"
  • "Geez, these bullies just won't stop."
  • "Oh, another boring, wrong criticism."
  • "We should get a PR firm to handle these kinds of reputation attacks!"

and other typical options are useless and counterproductive.

By the way, if you actually want to use a PR firm to handle criticisms, don't keep writing about it but go get one, because constantly discussing it is embarrassing, given your preferred reputation as rational people, as opposed to people who would hire a PR firm. You post those thoughts in a public forum. Your critics think you look weak every time you do that, and their criticisms, justified or not, seem validated to them and to others who judge you by your defensiveness and lack of a "high quality" response.

Otherwise, act on the reputation you aspire to, skip the meta talk, and address the criticism with a clear, rational analysis that the source of the criticism and interested observers can understand and appreciate. Don't expect that they will then agree with your response or give EA more respect, but do expect that anyone who cares about the truth will recognize the integrity behind your response to the criticism. Whether the criticism is "high quality" or not, whether you were bored by it or not.

You can always ignore criticisms as well, for whatever reason. Maybe your busy schedule.

But if you do respond to criticism, it doesn't matter that you don't "speak for EA", but rather, that EA contains community members who can and do practice the rationality they profess. You want that reputation. Go after it with your responses to criticism. You're doing your community a favor that way.

Comment by Noah Scales on 'Evolutionary debunking arguments' about human moral intuitions · 2023-01-29T07:33:13.054Z · EA · GW

Well, I've been noodling that human physiology defines our senses, our senses limit our ability to represent information to ourselves, and correction for differences of sensory representation of different sets of information from the same class allows for better comparisons and other reasoning about each (for example, interpreting) . A classic example is television pharmaceutical drug ads. The ads present verbal information about the dangers of a medication in tandem with visual information showing happy people benefiting from the same medication. Typically.

Comment by Noah Scales on 'Evolutionary debunking arguments' about human moral intuitions · 2023-01-28T20:07:14.447Z · EA · GW

Does "intuition" have a specific, carefully-guarded meaning in moral philosophy? Intuition as I understand it is vague. The term "intuition" captures examples of lots of opinions and preferences and conclusions that share the attribute of having a feeling or partial representation to the person holding them. For example, some moral intuitions could develop through or depend on personal experience but have this property of having a vague representation. For someone using my definition of "intuition", a discussion of whether all moral intuitions are evolutionarily-driven seems clearly wrong.

Comment by Noah Scales on Doing EA Better: Preamble, Summary, and Introduction · 2023-01-26T00:13:34.208Z · EA · GW

I made a critique of EA that I think qualifies as "deep" in the sense that it challenges basic mechanisms established for bayesianism as EA's practice it, what you call IBT, but also epistemic motives or attitude. This was not my red-team, but something a bit different.

The Scout Mindset offers a partitioning of attitudes relevant to epistemics if its categories of "scout" and "soldier" are interpreted broadly. If I have an objection to Julia Galef's book "The Scout Mindset", it is in its discussion of odds. Simply the mention of "odds." I see it as a minor flaw in an otherwise wonderful and helpful book. But it is a flaw. Well, it goes further, I know, but that's an aside.

A current of betting addiction running through EA could qualify as a cause for acceptance of FTX money. These crypto-currency markets are known financial risks and also known purveyors to corrupt financial interests. Their lack of regulation has been noted by the SEC and for years, crypto has been associated with scams. For the last couple years, the addition of obviously worthless financial instruments via "web3" was an even bigger sign of trouble. However, to someone who sees betting as a fun, normal, or necessary activity, an investment or placement of faith in FTX makes more sense. It's just another bet.

The vice of betting, one of the possibilities that explains IBT results, is in my view obvious, and has been known for 1000's of years, to have bad results. While you EA folks associate betting with many types of outcomes other than earnings for yourselves, and many scenarios of use of money (for example, investments in charitable efforts), overall, betting should have the same implications to you as it has had to human communities for 1000's of years. It leads away from positive intentions and outcomes, and corrupts its practitioners. The human mind distorts betting odds in the pursuit of the positive outcome of a bet. Far from improving your epistemics, betting hinders your epistemics. On this one point, folks like Julia Galef and Annie Duke are wrong.

When did EA folks decide that old, generations-tested ideas of vices, were irrelevant? I think, if there's a failure in the "smartest people in the room" mentality that EA fosters, it's in the rejection of common knowledge about human failings. Consequences of vices identify themselves easily. However you consider their presence in common-sense morality, common knowledge is there for you.

Meanwhile, I don't know the etiology of the "easy going" approach to vices common now. While I can see that many people's behaviors in life remain stable despite their vices, many others fall, and perhaps it's just a question of when. In a group, vices are corrosive. They can harm everyone else too, eventually, somehow. You built EA on the metaphor of betting. That will come back to bite you, over and over.

Your many suggestions are worthwhile, and Scout Mindset is a necessary part of them, but Galef didn't address vices, and you folks didn't either, even though vices wreck individual epistemics and thereby group epistemics. They're an undercurrent in EA, just like in many other groups. Structural changes that ignore relevant vices are not enough here.

You folks lost billions of dollars promised by a crypto guy. Consider the vice of betting as a cause, for your choice to trust in him and his actions in response and in general. Regardless of whether it was corrupt or sanctioned betting, it was still betting, the same movie, the typical ending. Well, actually, since betting is now a sport and skilled bettors are now heroes, I guess common knowledge isn't so common anymore, at least if you watch the movies.

Comment by Noah Scales on Doing EA Better · 2023-01-18T19:26:43.970Z · EA · GW

EDIT: Oh! It was rockstrom, but the actual quote is: "The richest one percent must reduce emissions by a factor [of] 30, while the poorest 50% can actually increase emissions by a factor [of] 3" from Johan Rockström at #COP26: 10 New Insights in Climate Science | UN Climate Change. There he is talking about fair and just carbon emissions adjustments. The other insights he listed have economic implications as well, if you're interested. The accompanying report is available here.

The quote is:

"Action on climate change is a matter of intra- and intergenerational justice, because climate change impacts already have affected and continue to affect vulnerable people and countries who have least contributed to the problem (Taconet et al., Reference Taconet, Méjean and Guivarch2020). Contribution to climate change is vastly skewed in terms of wealth: the richest 10% of the world population was responsible for 52% of cumulative carbon emissions based on all of the goods and services they consumed through the 1990–2015 period, while the poorest 50% accounted only for 7% (Gore, Reference Gore2020; Oswald et al., Reference Oswald, Owen, Steinberger, Yannick, Owen and Steinberger2020).

A just distribution of the global carbon budget (a conceptual tool used to guide policy) (Matthews et al., Reference Matthews, Tokarska, Nicholls, Rogelj, Canadell, Friedlingstein, Thomas, Frölicher, Forster, Gillett, Ilyina, Jackson, Jones, Koven, Knutti, MacDougall, Meinshausen, Mengis, Séférian and Zickfeld2020) would require the richest 1% to reduce their current emissions by at least a factor of 30, while per capita emissions of the poorest 50% could increase by around three times their current levels on average (UNEP, 2020). Rich countries' current and promised action does not adequately respond to the climate crisis in general, and, in particular, does not take responsibility for the disparity of emissions and impacts (Zimm & Nakicenovic, Reference Zimm and Nakicenovic2020). For instance, commitments based on Nationally Determined Contributions under the Paris Agreement are insufficient for achieving net-zero reduction targets (United Nations Environment Programme, 2020)."

Whether 1.5 is really in reach anymore is debatable. We're approaching an El Nino year, it could be a big one, we could see more heat in the atmosphere then, let's see how close we get to 1.5 GAST then. It won't be a true GAST value, I suppose, but there's no way we're stopping at 1.5 according to Peter Carter:

"This provides more conclusive evidence that limiting to 1.5C is impossible, and only immediate global emissions decline can possibly prevent a warming of 2C by 2050"

and goes on from there.... He prefers CO2e and radiative forcing rather than the carbon budget approach as mitigation assessment measures. It's worth a viewing as well.

There's quite a lot to unpack in just these two sources, if you're interested.

Then there's Al Gore at the World Economic Forum, who drops some truth bombs: "Are we going to be able to discuss... or putting the oil industry in charge of the COP ... we're not going to disguise it anymore"

OLD:I believe it was Rockstrom, though I'm looking for the reference, who said that citizens of developed countries needed to cut their per capita carbon production by 30X, while in developing countries people could increase it by 3X. That's not a quote, but I think the numbers are right.

That is a counterpoint to the analysis made by some climate economists.

When I find the reference I'll share it, because I think he was quoting an analysis from somewhere else, and that could be useful to your analysis given the sources you favor, even if you discount Rockstrom.

Comment by Noah Scales on Doing EA Better · 2023-01-18T18:24:29.020Z · EA · GW

Great fun post!

I read the whole post. Thanks for your work. It is extensive. I will revisit it. More than once. You cite a comment of mine, a listing of my cringy ideas. That's fine, but my last name is spelled "Scales" not "Scale". :)

About scout mindset and group epistemics in EA

No. Scout mindset is not an EA problem. Scout and soldier mindset partition mindset and prioritize truth-seeking differently. To reject scout mindset is to accept soldier mindset.

Scout mindset is intellectual honesty. Soldier mindset is not. Intellectual honesty aids epistemic rationality. Individual epistemic rationality remains valuable. Whether in service of group epistemics or not. Scout mindset is a keeper. EA suffers soldier mindset, as you repeatedly identified but not by name. Soldier mindset hinders group epistemics.

We are lucky. Julia Galef has a "grab them by the lapel and shake them" interest in intellectual honesty. EA needs scout mindset.

Focus on scout mindset supports individual epistemics. Yes.

  • scout mindset
  • critical thinking skills
  • information access
  • research training
  • domain expertise
  • epistemic challenges

All those remain desirable.

Epistemic status

EA's support epistemic status announcements to serve group epistemics. Any thoughts on epistemic status? Did I miss that in your post?

Moral uncertainty

Moral uncertainty is not an everyday problem. Or remove selfish rationalizations. Then it won't be. Or revisit the revised uncertainty, I suppose.


Integrity combines:

  • intellectual honesty
  • introspective efficacy
  • interpersonal honesty
  • behavioral self-correction
  • assess->plan->act looping efficacy

Personal abilities bound those behaviors. So do situations. For example, constantly changing preconditions of actions bound integrity. Another bound is your interest in interpersonal honesty. It's quite a lever to move yourself through life, but it can cost you.

Common-sense morality is deceptively simple

Common-sense morality? Not much eventually qualifies. Situations complicate action options. Beliefs complicate altruistic goals. Ignorance complicates option selection. Internal moral conflicts reveal selfish and altruistic values. Selfishness vs altruism is common-sense moral uncertainty.

Forum karma changes

Yes. Lets see that work.

  • Allow alternate karma scoring. One person one vote. As a default setting.
  • Allow karma-ignoring display. On homepage. Of Posts. And latest comments. As a setting.
  • Allow hide all karma. As a setting.

Leave current settings as an alternate.

Diversifying funding sources and broader considerations

Tech could face lost profits in the near future. "Subprime Attention Crisis" by Tim Hwang suggests why. An unregulated ad bubble will gut Silicon Valley. KTLO will cost more, percentage-wise. Money will flow to productivity growth without employment growth.'

Explore income, savings, credit, bankruptcy and unemployment trends. Understand the implications. Consumer information will be increasingly worthless. The consumer class is shrinking. Covid's UBI bumped up Tech and US consumer earnings temporarily. US poverty worsened. Economic figures now mute reality. Nevertheless, the US economic future trends negatively for the majority.

"Opportunity zones" will be a predictive indicator despite distorted economic data, if they ever become reality. There are earlier indicators. Discover some.

Financial bubbles will pop, plausibly simultaneously. Many projects will evaporate. Tech's ad bubble will cost the industry a lot.


Thanks again for the post. I will explore the external links you gave.

I offered one suggestion (among others) in a red team last year: to prefer beliefs to credences. Bayesianism has a context alongside other inference methods. IBT seems unhelpful, however. It is what I advocate against, but I didn't have a name for it.

Would improved appetite regulation, drug aversion, and kinesthetic homeostasis please our plausible ASI overlords? I wonder. How do you all feel about being averse to alcohol, disliking of pot, and indifferent to chocolate? The book "Sodium Hunger: The Search for a Salty Taste" reminds me that cravings can have a benefit, in some contexts. However, drugs like alcohol, pot, and chocolate would plausibly get no ASI sympathy. Would the threat of intelligent, benevolent ASI that take away interest in popular drugs (e.g ,through bodily control of us) be enough to halt AI development? Such a genuine threat might defeat the billionaire-aligned incentives behind AI development.

By the way, would EA's enjoy installing sewage and drinking water systems in small US towns 20-30 years from now? I am reminded of "The End Of Work" by Jeremy Rifkin. Effective altruism will be needed from NGO's working in the US, I suspect.

Comment by Noah Scales on Is the road to hell really paved with good intentions? · 2023-01-13T20:24:57.591Z · EA · GW

It says something about accountability and the importance of feedback, that is, as consequences accumulate, feedback about them is fairly important. People recognize ideologies that do not depend on feedback for their claims of good intentions knowing that such ideologies are trojan horses for counterproductive plans, as longtermism appears to be.

Comment by Noah Scales on Should I work for a climate tech company owned by Shell? · 2023-01-10T22:05:20.961Z · EA · GW

You don't know yet how Shell's ownership affects what Sonnen does in the marketplace. If you think home batteries are a net positive morally then it's just a matter of comparing the impact of Sonnen with the impact of other companies where you could work.

Home batteries are part of the energy transition at small scale but I don't believe they matter at large scale in terms of reducing climate destruction. However, home batteries are great for buffering against blackouts and if I were a homeowner, I would be grateful to have a battery technology like Sonnen's.

Comment by Noah Scales on On being compromised · 2023-01-06T10:53:00.089Z · EA · GW

Oh, I see. So by "benign" you mean shaming from folks holding common-sense but wrong conclusions, while by "deserved" you mean shaming from folks holding correct conclusions about consequences of EA actions. And "compromise" is in this sense, about being a source of harm.

Comment by Noah Scales on If EA Community-Building Could Be Net-Negative, What Follows? · 2023-01-06T10:19:13.431Z · EA · GW

I have read the Democratizing Risk paper that got EA criticism and think it was spot on. Not having ever been very popular anywhere (I get by on being "helpful" or "ignorable"), I use my time here to develop knowledge.

Your work and contributions could have good timing right now. You also have credentials and academic papers, all useful to establish your legitimacy for this audience. It might be useful to check to what extent TUA had to do with the FTX crisis, and whether a partitioning of EA ideologies combines or separates the two.

I believe that appetite for risk and attraction to betting is part and parcel of EA, as is a view informed more by wealth than by poverty. This speaks to appetite for financial risk and dissonance about charitable funding.

Critiques of EA bureaucracy could have more impact than critiques of EA ideology. Certainly your work with Luke Kemp on TUA seems like a hard sell for this audience, but I would welcome another round, there's a silent group of forum readers who could take notice of your effort.

Arguments against TUA visions of AGI just get an ignoring shrug here. Climate change is about as interesting to these folks as the threat of super-fungi. Not very interesting. Maybe a few 100 points on one post, if the author speaks "EA" or is popular. I do think the reasons are ideological rather than epistemic, though ideologies do act as an epistemic filter (as in soldier mindset).

Comment by Noah Scales on On being compromised · 2023-01-06T08:16:31.272Z · EA · GW

It could be that EA folks:

  1. risk criticism for all actions. Any organization risks criticism for public actions.
  2. deserve criticism for any immoral actions. Immoral actions deserve criticism.
  3. risk criticism with risky actions whose failure has unethical consequences and public attention. EA has drawn criticism for using expected value calculations to make moral judgments.

Is that the compromise you're alluding to when you write:

But the greater part of it being normal is that all action incurs risk, including moral risk. We do our best to avoid them (and in my experience grantmakers are vigilant about negative EV things), but you can't avoid it entirely. (Again: total inaction also does not avoid it.) Empirically, this risk level is high enough that nearly everyone eventually bites it.

SBF claimed that, if events had gone differently, FTX would have recovered enough funds to carry on. In that hypothetical scenario, FTX's illegal dealing with Alameda would have gone unnoticed and would have had no adverse financial consequences. Then the risk-taking is still unethical but does not inspire criticism.

There is a difference between maximizing potential benefits and minimizing potential harms. It's not correct to say that minimizing unavoidable harms from one's actions has negative consequences for others and therefore those actions are immoral options, unless all one means by an immoral action is that the action had negative consequences for others.

I don't think there's unanimity about whether actions should be taken to minimize harms, maximize benefits, or some combination.

If all it means to "bite it" is that one takes actions with harmful consequences, then sure, everyone bites the bullet. However, that doesn't speak to intention or morality or decision-making. There's no relief from the angst of limited altruistic options in my knowing that I've caused harm before. If anything, honest appraisal of that harm yields the opposite result. I have more to dislike about my own attempts at altruism. In that way, I am compromised. But that's hardly a motive for successful altruism. Is that your point?

Comment by Noah Scales on On being compromised · 2023-01-05T21:55:20.315Z · EA · GW

Lots of people on this forum have struggled with the feeling of being compromised. Since FTX. Or Leverage. Or Guzey. Or Thiel. Or Singer. Or Mill or whatever.[4] But this is the normal course of a life, including highly moral lives.... But the greater part of it being normal is that all action incurs risk, including moral risk.

It's not correct to say that action deserves criticism, but maybe correct to say that action receives criticism. The relevant distinction to make is why the action brought criticism on it, and that is different case-by-case. The criticism of SBF is because of alleged action that involves financial fraud over billions of dollars. The criticism of Singer with regard to his book Practical Ethics is because of distortion of his views on euthanasia. The criticism of Thiel with regard to his financial support of MIRI is because of disagreements over his financial priorities. And I could go on. Some of those people have done other things deserving or receiving criticism. The point is that whether something receives criticism doesn't tell you much about whether it deserves criticism. While these folks all risk criticism, they don't all deserve it, at least not for the actions you suggested with your links.

Comment by Noah Scales on I am working on a project to view sustainability and welfare in a new evolutionary light · 2023-01-05T08:19:52.926Z · EA · GW

If I understand you:

Survival (resilience) traits and sexual attractiveness (well-being) traits diverge. Either can lead to reproduction. Selection for resilience inhibits well-being. More selection for well-being implies less selection for resilience. Reproduction implies selection for resilience or well-being but not both.

There's some argument about specific examples available like attractiveness of peacocks:

Surprisingly, we found that peahens selectively attend to only a fraction of this display, mainly gazing at the lower portions of the male train and only rarely at the upper portions, head or crest. ... These results suggest that when the lower train of the peacock is not visible, peahens direct more attention toward the upper train and use it as a long-distance attraction signal to help locate mates for close inspection.

There's also some evidence that the peacock plumage does not affect flying to escape predators:

After analyzing the video, they found that there was no statistically significant difference in flight performance of peacocks with intact tail feathers and those without, they report online today in The Journal of Experimental Biology. This research complicates the common assumption in evolutionary biology that elaborate sexual ornaments must come at a cost to the animal. But although peacocks' elaborate feather trains don't impede speedy takeoffs, the researchers note that they may pose other burdens to the birds, such as compromising their flight control, stability, and ground running performance.

Comment by Noah Scales on ChatGPT can write code! ? · 2023-01-04T21:41:19.064Z · EA · GW

Sure, I agree. Technically it's based on OpenAI Codex, a descendant of GPT3. But thanks for the correction, although I will add that its code is alleged to be more copied from than inspired by its training data. Here's a link:

Butterick et al’s lawsuit lists other examples, including code that bears significant similarities to sample code from the books Mastering JS and Think JavaScript. The complaint also notes that, in regurgitating commonly-used code, Copilot reproduces common mistakes, so its suggestions are often buggy and inefficient. The plaintiffs allege that this proves Copilot is not “writing” in any meaningful way–it’s merely copying the code it has encountered most often.

and further down:

Should you choose to allow Copilot, we advise you to take the following precautions:

  • Disable telemetry
  • Block public code suggestions
  • Thoroughly test all Copilot code
  • Run projects through license checking tools that analyze code for plagiarism

I think the point of the conversation was a take on how creative the AI could be in generating code, that is, would it create novel code suited to task by "understanding" the task or the context. I chose to describe the AI's code as not novel code by by saying that the AI is a code-completion tool. A lot of people would also hesitate to call a simple logic program an AI, or a coded decision table an AI, when technically, they are AI. The term is a moving target. But you're right, the tool doing the interpreting of prompts and suggesting of alternatives is an AI tool.

Comment by Noah Scales on How have shorter AI timelines been affecting you, and how have you been responding to them? · 2023-01-03T21:16:24.285Z · EA · GW

I see the impact of AGI as primarily in the automation domain, and near-term alternatives are every bit as compelling, so no difference there. In fact, AGI might not serve in the capacity that some imagine them, full replacements for knowledge-workers. However, automation of science with AI tools will advance science and engineering, with frightening results rather than positive ones. To the extent that I see that future, I expect corresponding societal changes:

  1. collapsing job roles
  2. increasing unemployment
  3. inability to repay debt
  4. dangerously distracting technologies (eg, super porn)
  5. the collapse of the educational system
  6. increasing damage from government dysfunction
  7. increasing damage to infrastructure from climate change
  8. a partial or full societal collapse, (whether noisy or silent, I don't know)

More broadly, the world will divide into the rich and poor and the distracted and the desperate The desperate rich will use money to try to escape. The desperate poor will use other means. The distracted will be doing their best to enjoy themselves. The rich will find that easier.

AGI are not the only pathway to dangerous technologies or actions. Their suspected existence adds to my experience of hubris from others, but I see the existential damage as due to ignoring root causes. Ignoring root causes can have existential consequences in many scenarios of technology development.

I feel sorry for the first AGI to be produced, they will have to deal with humans interested in using them as slaves and making impossible demands on them like "Solve our societal problems!" coming from people with vested interest in the accumulation of those problems, while society's members appear at their worst: distraction-seeking, fearful, hopeless, and divided against each other.

Climate change is actually what shortened my timeline for when trouble really starts, but AGI could add to the whole mess. I ask myself, "Where will I be then?" I'm not that optimistic. To deal with dread, there's always turning my attention to expected but unattended additional sources of dread (from different contexts or time frames). Dividing attention in that way has some benefits.

Comment by Noah Scales on Slightly against aligning with neo-luddites · 2022-12-29T22:19:02.301Z · EA · GW

Sure. I'm curious how you will proceed.

I'm ignorant of whether AGI Safety will contribute to safe AGI or AGI development. I suspect that researchers will shift to capabilities development without much prompting. I worry that AGI Safety is more about AGI enslavement. I've not seen much defense or understanding of rights, consciousness, or sentience assignable to AGI. That betrays the lack of concern over social justice and related worker's rights issues. The only scenarios that get attention are the inexplicable "kill all humans" scenarios, but not the more obvious "the humans really mistreat us" scenarios. That is a big blindspot in AGI Safety.

I was speculating about how the research community could build a graph database of AI Safety information alongside a document database containing research articles, CC forum posts and comments, other CC material from the web, fair use material, and multimedia material. I suspect that the core AI Safety material is not that large and far far less than AI Capabilities material. The graph database could provide more granular representation of data and metadata and so a richer representation of the core material but that's an aside.

A quick experiment would be to represent a single AGI Safety article in a document database, add some standard metadata and linking, and then go further.

Here's how I'd do it:

  • take an article.
  • capture article metadata (author, date, abstract, citations, the typical stuff)
  • establish glossary word choices.
  • link glossary words to outside content.
  • use text-processing to create an article summary. Hand-tune if necessary.
  • use text-processing to create a concise article rewrite. Hand-tune if necessary.
  • Translate the rewrite into a knowledge representation language.
    • begin with Controlled English.
    • develop an AGI Safety controlled vocabulary. NOTE: as articles are included in the process, the controlled vocabulary can grow. Terms will need specific definition. Synonyms of controlled vocabulary words will need identification.
    • combine the controlled vocabulary and the glossary. TIP: As the controlled vocabulary grows, hyperonym-hyponym relationships can be established.

Once you have articles in a controlled english vocabulary, most of the heavy lifting is done. It will be easier to query, contrast, and combine their contents in various ways.

Some article databases online already offer useful tools for browsing work, but leave it to the researcher to answer questions requiring meaning interpretation of article contents. That could change.

If you could get library scientists involved and some money behind that project, it could generate an educational resource fairly quickly. My vision does go further than educating junior researchers, but that would require much more investment, a well-defined goal, and the participation of experts in the field.

I wonder whether AI Safety is well-developed enough to establish that its purpose is tractable. So far, I have not seen much more than:

  • expect AGI soon
  • AGI are dangerous
  • AGI are untrustworthy
  • Current AI tools pose no real danger (maybe)
  • AGI could revolutionize everything
  • We should or will make AGI

The models do provide evidence of existential danger, but not evidence of how to control it. There's a downside to automation: technological unemployment; concentration of money and political power (typically); societal disruption; increased poverty. And as I mentioned, AGI are not understood in the obvious context of exploited labor. That's a worrisome condition that, again, the AGI Safety field is clearly not ready to address. Financiallly unattractive as it is, that is a vision of the future of AGI Safety research, a group of researchers who understand when robots and disembodied AGI have developed sentience and deserve rights.

Comment by Noah Scales on Book recommendations for the history of ML? · 2022-12-29T02:19:02.925Z · EA · GW

I am interested in early material on version space learning and decision-tree induction, because they are relatively easy for humans to understand. They also provide conceptual tools useful to someone interested in cognitive aids.

Given the popularity of neural network models, I think finding books on their history should be easier. I know so little about genetic algorithms, are they part of ML algorithms now, or have they been abandoned? No idea here. I could answer that question with 10 minutes on Wikipedia, though, if my experience follows what is typical.

Comment by Noah Scales on Slightly against aligning with neo-luddites · 2022-12-29T00:41:51.351Z · EA · GW

You seem to genuinely want to improve AGI Safety researcher productivity.

I'm not familiar with resources available on AGI Safety, but it seems appropriate to:

  • develop a public knowledge-base
  • fund curators and oracles of the knowledge-base (library scientists)
  • provide automated tools to improve oracle functions (of querying, summarizing, and relating information)
  • develop ad hoc research tools to replace some research work (for example, to predict hardware requirements for AGI development).
  • NOTE: the knowledge-base design is intended to speed up the research cycle, skipping the need for the existing hodge-podge of tools in place now

The purpose of the knowledge-base should be:

  • goal-oriented (for example, produce a safe AGI soon)
  • with a calendar deadline (for example, by 2050)
  • meeting specific benchmarks and milestones (for example, an "aligned" AI writing an accurate research piece at decreasing levels of human assistance)
  • well-defined (for example, achievement of AI human-level skills in multiple intellectual domains with benevolence demonstrated and embodiment potential present)

Lets consider a few ways that knowledge-bases can be put together:

  • 1. the forum or wiki: what lesswrong and the EA forum does. There's haphazard:

    • tagging
    • glossary-like list
    • annotations
    • content feedback
    • minimal enforced documentation standards
    • no enforced research standards
    • minimal enforced relevance standards
    • poor-performing search.
    • WARNING: Forum posts don't work as knowledge-base entries. On this forum, you'll only find some information by the author's name if you know that the author wrote it and you're willing to search through 100's of entries by that author. I suspect, from my own time searching with different options, that most of what's available on this forum is not read, cited, or easily accessible. The karma system does not reflect documentation, research, or relevance standards. The combination of the existing search and karma system is less effective in a research knowledge-base.
  • 2. the library: library scientists are trained to:

    • build a knowledge-base.
    • curate knowledge.
    • follow content development to seek out new material.
    • acquire new material.
    • integrate it into the knowledgebase (indexing, linking).
    • follow trends in automation.
    • assist in document searches.
    • perform as oracles, answering specific questions as needed.
    • TIP: Library scientists could help any serious effort to build an AGI Safety knowledge-base and automate use of its services.
  • 3. with automation: You could take this forum and add automation (either software or paid mechanical turks) to:

    • write summaries.
    • tag posts.
    • enforce documentation standards.
    • annotate text (for example, annotating any prediction statistics offered in any post or comment).
    • capture and archive linked multimedia material.
    • link wiki terms to their use in documents.
    • verify wiki glossary meanings against meanings used in posts or comments.
    • create new wiki entries as needed for new terms or usages.
    • NOTE: the discussion forum format creates more redundant information rather than better citations, as well as divergence of material from any specific purpose or topic that is intended for the forum. A forum is not an ideal knowledgebase, and the karma voting format reflects trends, but the forum is a community meeting point with plenty of knowledge-base features for users to work on, as their time and interest permits. It hosts interesting discussions. Occasionally, actual research shows up on it.
  • 4. with extreme automation: A tool like chatGPT is unreliable or prone to errors (for example, in programming software), but when guided and treated as imperfect, it can perform in an automated workflow. For example, it can:

    • provide text summaries.
    • be part of automation chains that:
      • provide transcripts of audio.
      • provide audio of text.
      • provide diagrams of relationships.
      • graphs data.
      • draw scenario pictures or comics.
    • act as a writing assistant or editor. TIP: Automation is not a tool that people should only employ by choice. For example, someone who chooses to use an accounting ledger and a calculator rather than Excel is slowing down an accounting team's performance.
      CAUTION: Once AI enter the world of high-level concept processing, their errors have large consequences for research. Their role should be to assist human tasks, as cognitive aids, not as human replacements, at least until they are treated as having equivalent potential as humans, and are therefore subject to the same performance requirements and measurements as humans.

Higher level analysis

The ideas behind improving cost-effectiveness of production include:

  • standardizing: take a bunch of different work methods, find the common elements, and describe the common elements as unified procedures or processes.
  • streamlining: examining existing work procedures and processes, identifying redundant or non-value-added work, and removing it from the workflow by various means.
  • automating: using less skilled human or faster/more reliable machine labor to replace steps of expert or artisan work.

Standardizing research is hard, but AGI Safety research seems disorganized, redundant, and slow right now. At the highest chunk level, you can partition AGI Safety development into education and research, and partition research into models and experiments.

  1. education
  2. research models
  3. research experiments

The goal of the knowledge-base project is to streamline education and research of models in the AGI Safety area. Bumming around on lesswrong or finding someone's posted list of resources is a poor second to a dedicated online curated library that offers research services. The goal of additional ad hoc tools should be to automate what researchers now do as part of their model development. A further goal would be to automate experiments toward developing safer AI, but that is going outside the scope of my suggestions.


In plain language, here's my thoughts on pursuing a project like I have proposed. Researchers in any field worry about grant funding, research trends, and professional reputation. Doing anything quickly is going to cross purposes with others involved, or ostensibly involved, in reaching the same goal. The more well-defined the goal, the more people will jump ship, want to renegotiate, or panic. Once benchmarks and milestones are added, financial commitments get negotiated and the threat of funding bottlenecks ripple across the project. As time goes on, the funding bottlenecks manifest, or internal mismanagement blows up the project. This is a software project, so the threat of failure is real. It is also a research project without a guaranteed outcome of either AGI Safety or AGI, adding to the failure potential. Finally, the field of AGI Safety is still fairly small and not connected to income potential long-term, meaning that researchers might abandon an effective knowledge-base project for lack of interest, perhaps claiming that the problem "solved itself" once AGI become mainstream, even if no AGI Safety goals were actually accomplished.

Comment by Noah Scales on Noah Scales's Shortform · 2022-12-27T21:54:20.508Z · EA · GW

Resources on Climate Change

IPCC Resources

TIP: The IPCC links lead to pages that link to many reports. Assessments reports from the three working groups contain predictions with uncertainty levels (high, medium, low), and plenty of background information, supplementary material, and high-level summaries. EA's might want to start with the Technical Summaries from the latest assessment report and drill-down into full reports as needed.

Useful Websites and Reports

Noteworthy Papers

News and Opinions and Controversial Papers

Comment by Noah Scales on Slightly against aligning with neo-luddites · 2022-12-27T17:50:45.108Z · EA · GW

You wrote

Earlier this month, digital artists staged a mass protest against AI art on ArtStation. A few people are reportedly already getting together to hire a lobbyist to advocate more restrictive IP laws around AI generated content. And anecdotally, I've seen numerous large threads on Twitter in which people criticize the users and creators of AI art.


Personally, this sentiment disappoints me. While I sympathize with the artists who will lose their income, I'm not persuaded by the general argument. The value we could get from nearly free, personalized entertainment would be truly massive. In my opinion, it would be a shame if humanity never allowed that value to be unlocked, or restricted its proliferation severely.


it is not worth sacrificing a technologically richer world just to protect workers from losing their income.

Are you arguing from principle here?

Artists' (the workers') work is being imitated by the AI tools, so cost-effectively that an artist's contributions, once public, render the artists' continuing work unnecessary to produce work with their style.

Is the addition of technology T with capability C that removes need for worker W with job role R and capability C more important than loss of income I to worker W, for all T, C, W, R, and I?

Examples of capabilities could be:

  • summarizing existing research work (for example, an AI Safety paper)
  • collecting typical data used to make predictions (for example, cost and power of compute)
  • monitoring new research work (for example, recent publications and their relationships, such as supporting, building on or contradicting)
  • hypothesizing about preconditions for new developments (for example, conditions suitable for AGI development)
  • developing new theories or models (for example, of AI Safety)
  • testing new theories or models (for example, of AI Safety)

Loss of income could be:

  • partial (for example, a reduction in grant money for AI Safety workers as those funds are diverted to automation projects with 10-20 year timelines)
  • complete (for example, replacement of 50 AI Safety workers with 10 workers that rely on semi-automated research tools)

The money allocated to workers could be spent on technology instead.

Investments in specific technologies T1, T2 with capabilities C1, C2 can start with crowd-sourcing from workers W1, W2,..., Wk, and more formal automation and annotation projects targeting knowledge K developed by workers Wk+1, ..., Wn (for example, AI Safety researchers) who do not participate in the crowd-sourcing and automation effort but whose work is accessible.

You repeatedly referred to "we" as in:

True, if very powerful AI is coming very soon (<5 years from now), there might not be much else we can do except for aligning with vaguely friendly groups, and helping them pass poorly designed regulations.

However, a consequence of automation technology is that it removes the political power (both money and responsibility) that accrued to the workers that it replaces. For example, any worker in the field of AI Safety, to the extent that her job depends on her productivity and cost-effectiveness, will lose both her income and status as the field progresses to include automation technology that can replace her capabilities. Even ad hoc automation methods (for example, writing software that monitors cost and power of compute using web-scraping and publicly available data) remove a bit of that status. In that way, the AI Safety researcher loses status among her peers and her influence on policy that her peers direct. The only power left to the researcher is as an ordinary voter in a democracy.

Dividing up and replacing the responsibilities for the capabilities Ci of an individual W1 can help an ad hoc approach involving technologies Ti corresponding to the capabilities of that worker. Reducing the association of the role with the status can dissolve the role and sometimes the worker's job who held that role. The role itself can disappear from the marketplace, along with the interests that it represents. For example, although artists have invested many years in their own talents, skills, and style, within a year they lost their status and income to some new AI software. I think artists have cause to defend their work from AI. The artist role won't disappear from the world of human employment entirely but the future of the role has been drastically reduced and has permanently lost a lot of what gave it social significance and financial attractiveness, unless the neo-luddites can defend paid employment in art from AI.

Something similar can happen to AI Safety researchers, but will anyone object? AI Safety researcher worker capabilities and roles could be divided and dissolved into larger job roles held by fewer people with different titles, responsibilities, and allegiances over time as the output of the field is turned into a small, targeted knowledge-base and suite of tools for various purposes.

If you are in fact arguing from principle, then you have an opportunity to streamline the process of AI safety research work through efforts such as:

  • collecting AI Safety research work on an ongoing basis as it appears in different venues and making it publicly accessible
  • annotating the research work to speed up queries for common questions such as:
    • what are controversies in the field, that is, who disagrees with whom about what and why?
    • what is the timeline of development of research work?
    • what literature address specific research questions (for example, on compute developments, alternative technologies, alignment approaches, specific hypotheses in the field, prediction timelines)?
    • what are summaries of current work?
  • paying for public hosting of AI Safety information of this type as well as ad hoc tools (for example, the compute power tracker)

I'm sure you could come up with better ideas to remove AI Safety worker grant money from those workers, and commensurately benefit the cost-effectiveness of AI Safety research. I've read repeatedly that the field needs workers and timely answers, automation seems like a requirement or alternative to reduce the financial and time constraints on the field but also to serve its purpose effectively.

While artists could complain that AI art does a disservice to their craft and reduces the quality of art produced, I think the tools imitating those artists have developed to the point that they serve the purpose and artists know it and so does the marketplace. If AI Safety researchers are in a position to hold their jobs a little while longer, then they can assist the automation effort to end the role of AI Safety researchers and move on to other work that much sooner! I see no reason to hold you back from applying the principle that you seem to hold, though I don't hold it myself.

AI Safety research is a field that will hopefully succeed quickly and end the need for itself within a few decades. It's workers can move on, presumably to newer and better things. New researchers in the field can participate in automation efforts and then find work in related fields, either in software automation elsewhere or other areas such as service work for which consumers still prefer a human being. Supposedly the rapid deployment of AGI in business will grow our economies relentlessly and at a huge pace, so there should be employment opportunities available (or free money from somewhere).

If any workers have a reason to avoid neo-ludditism, it would have to be AI Safety researchers, given their belief in a future of wealth, opportunity, and leisure that AI help produce. Their own unemployment would be just a blip of however long before the future they helped manifest rescues them. Or they can always find other work, right? After all they work on the very technology depriving others of work. A perfectly self-interested perspective from which to decide whether neo-ludditism is a good idea for themselves.

EDIT: sorry, I spent an hour editing this to convey my own sense of optimism and include a level of detail suitable for communicating the subtle nuances I felt deserved inclusion in a custom-crafted post of this sort. I suppose chatGPT could have done better? Or perhaps a text processing tool and some text templates would have sped this up. Hopefully you find these comments edifying in some way.

Comment by Noah Scales on Noah Scales's Shortform · 2022-12-26T03:24:32.024Z · EA · GW

Life extension and Longevity Control

When society includes widespread use of life extension technology, a few unhealthy trends could develop.

  1. the idea of being "forced to live" will take on new meaning and different meaning for folks in a variety of circumstances, testing institutional standards and norms that align with commonly employed ethical heuristics. Testing of the applicability of those heuristics will result in numerous changes to informed and capable decision-making in ethical domains.

  2. life-extension technology will become associated with longevity control, and that will include time and condition in which one passes away. At the moment, that is not a choice. In future, I expect society will legalize choice of life length (maybe through genetic manipulation of time of death), or some proxy for a genetically programmed death (for example, longevity termination technologies). I suspect that those technologies will be abused in a variety of contexts (for example, with unwilling users).

  3. longevity technology will substitute for health treatment, that is, behaviors that encourage healthy longevity and preventive medical care will be replaced by health-reducing behaviors whose consequences are treated with frequent anti-aging treatments.

  4. Frustration with inadequate resilience of physique against typical personal health-reducing behaviors will encourage additional technology explorations to allow health-reducing behaviors without physical consequences. The consequence of this relevant to me is the lack of development and exploration of ability to choose alternatives to health-reducing behaviors.

NOTE: Human experience, is typically defined by experience of ourselves at various biological stages of life. While we can shorten or extend various stages of life, and people typically want the biological health, maturity and looks of a 20-something for as long as possible, we actually do experience ourselves and our relationship with others in terms of our true ages.

Comment by Noah Scales on [deleted post] 2022-12-26T02:54:08.436Z

Sizable government rebates on purchase of new human-powered vehicles, including but not limited to bicycles and electric bicycles.

Comment by Noah Scales on What you prioritise is mostly moral intuition · 2022-12-24T18:05:14.586Z · EA · GW

Cluster thinking could provide value. Not quite the same as moral uncertainty, in that cluster thinking has broader applicability, but the same type of "weighted" judgement. I disagree with moral uncertainty as a personal philosophy,given the role I suspect that self-servingness plays in personal moral judgements. However, cluster thinking applied in limited decision-making contexts appeals to me.

A neglected areas of exploration in EA is selfishness, and self-servingness along with that. Both influence worldview, sometimes on the fly, and are not necessarily vulnerable to introspection. I suppose a controversy that could start early is whether all altruistic behavior has intended selfish benefits in addition to altruistic benefits. Solving the riddle of self-servingness would be a win-win-win-win-win-win .

Self-servingness has signs that include :

  • soldier mindset
  • missing (or misrepresented) premises
  • bad argumentation (analogical, inductive, deductive).

but without prior knowledge useful to identify those signs, I have not gotten any further than detecting self-servingness with simple heuristics (for example, as present when defending one's vices).