Posts

Handling Moral Uncertainty with Average vs. Total Utilitarianism: One Method That Apparently *Doesn't* Work (But Seemed Like it Should) 2023-01-05T22:18:21.772Z
Why is "Argument Mapping" Not More Common in EA/Rationality (And What Objections Should I Address in a Post on the Topic?) 2022-12-23T21:55:56.410Z
How/When Should One Introduce AI Risk Arguments to People Unfamiliar With the Idea? 2022-08-09T02:57:31.685Z
Research + Reality Graphing to Support AI Policy (and more): Summary of a Frozen Project 2022-07-02T20:58:52.970Z
The COILS Framework for Decision Analysis: A Shortened Intro+Pitch 2022-05-07T19:01:33.239Z
Would Structured Discussion Platforms for EA Community Building Ideas be Valuable? (With Prototype Example) 2022-05-04T18:49:12.792Z
Would people like to see "curation comments" on posts with high numbers of comments? 2022-04-17T04:40:07.346Z
r/place is coming back on April 1st: EA pixel art, anyone? 2022-03-31T20:50:44.806Z
A dataset for AI/superintelligence stories and other media? 2022-03-29T21:41:26.750Z
How might a herd of interns help with AI or biosecurity research tasks/questions? 2022-03-20T22:49:11.613Z
EA Topic Suggestions for Research Mapping? 2022-03-05T16:46:52.835Z
"Epistemic maps" for AI Debates? (or for other issues) 2021-08-30T04:59:29.608Z
The TUILS/COILS Framework for Improving Pro-Con Analysis 2021-04-08T01:37:29.756Z
Harrison D's Shortform 2020-12-05T04:10:16.021Z
[Outdated] Introducing the Stock Issues Framework: The INT Framework's Cousin and an "Advanced" Cost-Benefit Analysis Framework 2020-10-03T07:18:54.045Z

Comments

Comment by Harrison Durland (Harrison D) on My attempt at explaining the case for AI risk in a straightforward way · 2023-03-26T18:48:08.611Z · EA · GW

Which issue are you referring to? (External credibility?) 

I don’t see a reason to not share the paper, although I will caveat that it definitely was a rushed job. https://docs.google.com/document/d/1ctTGcmbmjJlsTQHWXxQmhMNqtnVFRPz10rfCGTore7g/edit

Comment by Harrison Durland (Harrison D) on My attempt at explaining the case for AI risk in a straightforward way · 2023-03-26T03:45:44.969Z · EA · GW

I think that another major problem is simply that there is no one-size-fits-all intro guide. I think I saw some guides by Daniel Eth (or someone else?) and a few other people that were denser than the guide you’ve written here, and yeah the intro by Kelsey Piper is also quite good.

I’ve wondered if it could be possible/valuable to have a curated list of the best intros, and perhaps even to make a modular system, so people can customize better for specific contexts. (Or maybe having numerous good articles would be valuable if eventually someone wanted to and could use them as part of a language model prompt to help them write a guide tailored to a specific audience??)

Comment by Harrison Durland (Harrison D) on My attempt at explaining the case for AI risk in a straightforward way · 2023-03-25T19:08:30.143Z · EA · GW

FWIW, I just wrote a midterm paper in the format of a memo to the director of OSTP on basically this topic ("The potential growth of AI capabilities and catastrophic risks by 2045"). One of the most frustrating aspects of writing the paper was trying to find externally-credible sources for claims with which I was broadly familiar, rather than links to e.g., EA Forum posts. I think it's good to see the conceptual explainers, but in the future I would very much hope that a heavily time-constrained but AI-safety-conscious staffer can more easily find credible sources for details and arguments such as the analogy to human evolution, historical examples and analyses of discontinuities in technological progress, the scaling hypothesis and forecasts based heavily on compute trends (please, Open Phil, something other than a Google Doc! Like, can't we at least get an Arxiv link?), why alignment may be hard, etc. 

I guess my concern is that as the explainer guides proliferate, it can be harder to find the guides that actually emphasize/provide credible sources... This concern probably doesn't make new guides net negative, but think it could potentially be mitigated, maybe by having clear, up-front links to other explainers which do provide better sources. (Or if there were some spreadsheet/list of "here are credible sources for X claim, with multiple variant phrasings of X claim," that might also be nice...)

Comment by Harrison Durland (Harrison D) on Harrison D's Shortform · 2023-03-24T03:55:14.350Z · EA · GW

The following is a midterm assignment I submitted for my Cyber Operations class at Georgetown, regarding the risk of large AI model theft. I figured I would just publish this since it's fairly relevant to recent discussions and events around AI model theft. (I also am posting this so I have a non-Google-Doc link to share with people)

Note: In this assignment I had a 500-word limit and was only tasked to describe a problem's relevance to my client/audience while briefly mentioning policy options. In an upcoming memo assignment I will need to actually go into more detail on the policy recommendations (and I'd be happy to receive suggestions for what CISA should do if you have any).

(I also acknowledge that the recommendations I lay out here are a bit milquetoast, but I genuinely just didn't know what else to say...)

-------------

Memorandum for the Cybersecurity and Infrastructure Security Agency (CISA)

SUBJECT: Securing Large AI Models Against Theft

Large artificial intelligence (AI) models such as ChatGPT have increasingly demonstrated AI’s potential. However, as proprietary models become more powerful it is increasingly important to protect them against theft. CISA should work to facilitate information sharing that supports public policy and private responses. The following four sections will discuss some of the threat motivations/trends, potential consequences, market failures, and policy recommendations for CISA.

Motivations and Relevant Trends Regarding AI Model Theft

There are many reasons to expect that hackers will attempt to exfiltrate proprietary AI models:

  1. China and other actors have repeatedly stolen sensitive data and intellectual property (IP).[1]
  2. Future models may prove to have such significant economic or military value that state actors are willing to expend substantial effort/assets to steal them.
  3. Current large models have high up-front development (“training”) costs/requirements but comparatively low operational costs/requirements after training.[2] This makes theft of models attractive even for non-state actors. Additionally, recent export controls on semiconductors to China could undermine China’s ability to train future large models,[3] which would further increase Beijing’s incentive to support model theft.
  4. Someone reportedly leaked Meta’s new large language model (LLaMA) within days of Meta providing model access to researchers.[4]

 

Potential Consequences of AI Model Theft

Theft of powerful AI models—or the threat thereof—could have significant negative consequences beyond straightforward economic losses:

  1. Many powerful AI models could be abused: 
    1. Content generation models could enhance disinformation and spear phishing campaigns.[5]
    2. Image recognition models could empower semi-autonomous weapons or authoritarian surveillance.[6]
    3. Simulation models could facilitate the design of novel pathogens.[7]
  2. The mere threat of theft/leaks may discourage efforts to improve AI safety and interpretability that involve providing more access to powerful models.[8]
  3. Enhanced Chinese AI research could intensify AI racing dynamics that prove catastrophic if “very powerful systems”[9] are attainable over the next 15 years.[10]

 

Why Traditional Market Incentives May Fail to Mitigate These Risks

Many companies will have some incentives to protect their models, but there are some reasons to expect their efforts will be suboptimal relative to the risks:

  1. The risks described in the previous section are largely externalities and companies that do not appropriately guard against these risks may out-compete companies that do.
  2. Unauthorized use of models may be limited to foreign jurisdictions where the companies did not expect to make substantial profits (e.g., an off-limits Chinese economy).
  3. Market dynamics may disincentivize some prosocial actions such as cybersecurity incident disclosures.[11]

 

Suggestions for CISA

CISA should explore some options to inform and facilitate public policy and private responses to these threats:

  1. Map relevant actors and stakeholders.
  2. Evaluate and/or propose platforms and frameworks for information sharing.
  3. Assess the presence and impact of market failures.
  4. Collect research relevant to actions that other actors could take (e.g., programs at DARPA/IARPA,[12] mandatory incident disclosure legislation).
  5. Begin drafting a report which incorporates the previous suggestions and elicits input from relevant actors.

-------------

References

Allen, Gregory, Emily Benson, and William Reinsch. 2022. “Improved Export Controls Enforcement Technology Needed for U.S. National Security.” Center for Strategic and International Studies. November 30, 2022. https://www.csis.org/analysis/improved-export-controls-enforcement-technology-needed-us-national-security.

Brooks, Chuck. 2023. “Cybersecurity Trends & Statistics for 2023: More Treachery and Risk Ahead as Attack Surface and Hacker Capabilities Grow.” Forbes. March 5, 2023. https://www.forbes.com/sites/chuckbrooks/2023/03/05/cybersecurity-trends--statistics-for-2023-more-treachery-and-risk-ahead-as-attack-surface-and-hacker-capabilities-grow/?sh=2c6fcebf19db.

Calma, Justine. 2022. “AI Suggested 40,000 New Possible Chemical Weapons in Just Six Hours.” The Verge. March 17, 2022. https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx.

Cottier, Ben. 2022. “The Replication and Emulation of GPT-3.” Rethink Priorities. December 21, 2022. https://rethinkpriorities.org/publications/the-replication-and-emulation-of-gpt-3.

———. 2023. “Trends in the Dollar Training Cost of Machine Learning Systems.” Epoch. January 31, 2023. https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.

Cox, Joseph. 2023. “How I Broke into a Bank Account with an AI-Generated Voice.” Vice. February 23, 2023. https://www.vice.com/en/article/dy7axa/how-i-broke-into-a-bank-account-with-an-ai-generated-voice.

Dickson, Ben. 2020. “The GPT-3 Economy.” TechTalks. September 21, 2020. https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/.

Feldstein, Steven. 2019. “The Global Expansion of AI Surveillance.” Carnegie Endowment for International Peace. September 17, 2019. https://carnegieendowment.org/2019/09/17/global-expansion-of-ai-surveillance-pub-79847.

“Guaranteeing AI Robustness against Deception (GARD).” n.d. DARPA. Accessed March 11, 2023. https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception.

Humphreys, Brian. 2021. “Critical Infrastructure Policy: Information Sharing and Disclosure Requirements after the Colonial Pipeline Attack.” Congressional Research Service. May 24, 2021. https://crsreports.congress.gov/product/pdf/IN/IN11683.

Longpre, Shayne, Marcus Storm, and Rishi Shah. 2022. “Lethal Autonomous Weapons Systems & Artificial Intelligence: Trends, Challenges, and Policies.” Edited by Kevin McDermott. MIT Science Policy Review 3 (August): 47–56. https://doi.org/10.38105/spr.360apm5typ.

Nakashima, Ellen. 2015. “Chinese Breach Data of 4 Million Federal Workers.” The Washington Post, June 4, 2015. https://www.washingtonpost.com/world/national-security/chinese-hackers-breach-federal-governments-personnel-office/2015/06/04/889c0e52-0af7-11e5-95fd-d580f1c5d44e_story.html.

“Not My Problem.” 2014. The Economist. July 10, 2014. https://www.economist.com/special-report/2014/07/10/not-my-problem.

Rasser, Martijn, and Kevin Wolf. 2022. “The Right Time for Chip Export Controls.” Lawfare. December 13, 2022. https://www.lawfareblog.com/right-time-chip-export-controls.

Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023. https://ourworldindata.org/ai-timelines.

Sganga, Nicole. 2022. “Chinese Hackers Took Trillions in Intellectual Property from about 30 Multinational Companies.” CBS News. May 4, 2022. https://www.cbsnews.com/news/chinese-hackers-took-trillions-in-intellectual-property-from-about-30-multinational-companies/.

“TrojAI: Trojans in Artificial Intelligence.” n.d. IARPA. Accessed March 11, 2023. https://www.iarpa.gov/research-programs/trojai.

Vincent, James. 2023. “Meta’s Powerful AI Language Model Has Leaked Online — What Happens Now?” The Verge. March 8, 2023. https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse.


[1] Nakashima, Ellen. 2015. “Chinese Breach Data of 4 Million Federal Workers.” The Washington Post, June 4, 2015. https://www.washingtonpost.com/world/national-security/chinese-hackers-breach-federal-governments-personnel-office/2015/06/04/889c0e52-0af7-11e5-95fd-d580f1c5d44e_story.html; and

Sganga, Nicole. 2022. “Chinese Hackers Took Trillions in Intellectual Property from about 30 Multinational Companies.” CBS News. May 4, 2022. https://www.cbsnews.com/news/chinese-hackers-took-trillions-in-intellectual-property-from-about-30-multinational-companies/.

[2] For example, a single successful training run of GPT-3 reportedly required dozens of terabytes of data and cost millions of dollars of GPU usage, but the trained model is a file smaller than a terabyte in size and actors can operate it on cloud services that cost under $40 per hour. Sources: 
Cottier, Ben. 2022. “The Replication and Emulation of GPT-3.” Rethink Priorities. December 21, 2022. https://rethinkpriorities.org/publications/the-replication-and-emulation-of-gpt-3; and
Dickson, Ben. 2020. “The GPT-3 Economy.” TechTalks. September 21, 2020. https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/.

Additionally, one report suggested that by 2030, state-of-the-art models may cost hundreds of millions or even >$1B dollars to train (although the report highlights that these estimates could significantly change). Source: Cottier, Ben. 2023. “Trends in the Dollar Training Cost of Machine Learning Systems.” Epoch. January 31, 2023. https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.

[3] For discussion regarding this claim, see: Allen, Gregory, Emily Benson, and William Reinsch. 2022. “Improved Export Controls Enforcement Technology Needed for U.S. National Security.” Center for Strategic and International Studies. November 30, 2022. https://www.csis.org/analysis/improved-export-controls-enforcement-technology-needed-us-national-security; and

Rasser, Martijn, and Kevin Wolf. 2022. “The Right Time for Chip Export Controls.” Lawfare. December 13, 2022. https://www.lawfareblog.com/right-time-chip-export-controls.

[4] Vincent, James. 2023. “Meta’s Powerful AI Language Model Has Leaked Online — What Happens Now?” The Verge. March 8, 2023. https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse

[5] Brooks, Chuck. 2023. “Cybersecurity Trends & Statistics for 2023: More Treachery and Risk Ahead as Attack Surface and Hacker Capabilities Grow.” Forbes. March 5, 2023. https://www.forbes.com/sites/chuckbrooks/2023/03/05/cybersecurity-trends--statistics-for-2023-more-treachery-and-risk-ahead-as-attack-surface-and-hacker-capabilities-grow/?sh=2c6fcebf19db
Cox, Joseph. 2023. “How I Broke into a Bank Account with an AI-Generated Voice.” Vice. February 23, 2023. https://www.vice.com/en/article/dy7axa/how-i-broke-into-a-bank-account-with-an-ai-generated-voice.

[6] Feldstein, Steven. 2019. “The Global Expansion of AI Surveillance.” Carnegie Endowment for International Peace. September 17, 2019. https://carnegieendowment.org/2019/09/17/global-expansion-of-ai-surveillance-pub-79847;
Longpre, Shayne, Marcus Storm, and Rishi Shah. 2022. “Lethal Autonomous Weapons Systems & Artificial Intelligence: Trends, Challenges, and Policies.” Edited by Kevin McDermott. MIT Science Policy Review 3 (August): 47–56. https://doi.org/10.38105/spr.360apm5typ (p. 49).

[7] Calma, Justine. 2022. “AI Suggested 40,000 New Possible Chemical Weapons in Just Six Hours.” The Verge. March 17, 2022. https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx

[8] The example of Meta’s LLaMA, mentioned earlier, provides both some support and rebuttal for this concern: Meta has insisted it plans to continue sharing access despite the leaks, but there are good reasons to think this event will discourage other companies from implementing similar access rules. Source: Vincent, “Meta’s Powerful AI Language Model Has Leaked Online.”

[9] By this, I am referring to systems such as highly autonomous cyber systems (which could conceivably cause unintended havoc on a scale far greater than Stuxnet), AI systems in nuclear forces or strategic operations (e.g., early warning systems, command and control, and tracking foreign nuclear assets such as missile submarines), or outright “human-level” artificial general intelligence (AGI).

[10] Surveys of AI experts provide a mixed range of forecasts, but in a 2022 survey a non-trivial portion of such experts forecasted a 50% chance that “human-level AI” (roughly defined as a system that is better than humans at practically all meaningful tasks) will exist by 2035. Additionally, half of the surveyed experts forecasted a 50% chance of this outcome by 2061. Notably however, some types of “very powerful systems” (e.g., highly autonomous cyber systems) may not even require “human-level AI.” For data and further discussion regarding these forecasts, see Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023. https://ourworldindata.org/ai-timelines.

[11] For sources on this claim, see: “Not My Problem.” 2014. The Economist. July 10, 2014. https://www.economist.com/special-report/2014/07/10/not-my-problem; and 
Humphreys, Brian. 2021. “Critical Infrastructure Policy: Information Sharing and Disclosure Requirements after the Colonial Pipeline Attack.” Congressional Research Service. May 24, 2021. https://crsreports.congress.gov/product/pdf/IN/IN11683

[12] DARPA and IARPA are already working on some projects related to the security and reliability of AI models, including GARD at DARPA and TrojAI at IARPA. Sources: “Guaranteeing AI Robustness against Deception (GARD).” n.d. DARPA. Accessed March 11, 2023. https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception; and
“TrojAI: Trojans in Artificial Intelligence.” n.d. IARPA. Accessed March 11, 2023. https://www.iarpa.gov/research-programs/trojai.


 

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-08T03:25:17.168Z · EA · GW

I’m not sure I understand the concern with (1); I would first say that I think infinities are occasionally thrown around too lightly, and in this example it seems like it might be unjustified to say there are infinite possible values, especially since we are talking about units of people/population (which is composed of finite matter and discrete units). Moreover, the actual impact of a difference between 1.0000000000002% and 1.00000000000001% in most values seems unimportant for practical decision-making considerations—which, notably, are not made with infinite computation and data and action capabilities—even if it is theoretically possible to have such a difference. If something like that which seems so small is actually meaningful (e.g., it flips signs), however, then that might update you towards beliefs like “within analytical constraints the current analysis points to [balancing out |OR| one side being favored].” In other words, perhaps not pure uncertainty, since now you plausibly have some information that leans one way or another (with some caveats I won’t get into).

I think I would agree to some extent with (2). My main concern is mostly that I see people write things that (seemingly) make it sound like you just logically can’t do expected utility calculations when you face something like pure uncertainty; you just logically have to put a “?” in your models instead of “1/n,” which just breaks the whole model. Sometimes (like the examples I mentioned), the rest of the model is fine!

I contest that you can use “1/n”, it’s more just a matter of “should you do so given that you run the risk of misleading yourself or your audience towards X, Y, and Z failure modes (e.g., downplaying the value of doing further analysis, putting too many eggs in one basket/ignoring non-linear utility functions, creating bad epistemic cultures which disincentivize people from speaking out against overconfidence, …).”

In other words, I would prefer to see clearer disentangling of epistemic/logical claims from strategic/communication claims.

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-08T02:42:38.250Z · EA · GW

I'm not sure exactly what you mean by this, and I expect this will make it more complicated to think about than just giving utility differences with the counterfactual.

I just added this in hastily to address any objection that says something like “What if I’m risk averse and prefer a 100% chance of getting 0 utility instead of an x% chance of getting very negative utility.” It would probably have been better to just say something like “ignore risk aversion and non-linear utility.”

I would often find it deeply unsatisfying (i.e. it seems unjustifiable) to represent my beliefs with a single probability distribution; I'd feel like I'm pulling numbers out of my ass, and I don't think we should base important decisions on such numbers. So, I'd often rather give ranges for my probabilities. You literally can give single distributions/precise probabilities, but it seems unjustifiable, overconfident and silly.

I think this boils down to my point about the fear of miscommunicating—the questions like “how should I communicate my findings,” “what do my findings say about doing further analysis,” and “what are my findings current best-guess estimates.” If you think it goes beyond that—that it is actually “intrinsically incorrect-as-written,” I could write up a longer reply elaborating on the following: I’d pose the question back at you and ask whether it’s really justified or optimal to include ambiguity-laden “ranges” assuming there will be no miscommunication risks (e.g., nobody assumes “he said 57.61% so he must be very confident he’s right and doing more analysis won’t be useful”)? If you say “there’s a 1%-99% chance that a given coin will land on heads” because the coin is weighted but you don’t know whether it’s for heads or tails, how is this functionally any different from saying “my best guess is that on one flip the coin has a 50% chance of landing on heads”? (Again, I could elaborate further if needed)

if you actually tried to build a model, it would be extraordinarily unlikely for you to get 50-50

Sure, I agree. But that doesn’t change the decision in the example I gave, at least when you leave it at “upon further investigation it’s actually about 51-49.” In either case, the expected benefit-cost ratio is still roughly around 2:1. When facing analytical constraints and for this purely theoretical case, it seems optimal to do the 1/n estimate rather than “NaN” or “” or “???” which breaks your whole model and prevents you from calculating anything, so long as you’re setting aside all miscommunication risks (which was the main point of my comment: to try to disentangle miscommunication and related risks from the ability to use 1/n probabilities as a default optimal). To paraphrase what I said for a different comment, in the real world maybe it is better to just throw a wrench in the whole model and say “dear principal: no, stop, we need to disengage autopilot and think longer.” But I’m not at the real world yet, because I want to make sure I am clear on why I see so many people say things like you can’t give probability estimates for pure uncertainty (when in reality it seems nothing is certain anyway and thus you can’t give 100.0% “true” point or range estimates for anything).

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-08T02:13:17.071Z · EA · GW

I’m not sure I disagree with any of this, and in fact if I understood correctly, the point about using uniform probability distributions is basically what I was suggesting: it seems like you can always put 1/n instead of a “?” which just breaks your model. I agree that sometimes it’s better to say “?” and break the model because you don’t always want to analyze complex things on autopilot through uncertainty (especially if there’s a concern that your audience will misinterpret your findings), but sometimes it is better to just say “we need to put something in, so let’s put 1/n and flag it for future analysis/revision.”

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-08T02:04:46.131Z · EA · GW

It feels more natural, but I’m unclear what this example is trying to prove. It still reads to me like “if we think rain is 50% likely tomorrow then it makes sense to say rain is 50% likely tomorrow” (which I realize is presumably not what is meant, but it’s how it feels).

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-08T01:49:12.781Z · EA · GW

I actually think it probably (pending further objections) does have a somewhat straightforward answer with regards to the rather narrow, theoretical cases that I have in mind, which relate to the confusion I had which started this comment chain.

It’s hard to accurately convey the full degree of my caveats/specifications, but one simple example is something like “Suppose you are forced to choose whether to do X or nothing (Y). You are purely uncertain whether X will lead to outcome Great (Q), Good (P), or Bad (W), and there is guaranteed to be no way to get further information on this. However, you can safely assume that outcome Q is guaranteed to lead to +1,000 utils, P is guaranteed to lead to +500 utils, and W is guaranteed to lead to -500 utils. Doing nothing is guaranteed to lead to 0 utils. What should you do, assuming utils do not have non-linear effects?”

In this scenario, it seems very clear to me that a strategy of “do nothing” is inferior to doing X: even though you don’t know what the actual probabilities of Q, P, and W are, I don’t understand how the 1/n default will fail to work (across a sufficiently large number of 1/n cases). And when taking the 1/n estimate as a default, the expected utility is positive.

Of course, outside of barebones theoretical examples (I.e., in the real world) I don’t think there is a simple, straightforward algorithm for deciding when to pursue more information vs. act on limited information with significant uncertainty.

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-07T04:58:13.432Z · EA · GW

Perhaps this is a nice explanation for some people with mathematical or statistical knowledge, but alas it goes way over my head.

(Specifically, I get lost here: “ We just consider all probability distributions that predict that the odd bits will be zero with probability one, and without saying anything at all - the even bits, they can be anything.”)

(Granted, I now at least think I understand what a convex set is, although I fail to understand its relevance in this conversation.)

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-07T04:46:48.268Z · EA · GW

In this case probabilistically modelling the phenomenon doesn’t necessarily get you the right “value of further investigation” (because you’re not modelling hypothesis X)

I basically agree (although it might provide a decent amount of information to this end), but this does not reject the idea that you can make a probability estimate equally or more accurate than pure 1/n uncertainty.

Ultimately, if you want to focus on “what is the expected value of doing further analyses to improve my probability estimates,” I say go for it. You often shouldn’t default to accepting pure 1/n ignorance. But I still can’t imagine a situation that truly matches “Level 4 or Level 5 Uncertainty,” where there is nothing as good as or better than pure 1/n ignorance. If you truly know absolutely and purely nothing about a probability distribution—which almost never happens—then it seems 1/n estimates will be the default optimal distribution, because anything else would require being able to offer supposedly-nonexistent information to justify that conclusion.

Ultimately, a better framing (to me) would seem like “if you find yourself at 1/n ignorance, you should be careful not to accept that as a legitimate probability estimate unless you are really rock solid confident it won’t improve.” No?

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-06T19:00:44.548Z · EA · GW

But someone else with slightly different (but pretty arbitrary) precise probabilities could get the opposite sign and still huge expected impact. It would seem bad to bet a lot on one side if the sign and magnitude of the expected value is sensitive to arbitrarily chosen numbers.

I wonder if the problem here is a failure to disentangle “what is our best estimate currently” and “what do we expect is the value of doing further analysis, given how fragile our current estimates are.”

If my research agent Alice said “I think there’s a 50% chance that doing X leads to +2,000,000,000 utils and a 50% chance that doing X leads to -1,000,000,000 utils (and the same probabilities that not doing X leads to the opposite outcomes), but these probability estimates are currently just pure 1/n uncertainty; such estimates could easily shift over time pending further analysis” I would probably say “wow I don’t like the uncertainty here, can we maybe do further analysis to make sure we’re right before choosing to do X?”

In other words, the concern seems to be that you don’t want to misrepresent the potential for new information to change your estimates.

However, suppose Alice actually says “… and no matter how much more research effort we apply (within real-world constraints) we are confident that our probability estimates will not meaningfully change.” At that point, there is no chance at improving, so you are stuck with pure, 1/n ignorance.

Perhaps I’m just unclear what it would even mean to be in a situation where you “can’t” put a probability estimate on things that does as good as or better than pure 1/n ignorance. I can understand the claim that in some scenarios you perhaps “shouldn’t” because it risks miscommunicating about the potential value of trying to improve your probability estimates, but that doesn’t seem like an insurmountable problem (I.e., we could develop better terms and communication norms for this)?

Comment by Harrison Durland (Harrison D) on Model-Based Policy Analysis under Deep Uncertainty · 2023-03-06T16:43:59.336Z · EA · GW

Thank you for this post, overall I think it is interesting and relevant at least for my interests. There was one thing I wanted clarification on, however:

Level 4 uncertainties refer to a situation in which you know about what outcomes are possible but you do not know anything about their probability distributions, not even the ranking.

I’m often confused by these kinds of claims, as I don’t fully understand the assertion and/or problem here: if you genuinely cannot do better than assigning 1/n probability to each of n outcomes, then that is a legitimate distribution that you could use for expected-utility calculations. The reality is that oftentimes we do know at least slightly better than pure ignorance, but regardless, I’m just struggling to see why even pure ignorance is such a problem for expected utility calculations which acknowledge this state of affairs?

Comment by Harrison Durland (Harrison D) on Hosting a trivia night for our university EA club. Help me think of some EA related trivia questions. · 2023-03-04T04:10:20.702Z · EA · GW

My go-to response for many of these kinds of problems has become “Have you tried asking Chat-GPT?”

That’s not to say it’ll be great, and in fact it may require some degree of prompt engineering, but it seems worth a shot!

Comment by Harrison Durland (Harrison D) on Some more projects I’d like to see · 2023-03-01T03:51:16.953Z · EA · GW

[P]eople also talk about ‘belief-mapping’ software as if it’s a well-established micro-genre. I’m still not exactly sure what this is, but here’s something that sounds to me like ‘belief-mapping’: I’d love a way to log my credences in different things, as well as richer kinds of beliefs like confidence intervals and distributional forecasts. A combination of a spreadsheet plus Foretold.io can do all of this so far. But it could also be neat to express how beliefs must relate. For instance, if I update my credence in (A), and I’ve already expressed my credence in (B|A), then the software can tell me what my new credence in (B) should be, and update it if it seems reasonable. I could also say things like: “my credence in the disjunction A or B or C is 80% — so when I change one of A or B or C, please adjust the other two to add back up to 80%”. Or suppose I give some probability distribution over time for a “when will…” question, and then time passes and the event doesn’t happen.

 

I wrote a forum post about what I call "epistemic mapping" as well as a related post on "Research graphing" to support AI policy research (and other endeavors). I've begun working on a new, more-focused demo (although I'm still skeptical that it will get any traction/interest). I also just wrote up a shortform response to Leverage's report on "argument mapping" (which multiple people have directed me towards, with at least one of those people citing it as a reason to be skeptical about "argument mapping"). I've even published a relevant article on this topic ("Complexity Demands Adaptation...").

In short, I'm a big advocate for trying new research and analysis methods under the umbrella of what you might broadly call "belief-mapping."

Despite my enthusiasm for some of this, I am honestly quite skeptical of attempts to create end-to-end calculations for analyses which feature very interconnected, dynamic, hard-to-evaluate, and hard-to-explicate variables—which applies to most of the world beyond a few disciplines. At least, I'm skeptical of tools which try to do much more than Squiggle/Excel can already do (e.g., mapping the relationships between a lot of credences). In my view, the more important and/or fixable failure modes are "I can't remember all the reasons—and sub-reasons—I had for believing X," and "I don't know other people's arguments (or responses to my counterarguments) for believing X."[1]

I've been meaning to write up my thoughts on this kind of proposal for a few weeks (perhaps as part of a "Why most prior attempts at 'argument mapping' have failed, and why that doesn't spell doom for all such methods" post series), but am also skeptical that it will reach a sufficiently large audience to make it worthwhile. If someone were actually interested, I might try to organize my thoughts more quickly and neatly, but otherwise IDK.

  1. ^

    While paragraphs and bullet points can and do mitigate this to some extent, it really struggles to deal with complex debates (for a variety of reasons I have listed on various Notion pages but have yet to write a published post/comment on).

Comment by Harrison Durland (Harrison D) on Why I think it's important to work on AI forecasting · 2023-02-28T16:22:32.431Z · EA · GW

Like, so what if AGI arrives in 2035 versus 2038? I actually totally agree with this intuition.

I don’t think this is a well specified intuition. It would probably be really valuable if people could forecast the ability to build/deploy AGI to within roughly 1 year, as it could inform many people’s career planning and policy analysis (e.g., when to clamp down on export controls). In this regard, an error/uncertainty of 3 years could potentially have a huge impact.

However, I think a better explanation for the intuition I (and possibly you) have is tractability—that such precision is not practical at the moment, and/or would have such diminishing marginal returns so as to make other work more valuable.

Comment by Harrison Durland (Harrison D) on New database for economics research · 2023-02-24T15:30:18.229Z · EA · GW

I’m excited to see new work in this vein, as I have long been a proponent of better research aggregation methods, however I must say I do not find the project’s superficial dismissal of neoliberalism a particularly good sign of the project’s epistemics: “Here are a few (possibly cherry-picked?) charts that show some change in trends around 1979, with no counterfactual comparison/analysis; neoliberalism must have been to blame! That’s why we need to be more rigorous and thoughtful moving forward as we analyze policy!”

Comment by Harrison Durland (Harrison D) on The Importance of Holding Beliefs for the Right Reasons · 2023-02-23T05:42:56.579Z · EA · GW

My old article on “Witch Doctor Theory” seems relevant here:

Some of the primary consequences [of relying on dubious explanations/arguments] are:

  1. Having flawed foundations, leading to incorrect conclusions elsewhere. For example, as detailed later, having the wrong understanding of fiat power in policy debate leads to incorrect beliefs about topicality requirements for cases.
  2. Having weak foundations, leading to abandonment of the correct conclusion when the explanation is shown to be false. Referencing the poison oak as an example, suppose that a village child who has never seen the effects of poison oak is only told that it does not have evil spirits (or just stops believing in evil spirits). That child might then believe that it’s okay to touch the poison oak. Thus, it’s best not to base your conclusions on weak foundations.
  3. Being unable to convince others of the correct conclusion. Again referencing the poison oak example, suppose someone who did not believe in evil spirits and had never heard of poison oak was told that it was harmful because it contained evil spirits. That person would not be convinced of either the explanation or the conclusion because he does not see the reasoning as compelling.
Comment by Harrison Durland (Harrison D) on There can be highly neglected solutions to less-neglected problems · 2023-02-11T15:59:38.548Z · EA · GW

I suggest that a better definition would be: “How many people, or dollars, are currently being dedicated to this particular solution?”

Ultimately, I think this is just a band-aid solution to the fundamental problem of the INT framework: as I and others have written elsewhere, the INT framework is just a heuristic for thinking about overall cause areas; it is invalid (or prone to mislead, or inefficient) when it comes to evaluating specific decisions.

In contrast, I’ve spent a sizable amount of time developing an alternative framework which I think is actually reliable for evaluating specific decisions, here: https://forum.effectivealtruism.org/posts/gwQNdY6Pzr6DF9HKK/the-coils-framework-for-decision-analysis-a-shortened-intro

Comment by Harrison Durland (Harrison D) on A Manifesto · 2023-02-09T15:14:19.991Z · EA · GW

My church also packed meals for FMSC for many years (our youth group also did 30 Hour Famine and would sometimes do this during that time).

Comment by Harrison Durland (Harrison D) on A Manifesto · 2023-02-07T16:49:56.625Z · EA · GW

packing meals with Feed My Starving Children

Same!

Comment by Harrison Durland (Harrison D) on Launching The Collective Intelligence Project: Whitepaper and Pilots · 2023-02-06T18:18:16.440Z · EA · GW

In case you aren’t familiar with them, you might consider reaching out to people at the Canonical Debate Lab, as it seems potentially relevant to your work.

Comment by Harrison Durland (Harrison D) on Launching The Collective Intelligence Project: Whitepaper and Pilots · 2023-02-06T18:14:19.844Z · EA · GW

The post seems to have duplicate passages, starting with the second instance of “CIP is an incubator for new governance models for transformative technology” (and ending with the second item in the numbered list).

Edit: it also duplicates the passage starting with “We want to catalyze an ecosystem of aligned governance research and development projects.”

Comment by Harrison Durland (Harrison D) on Polis: Why and How to Use it · 2023-02-01T14:50:52.239Z · EA · GW

I wasn’t aware of this sequence, but I’m glad to see someone working on this topic! That being said, I am disappointed by the apparent lack of reference to Kialo or any other form of what I call argument/debate management systems

Comment by Harrison Durland (Harrison D) on Would Structured Discussion Platforms for EA Community Building Ideas be Valuable? (With Prototype Example) · 2023-02-01T14:40:25.454Z · EA · GW

That’s fair, although the user base in this case would mainly just be community builders rather than EA more generally, so I would figure that if it is considered beneficial enough the transition costs shouldn’t be that insurmountable.

Comment by Harrison Durland (Harrison D) on There should be a public adversarial collaboration on AI x-risk · 2023-01-23T16:00:42.329Z · EA · GW

Whatever people end up doing, I suspect it would be quite valuable if serious effort is put into keeping track of the arguments in the debate and making it easier for people to find responses for specific points, and responses to those responses, etc. As it currently stands, I think that a lot of traditional text-based debate formats are prone to failure modes and other inefficiencies.

Although I definitely think it is good to find a risk-skeptic who is willing to engage in such a debate

  1. I don’t think there will be one person who speaks for all skeptical views (e.g., Erik Larsen vs. Yann LeCun vs. Gary Marcus);

  2. I think meaningful progress could be made towards understanding skeptics’ points of view even if no skeptic wants to participate or contribute to a shared debate map/management system, so long as their arguments are publicly available (I.e., someone else could incorporate it for them).

Comment by Harrison Durland (Harrison D) on Doing EA Better · 2023-01-19T15:59:08.244Z · EA · GW

As a general note, when evaluating the goodness of a pro-democratic reform in a non-governmental context, it’s important to have a good appreciation of why one has positive feelings/priors towards democracy. One really important aspect of democracy’s appeal in governmental contexts is that for most people, government is not really a thing you consent to, so it’s important that the governmental structure be fair and representative.

The EA community, in contrast, is something you have much more agency to choose to be a part of. This is not to say “if you don’t like the way things are, leave,”—I am definitely pro-criticism/feedback—it’s just important to avoid importing wholesale one’s feelings towards democracy in governmental settings vs. settings like EA where people have more agency/freedom to participate, especially since democratic decision-making does have many disadvantages.

Comment by Harrison Durland (Harrison D) on Doing EA Better · 2023-01-19T15:46:50.130Z · EA · GW

I’ve hypothesized that one potential failure mode is that experts are not used to communicating with EA audiences, and EA audiences tend to be more critical/skeptical of ideas (on a rational level). Thus, it may be the case that experts aren’t always as explicit about some of the concerns or issues, perhaps because they expect their audiences to defer to them or they have a model of what things people will be skeptical of and thus that they need to defend/explain, but that audience model doesn’t apply well to EA. I think there may be a case/example to highlight with regards to nuclear weapons or international relations, but then again it is also possible that the EA skepticism in some of these cases is valid due to higher emphasis on existential risks rather than smaller risks.

Comment by Harrison Durland (Harrison D) on Doing EA Better · 2023-01-19T15:33:05.468Z · EA · GW

The most important solution is simple: one person, one vote.

I disagree with this: I may have missed a section where you seriously engaged with the arguments in favor of the current karma-weighted vote system, but I think there are pretty strong benefits of a system that puts value on reputation. For example, it seems fairly reasonable that the views of someone who has >1000 karma are considered with more weight than someone who just created an account yesterday or who is a known troll with -300 karma.

I think there are some valid downsides to this approach, and perhaps it would be good to put a tighter limit on reputation weighting (e.g., no more than 4x weight), but “one person one vote” is a drastic rejection of the principle of reputation, and I’m disappointed with how little consideration was apparently given to the potential negatives of this reform / positives of the current system.

Comment by Harrison Durland (Harrison D) on Where to find EAG funding support? · 2023-01-17T02:06:09.831Z · EA · GW

Could you provide more details, namely what EAG you’ve applied for and where you live/go to school?

Comment by Harrison Durland (Harrison D) on How should EA navigate the divide between those who prioritise epistemics vs. social capital? · 2023-01-16T07:06:58.138Z · EA · GW

you have the people[1] who want EA to prioritise epistemics on the basis that if we let this slip, we'll eventually end up in a situation where our decisions will end up being what's popular rather than what's effective.

And relatedly, I think that such concerns about longterm epistemic damage are overblown. I appreciate that allowing epistemics to constantly be trampled in the name of optics is bad, but I don’t think that’s a fair characterization of what is happening. And I suspect that in the short term optics dominate due to how they are so driven by emotions and surface-level impressions, whereas epistemics seem typically driven more by reason over longer time spans and IMX are more the baseline in EA. So, there will be time to discuss what if anything “went wrong” with CEA’s response and other actions, and people should avoid accidentally fanning the flames in the name of preserving epistemics, which I doubt will burn.

(I’ll admit what I wrote may be wrong as written given that it was somewhat hasty and still a bit emotional, but I think I probably would agree with what I’m trying to say if I gave it deeper thought)

Comment by Harrison Durland (Harrison D) on How should EA navigate the divide between those who prioritise epistemics vs. social capital? · 2023-01-16T06:46:03.410Z · EA · GW

I think that some sort of general guide on “How to think about the issue of optics when so much of your philosophy/worldview is based on ignoring optics in the sake of epistemics/transparency (including embedded is-ought fallacies about how social systems ought to work), and your actions have externalities that affect the community” might be nice, if only so people don’t have to constantly reexplain/rehash this.

But generally, this is one of those things where it becomes apparent in hindsight that it would have been better to hash out these issues before the fire.

It’s too bad that Scout Mindset not only doesn’t seem to address this issue effectively, it also seems to push people more towards the is-ought fallacy of “optics shouldn’t matter that much” or “you can’t have good epistemics without full transparency/explicitness” (in my view: https://forum.effectivealtruism.org/posts/HDAXztEbjJsyHLKP7/outline-of-galef-s-scout-mindset?commentId=7aQka7YXrhp6GjBCw)

Comment by Harrison Durland (Harrison D) on Zero Utils: why utility wants to be additive · 2023-01-14T19:11:20.286Z · EA · GW

Could you provide a tl;dr here (or there on the article, I suppose)?

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-13T21:13:46.244Z · EA · GW

I appreciate the summary, and I'm especially glad to see it done with an emphasis on relatively hierarchical bullet points, rather than mostly paragraph prose. (And thanks for the reference to my comment ;)

Nonetheless, I am tempted to examine this question/debate as a case study for my strong belief that, relative to alternative methods for keeping track of arguments or mapping debates, prose/bullets + comment threads are an inefficient/ineffective method of

  1. Soliciting counterarguments or other forms of relevant information (e.g., case studies) from a crowd of people who may just want to focus on or make very specific/modular contributions, and 
  2. Showing how relevant counterarguments and information relate to each other—including where certain arguments have not been meaningfully addressed within a branch of arguments (e.g., 3 responses down), especially to help audiences who are trying to figure out questions like "has anyone responded to X."

I'm not even confident that this debate necessarily has that many divisive branches—it seems quite plausible that there are relatively few cruxes/key insights that drive the disagreement—but this question does seem fairly important and has generated a non-trivial amount of attention and disagreement.

Does anyone else share this impression with regards to this post (e.g., "I think that it is worth exploring alternatives to the way we handle disagreements via prose and comment threads"), or do people think that summaries like this comment are in fact sufficient (or that alternatives can't do better, etc.)?

Comment by Harrison Durland (Harrison D) on "The job interview process is borderline predatory" · 2023-01-13T20:38:53.446Z · EA · GW

I understand your frustration, and have myself been in your shoes a few times. I think that many employers/recruiters in EA are aware of these downsides, as I have seen a variety of posts by people discussing this in the past. Additionally, as Samuel points out in a separate comment, many if not all of the work-trials/etc. I've participated in have been compensated, which seems quite reasonable/non-predatory.

Unfortunately, for some positions/situations I don't think there will be any process which satisfies everyone, as they always seem to have downsides. I can especially speak to my experience applying to positions in non-EA think tanks and elsewhere, where I've suspected that most of the interview/review processes are ridiculously subjective or plainly ineffective. Setting aside the process of selecting applicants for proceeding to the interview stage (which I suspect is probably under-resourced/flawed), I've had multiple interviews where I came away thinking "are you seriously telling me that's how they evaluate candidates? That's how they determine if someone is a good researcher? Do they not apply any scrutiny to my claims / are my peers just getting away with total BS here [as I've heard someone imply on at least one occasion]? Do they not want to know any more concrete details about the relevant positions or projects even after I said I could describe them in more detail?" 

Many of the EA-org interviews I've done may not feel "personal," but I'll gladly take objectivity and skill-testing questions over smiles and "tell me your strengths and weaknesses."

That being said, I do sympathize with you, and I do tend to find that it's much more frustrating to be turned down by an EA org after so much effort, but in the end I still think I would prefer to see this kind of deeper testing/evaluation more often.

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-12T15:18:38.661Z · EA · GW

It’s unclear to me that just because the number/liquidity of traders “in the know” is not very small (e.g., it is more than 0.1% of capital) this leads to the market correcting itself. At least, I have some reservations about what I interpret to be the causal process. Suppose that some set of early investors correctly think that ~3% of investors will adopt their own reasoning and engage in similar actions (e.g., “shorting” the long-term bond market) about 6 years before AGI.

But despite all of their reasoning, a very large portion of capital-weighted investors still don’t believe (A) the whole AGI argument, or (B) that there’s much worth doing once they do believe the whole AGI argument (e.g., “well, I guess I should just try not to die before AGI and enjoy my last normal years with my family/friends”).

I see a few potential problems, but am not sure about enough details to know whether the market would suffer from these problems:

  1. It seems plausible that large institutional investors will just balance against any large uptick early on, preventing investors from getting much of any profits in the first 10 or so years (leaving only 5-ish years for profits to start accumulating (albeit without considering discount rates));

  2. Even once the potential for profit opens up or even if the previous point doesn’t apply very strongly, some investors might eventually think they’ll be left “holding the bag” if they ever run into a multi-year plateau in beliefs/capital movement. This could be a scenario where most of the “AGI-open-minded” investors have been tapped, but most other people in society are still skeptical (I.e., it isn’t a smooth distribution of open-mindedness). Short-term profit relies on the rates increasing after you go short, but if you don’t expect the rates to increase then you won’t enter the market and adjust the prices. But the expectation that the person after you might also have this reasoning in its recursive form disincentivizes you from entering, creating a cascading effect.

  3. “Well, I’ll profit eventually, even if it takes 10 years of waiting”—not necessarily, or at that point you may not really enjoy the profits, as it may be “I have 8-figure assets but 3 years left of (normal) life.” I’m not confident that this is a sufficiently appealing offer to the people who could take you up on it and move the market.

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-11T21:10:51.001Z · EA · GW

At least, the small number of traders necessary to move the market are not dumb. They will understand the logic of this post. A mass ignoring of interest rates in favor of tech equity investing is not a stable equilibrium.

Could you try to give an estimate as to how much money would be necessary to move the markets? I'm not particularly familiar with the Treasuries market, but I'm not convinced that a small number of traders or even a few billion dollars per year in "smart money" could significantly change it, at least not enough to send signals separate from surrounding noise about the views.

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-11T20:56:23.910Z · EA · GW

[Edit: this is no longer applicable, sheesh stop downvoting]
Your tweets appear to be set to private (thus impacting the accessibility of the last link).

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-11T20:55:23.769Z · EA · GW

I've been discussing this concept for some time now, so I'm glad to see some people take a more formal stab at it. However, I must say that I'm overall disappointed with this post. I'll just lay out a few summary points, and if people are actually still reading this deep into the comments and want to hear more thoughts, I can oblige later:

  1. With the *slight* exception of the "you could be earning alpha" section, it does not really get deep into the causal mechanisms for why you should expect markets to be efficient.
  2. I think this post should have done a better job of aggregating and responding to contrary viewpoints; I feel like the post largely bypassed the key arguments (cruxes) of existing critics and went straight to people who were not familiar with the EMH+AGI debate , especially with all the references to empirical evidence (see next point).
    Granted, "better job" implies that the article did this at all, which I don't recall it really doing, aside from occasional references to other viewpoints (IIRC).
  3. The empirical sections I thought were decent, but they missed the crux of the debate. 
    1. The fact is, we don't seem to have much of any precedent of this kind of scenario, with some debatable exceptions regarding the Cold War / Cuban Missile Crisis—yet the authors didn't even spend that much time focusing on these examples which seemed to be the most relevant.
    2. Overall, I thought that the empirical sections were not very helpful for the debate, aside from perhaps targeting audiences who are the very early stage of the debate and hastily think "I'll dismiss EMH in general because of X." 
      (I am normally a big proponent of EMH-style reasoning; it's not like I and many other people I know that are part of this EMH+AGI debate are saying "EMH has never worked!"[1]).
  4. One cross-cutting objection off the top of my head is that the people with Special And Justified Knowledge may not be able to profit fast enough to correct the market. [I have read other comments' responses, and respond to one response in the next point]
    This especially applies to two closely related causal mechanisms of the EMH, profit snowballing and dogpiling: "Suppose you have someone who has better insights than everyone else about some asset. They may not be rich and for various related reasons they are unable to immediately correct the market (i.e., the market is actually temporarily inefficient). However, if they are right/superior, they either a) can keep profiting over and over again until they become liquid/rich enough to individually correct the market, and/or b) other people see that this person is profiting over and over again so they jump in and contribute to market correction."
    The problem is that it might be the case that there is only one or two cycles for profit with AGI until the world goes crazy, but it could take many years for this strategy to actually profit, during which time the market will be "temporarily" inefficient. If real interest rates don't rise for 15 years, and only start to rise ~5 years before AGI, the market is inefficient for 15 years because small players can't profit to fix the situation.
  5. "But those are just two causal mechanisms," the authors/defenders hypothetically reply, "and sometimes the market still corrects even without those mechanisms; look at the big short! And there are probably enough AI-conscious investors such that they could alter the market..."
    1. First, I think it's worthwhile to highlight my view that the debate can unproductively explode at this point because the original authors didn't (IMO) do a good job of laying out their own causal mechanisms. This forces critics into a game of whack-a-mole filled with delays at the need to comment, wait for responses, address new causal mechanisms, parse out alternate branches of disagreement, etc. 
      (However, I think the following subpoint addresses a fairly large part of the debate)
    2. Second, I don't think that the authors did a good job of differentiating between "sudden surprise takeoffs" (e.g., ~1 year of warning time and ~1/3rd of people believe this) vs. "forecastable takeoffs" (e.g., ~10 years of warning time and >1/10th of people believe this). This seems somewhat cruxy in at least one direction—against the authors' viewpoint. Ultimately, (correct me if I'm wrong) it seems that the authors' proposed strategy for profit relies on the belief that as you get closer to expected AGI date, more/richer people will start to agree with your predictions (and still see benefits from getting in on profit): otherwise, prescient investors could believe "AI is very likely to occur around year X, but  very few people or institutions will recognize this before X-3 years, such that real interest rates probably won't change much at all until it's too late, and when they do change
      1. The counterparty/non-payment risk may be high;
      2. I prefer a 50% chance of being moderately wealthy for 10 years to a 50% chance of being really rich for ~2 years before I die (with a 50% chance of being poor for 10 years if I bet big and am wrong);
      3. The world might experience chaos which undermines my ability to spend money on things I value, etc."
    3. Third, I don't think the claim that "there are probably enough AI-conscious investors..." is supported in this post, and I'm hesitant on this point. I am willing to budge, and this could be a fairly important point if we are in a "forecastable/slow takeoff" scenario, but I would like to see the post focusing on that leaf of the debate, not trying to recreate the trunk of the debate tree. And again, if we are in a "sudden short timeline" scenario, I suspect that this possibility doesn't matter all that much.
  1. ^

    Sure, some people may hold this view, but a) I'm skeptical you'll convince them with this article, and b) you can't just focus on empirics and then declare victory when there are still many critics who have objections you haven't directly addressed.

Comment by Harrison Durland (Harrison D) on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years · 2023-01-11T19:22:49.111Z · EA · GW

I don’t know whether I defend Yudkowsky’s view (I only skimmed), but:

  1. The “profit” in the scenario you describe doesn’t seem sufficient to move the market (as small investors), because it isn’t “profit” that you can keep growing and reinvesting (snowballing) until your Special Insights (TM) fix the market;

  2. If you end up with a scenario where 2040 arrives and it’s Utopia, then there may not be any serious incentive to care, whereas there would be downside risks if it doesn’t occur: if you’re right, cool, you lived slightly better for 20 years out of a 1,000,000,000-year life of happiness; if you’re wrong you might be in so much debt you can’t pay medical bills/whatever, and die before actual AGI date.

  3. I’m not confident you could get massive loans with no collateral and no business model or whatnot to repay the loans come 2040.

Comment by Harrison Durland (Harrison D) on Harrison D's Shortform · 2023-01-11T01:07:56.656Z · EA · GW

I have created an interactive/explorable version of my incomplete research graph of AI policy considerations, which is accessible here: https://kumu.io/hmdurland/ai-policy-considerations-mapping-ora-to-kumu-export-test-1#untitled-map 

I'm not sure this is worth a full post, especially since the original post didn't really receive much positive feedback (or almost any feedback period). However, I was excited to discover recently that Kumu seems to handle the task of exporting from ORA fairly well, and I figured "why not make it accessible", rather than just relying on screenshots (as I did in the original article).

To rehash the original post/pitch, I think that a system like this, could 

1a) reduce the time necessary to conduct literature reviews and similar tasks in AI policy research; 

1b) improve research quality by reducing the likelihood that researchers will overlook important considerations prior to publishing or that they will choose a suboptimal research topic; and 

2) serve as a highly-scalable/low-oversight task for entry-level researchers (e.g., interns/students) who want to get experience in AI policy but were unsuccessful in applying to other positions (e.g., SERI) that suffer from mentorship constraints—whereas I think that this work would require very little senior researcher oversight on a per-contributor basis (perhaps like a 1 to 30 ratio, if senior researchers are even necessary at all?).

The following example screenshots from Kumu will be ugly/disorienting (as it was with ORA), as I have put minimal effort into optimizing the view, and it really is something you need to zoom in for since you otherwise cannot read the text. Without further ado, however, here is a sample of what's on the Kumu project:

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-10T04:19:24.586Z · EA · GW

I don't really think it's responding well to what UBI proponents usually mean when they talk about it being universal. 

I would be open to this response, but in every conversation that I can recall having with someone who advocates for a UBI (including other people in this comment section), this has been wrong, at least initially: they start out by describing it as a series of payments to all people, regardless of income/etc. This is also true for popular proponents of UBI programs, like Andrew Yang. And usually people justify it by saying things like what you said in your own comment, about "oh, well if it isn't given to rich people then it'll disincentivize them from working, because then they'll hit the cutoff and lose all the benefits."

But of course, once someone highlights how this is almost strictly inferior to a more nuanced system (e.g., special negative income taxes), some proponents will just start retreating to a vague definition of UBIs which water down the whole concept and abandon a lot of the arguments they were just making in favor of it.

Now, I'll admit I didn't see the original point about temporary positions and pay periods. I don't think I've heard people make this point before, but it's still unclear to me what it actually looks like in practice (e.g., "what if you get a job which pays $10K a month, but then lose the job on the first week and are thus ineligible for benefits that month"?)... Given the lack of clarity, and what I've been able to imagine on my own, I don't find this very compelling, as it seems to refer to a rather rare scenario where people may just have to manage the risks and/or deal with special bureaucracy despite it being inconvenient. The fact is, I can't imagine how this could be significant enough to overcome 10s or 100s of billions of dollars in cost savings by restricting payments to people who are already making decent incomes. However, if someone were to clarify the argument, I would be willing to consider it further.

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-09T22:36:39.900Z · EA · GW

You write that:

It's also important to recognize a difference between universal disbursement of a UBI and the net amount received when accounting for taxes.

I would first encourage you to read the rest of the discussion, as I think I address most of the first half of that paragraph (over the legitimacy of some definitions of a "UBI") in my back-and-forth with Michael Simm.

In the second half of the paragraph (especially "the benefit of universality and regularity is to create a greater freedom of movement for recipients without having to be concerned about the risks of temporarily losing entitlement [...]") you seem to retreat to the point that I addressed in my other reply.

Ultimately, I see the choices here as ultimately boiling down to 

  • "create a UBI which is not actually a UBI, because it is not universal or it adds special taxes/workarounds to functionally just undo the payment to richer people (rather than just having a special negative income tax system which does not send out payments to rich people in the first place)" (the definitional debate), or
  • "create a UBI which provides payments to people regardless of income, spending 10s if not 100s of billions of dollars in payments for people who are not poor."
Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-09T22:28:10.911Z · EA · GW

This concern is largely not applicable to a common-sense program that I describe, such as a negative income tax, as there is no threshold where you suddenly lose most of your benefits. 

First, the reduction of benefits would not begin until one is already making a fairly decent salary (e.g., $60K). Second, the benefits would only reduce gradually over a span of, say, $60K, which means that every dollar you earn only costs a fraction of a dollar of benefits. To repaste one of my responses from elsewhere (with minor edits):

  1. To make the math easier, let's just say the income range points were actually $60K to $110K. Even if one assumes a linear progression (rather than a more nuanced, super-linear system which I would recommend but won't bother discussing here), then this means that for every extra dollar one makes after $60K, their benefits are only reduced by 20 cents ($0.20), thus meaning that they still make $0.80 with every extra dollar of income. (There is an added question of how this would be designed to interact with tax brackets, but the bottom line is that one can still be incentivized to make more money.)
  2. If one thinks that the incentive reduction described above is too steep, you can expand the window of eligibility (e.g., $60K to $160K) to reduce the slope and still likely save tens of billions of dollars.
Comment by Harrison Durland (Harrison D) on Forecasting extreme outcomes · 2023-01-09T17:55:21.349Z · EA · GW

One thing I might recommend in a document like this is to make it clear up front with concrete real examples what the use case of this theorem is. You eventually mention something about forecasting extreme height, but I was a bit confused about that and some readers may not reach that. More generally, after a quick read I am still a bit unclear why I would want to use/know this concept.

For example, you could write something like “Suppose that you are trying to forecast [real thing of interest]. Some of the relevant variables at play here are [ABC]. A naive approach might be [x] or assume [y], but actually, according to this theorem, [____].”

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-09T03:45:24.347Z · EA · GW

(Responding to two of your comments at once)

First, just ignore the whole thing about TANF; I included that on a whim and I'm aware of the many criticisms of the program, but I have no interest in defending it as it is not central to the main argument I'm making. That aside...

You say that:

The fundamental issue (unless I'm misunderstanding) with tax-credit is that if you're poor and have not made a high income, you can't get any benefit from them.

I suppose I should not have used the term "tax credit," as I think most tax credits are non-refundable, but there are refundable tax credits (i.e., you can be paid by the government in excess of what you owe in taxes). What I have in mind would be a refundable tax credit, which also applies for people who do not pay excess amounts in income taxes. (i.e., it would be like receiving a $X check if you have not paid any taxes)

 

You say that:

UBI is a government policy that distributes funds to all people equally, through redistributive taxes that are high enough for the wealthy to pay for the full amount.

Curiously, in a separate comment you also say:

I don't think the 'Universality' of UBI has ever meant that every person will benefit from it. The universality thing points to the fact that no-one will ever go below it.

I'm not exactly sure how to reconcile these two comments, but what I will say is: I do not get the impression that people usually have the "the government ensures all people will stay above X income" interpretation in mind when talking about a UBI (except sometimes when they are retreating to a definitional motte, as currently appears to be the case); most people have in mind "a government policy that distributes funds to all people equally," as you wrote in one of your comments. (I thought this is also what Andrew Yang had in mind, but I must admit some unfamiliarity with the details of his plan.)

Ultimately, I'll just conclude by saying that I think that redefining UBI in such a way as to allow a policy such as a negative income tax just destroys the definition of a UBI, and I think you will not find many people who are sympathetic to such definitional stretching (although they may unfortunately be unaware of what's happening).

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-09T01:19:19.128Z · EA · GW

We are disagreeing, if only insofar as that I disagree that a UBI can be universal and also functionally equivalent to an income-adjusted tax credit. Or at least, I think it is disingenuous to call that a UBI. And I think that if a UBI genuinely gives money to people making more than ~$120K a year without doing this whole bait-and-switch, it's a terrible policy relative to alternatives (e.g., income-adjusted tax credits). (Which to be clear is not even to suggest that I think income-adjusted tax credits are a good policy; I simply think that they are almost strictly superior to a UBI, except maybe for political tractability or other non-policy reasons.) 

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-09T01:14:31.427Z · EA · GW

A UBI by definition cannot distribute the new money in a way that is not universal. If you want to keep current taxes or even change tax brackets so that they are more progressive by increasing taxes on the wealthy that could still be a UBI, but it would mean that people making >$150K annually receive additional money for the government that they clearly don't need, even if they are in a 50% income tax bracket. However, I don't think you can in good faith call a program that adds a new special tax that only applies to the UBI and thus deposits money in rich people's bank accounts only to immediately yank 100% of the money back a "UBI," as that clearly goes against the spirit of the "Universal" characteristic.

 

You insist that this policy was supported by Milton Friedman:

UBI has always been a fundamentally redistributive policy proposal and is what people like Dr. Luther King Jr. and Milton Friedman meant back in the 60s.

However, I doubt this is an accurate interpretation of Milton Friedman, and I have only ever heard him talk about negative income taxes. From one quickly-found source:

The crowd at Yang's Washington, D.C., rally on April 15 burst into applause at the mention of Friedman. But the famous free-market economist's idea of UBI doesn't square up with Yang's. Friedman advocated for a negative income tax, which replaces levies on low-income individuals with supplemental funds from the government. Friedman's plan consequently ensures that everyone in society receives a guaranteed minimum income, but it doesn't redistribute money to people who don't need it.

 

Wealthy people are constantly afraid of losing their wealth, and UBI would meaningfully reduce that fear.

A UBI would not save rich people from losing >90% of their income if something catastrophic were to happen to their business/etc., and it does not act as a better social safety net than alternative policies such as negative income tax or other means-tested/emergency assistance (e.g., TANF).

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-08T22:21:54.849Z · EA · GW

And now we move to the motte, it seems. A program which simultaneously adds a UBI and a special tax specifically designed to take away the UBI from richer income groups is a UBI in name only, if even that; it is functionally just a more complicated form of income-adjusted welfare (e.g., a negative income tax credit but with extra steps). 

It's important to recognize that the new tax would actually have to be genuinely special/separate from the rest of the system in order to be functionally similar to what I have described, given that the highest income tax bracket in the US is only 37% (if I understand correctly).

Such an "intelligent tax framework" (especially if it includes Pigovian taxes) could involve imposing more administrative or cognitive costs on people who are trying to figure out how much they owe in taxes, unless one is actually just saying "a tax framework which is only different in that it hides the fact that some people are actually just paying the same/more in taxes even after accounting for the UBI.

Again, it might be plausible that this sneaky setup has political justifications, although I am fairly skeptical of that, given that I don't think people are that gullible / I think some commentators will be very outspoken that this is just a deceptive form of income-adjusted welfare. Moreover, I'm also quite skeptical of the argument that the people who make more than the cutoff would actually care enough about such a relatively small payment to lobby against it, unless one is supposing that the new system would not actually replace/eat most pre-existing welfare (in which case I think the new program would unacceptably increase the deficit in ways likely to create its own political backlash). 

Also, it is almost a red herring to claim benefits of imposing Pigovian Taxes as a way to fund the UBI, except insofar as one is appealing to the argument that such taxes are not politically feasible without reimbursement. But one does not need a "UBI" (emphasis on the "universal" and "basic" characteristics) to impose Pigovian Taxes, and in fact there are many proposals that take the form of limited rebates tied to specific taxes (e.g., carbon tax rebates). Additionally, it is possible that politicians/voters would still demand an end to the taxes while maintaining the UBI.

In the end, I disagree that one can have a UBI that is functionally equivalent to the kind of income-adjusted tax credit I've described, while still genuinely calling the UBI a "universal" basic income.

Comment by Harrison Durland (Harrison D) on Should UBI be a top priority for longtermism? · 2023-01-08T18:57:41.678Z · EA · GW

Re-reading my original comment, I may have miscommunicated a bit: I am aware of arguments such as those that you mention above, I just don't find them very compelling—in fact, they don't even seem to come close to justifying the added costs, they are just noisy observations.

As background for the following responses, assume that >10M Americans make more than $120K annually as individuals.[1] So, ignoring all the savings that are saved by reduced payments to people making between $60K and $120K, and ignoring the option to set the bar differently for household income (e.g., the option to reduce payments for children of wealthy parents): 
If the UBI is set at just $10K annually,[2] this translates to $100,000,000,000 ($100B) in cost savings by not providing income to already-well-off people. The unique advantages of UBI (or unique disadvantages of more-nuanced systems) need to overcome this threshold to even begin to justify a simplistic UBI.

With that in mind, I'll respond to your points:

  1. Administrative costs
    1. You still need administrative costs for a simplistic UBI to reduce fraud, ensure delivery of payments, etc. Thus, one cannot attribute the full administrative cost to ensuring income eligibility / it is not possible to run a UBI without administrative costs.
    2. There is no way that ensuring income eligibility would come even close to the cost savings identified above (~$100B). Just for a very quick illustration, the entire budget of the IRS in the current, more complex system is only ~$15B (if I read this website correctly).
  2. Personal costs
    1. Again, this is not entirely unique to a more-nuanced system; similar efforts may have to take place with a simple UBI.
    2. I would assume that you could automatically be means-tested based on your reported taxable income. If you already report making $100K a year for income tax purposes, I don't see why it would take much more reporting/administrative effort.
  3. Distortion of incentives
    1. The system I described does not significantly suffer from this. To make the math easier, let's just say the range were actually $60K to $110K. Even if one assumes a linear progression (rather than a more nuanced, super-linear system which I would recommend but won't bother discussing here), then this means that for every extra dollar one makes after $60K, their benefits are only reduced by 20 cents ($0.20), thus meaning that they still make $0.80 with every extra dollar of income. (There is an added question of how this would be designed to interact with tax brackets, but the bottom line is that one can still be incentivized to make more money.)
    2. If one thinks that the incentive reduction described above is too steep, you can expand the window of eligibility (e.g., $60K to $160K) to reduce the slope and still likely save tens of billions of dollars.
  1. ^

    According to governmental stats reported on Wikipedia, roughly 20M Americans make more than $100K annually.

  2. ^

    There are longer debates to be had whether this is actually "basic" (sufficient for living), but that can of worms need not be opened here.