Posts

ARC is hiring alignment theory researchers 2021-12-14T20:17:49.887Z
Hiring engineers and researchers to help align GPT-3 2020-10-01T18:52:21.088Z
Altruistic equity allocation 2019-10-16T05:54:49.426Z
Ought: why it matters and ways to help 2019-07-26T01:56:34.037Z
Donor lottery details 2017-01-11T00:52:21.116Z
Integrity for consequentialists 2016-11-14T20:56:27.585Z
What is up with carbon dioxide and cognition? An offer 2016-04-06T01:18:03.612Z
Final Round of the Impact Purchase 2015-12-16T20:28:45.709Z
Impact purchase round 3 2015-06-16T17:16:12.858Z
Impact purchase: changes and round 2 2015-04-20T20:52:29.894Z
$10k of Experimental EA Funding 2015-02-25T19:54:29.881Z
Economic altruism 2014-12-05T00:51:44.715Z
Certificates of impact 2014-11-11T05:22:42.438Z
On Progress and Prosperity 2014-10-15T07:03:21.055Z
Three Impacts of Machine Intelligence 2013-08-23T10:10:22.937Z
The best reason to give later 2013-06-14T04:00:31.000Z
Giving now vs. later 2013-03-12T04:00:04.000Z
Risk aversion and investment (for altruists) 2013-02-28T05:00:34.000Z
Why might the future be good? 2013-02-27T05:00:49.000Z
Replaceability 2013-01-22T05:00:52.000Z

Comments

Comment by Paul_Christiano on How would a language model become goal-directed? · 2022-07-17T18:42:57.638Z · EA · GW

Here is my story, I'm not sure if this is what you are referring to (it sounds like it probably is).

Any prediction algorithm faces many internal tradeoffs about e.g. what to spend time thinking about and what to store in memory to reference in the future. An algorithm which makes those choices well across many different inputs will tend to do better, and in the limit I expect it to be possible to do better more robustly by making some of those choices in a consequentialist way (i.e. predicting the consequences of different possible options) rather than having all of them baked in by gradient descent or produced by simpler heuristics.

If systems with consequentialist reasoning are able to make better predictions, then gradient descent will tend to select them.

Of course all these lines are blurry. But I think that systems that are "consequentialist" in this sense  will eventually tend to exhibit the failure modes we are concerned about, including (eventually) deceptive alignment.

I think making this story more concrete would involve specifying particular examples of consequentialist cognition, describing how they are implemented in a given neural network architecture, and describing the trajectory by which gradient descent learns them on a given dataset. I think specifying these details can be quite involved both because they are likely to involve literally billions of separate pieces of machinery functioning together, and because designing such mechanisms is difficult (which is why we delegate it to SGD). But I do think we can fill them in well enough to verify that this kind of thing can happen in principle (even if we can't fill them in in a way that is realistic, given that we can't design performant trillion parameter models by hand).

Comment by Paul_Christiano on How would a language model become goal-directed? · 2022-07-16T20:42:21.448Z · EA · GW

Some examples of more exotic sources of consequentialism:

  • Some consequentialist patterns emerge within a large model and deliberately acquire more control over the behavior of the model such that the overall model behaves in a consequentialist way. These could emerge randomly, or e.g. while a model is explicitly reasoning about a consequentialist (I think this latter example is discussed by Eliezer in the old days though I don't have a reference handy). They could either emerge within a forward pass, over a period of "cultural accumulation" (e.g. if language models imitate each other's outputs), or during gradient descent (see gradient hacking).
  • An attacker publishes github repositories containing traces of consequentialist behavior (e.g. optimized exploits against the repository in which they are included). They also place triggers in these repositories before the attacks, like stretches of low-temperature model outputs, such that if we train a model on github and then sample autoregressively the model may eventually begin imitating the consequentialist behavior included in these repositories (since long stretches of low-temperature model outputs occur rarely in natural github but occur just before attacks in the attacker's repositories). This is technically a special case of "#1 imitating consequentialists" but it behaves somewhat strangely since the people training the system weren't aware of the presence of the consequentialist.
  • An attacker selects an input on which existing machinery for planning or prediction within a large model is repurposed for consequentialist behavior. If we have large language models that are "safe" only because they aren't behaving as consequentialists, this could be a bad situation. (Compromised models could themselves design and deploy similar attacks to recruit still more models; so even random failures at deployment time could spread like a virus without any dedicated attacker. This bleeds into the first failure mode.)
  • A language model can in theory run into the same problem as described in what does the universal prior actually look like?,  even if it is only reasoning abstractly about how to predict the physical universe (i.e. without actually containing malign consequentialists). Technically this is also a special case of #1 imitating a consequentialist, but again it can be surprising since e.g. the consequentialist wasn't present at training time and the person deploying the system didn't realize that the system might imitate a consequentialist.

I find it interesting to think about the kind of dynamics that can occur in the limit of very large models, but I think these dynamics are radically less important than #1-#5 in my original answer (while still not being exhaustive). I think that they are more speculative, will occur later if they do occur, and will likely be solved automatically by solutions to more basic issues. I think it's conceivable some issues of this flavor will occur in security contexts, but even there I think they likely won't present an alignment risk per se (rather than just yet another vector for terrible cybersecurity problems) for a very long time.

Comment by Paul_Christiano on How would a language model become goal-directed? · 2022-07-16T16:55:28.299Z · EA · GW

Here are five ways that you could get consequentialist behavior from large language models:

  1. They may imitate the behavior of a consequentialist.
  2. They may be used to predict which actions would have given consequences, decision-transformer style ("At 8 pm X happened, because at 7 pm ____").
  3. A sufficiently powerful language model is expected to engage in some consequentialist cognition in order to make better predictions, and this may generalize in unpredictable ways.
  4. You can fine-tune language models with RL to accomplish a goal, which may end up selecting and emphasizing one of the behaviors above (e.g. the consequentialism of the model is redirected from next-word prediction to reward maximization; or the model shifts into a mode of imitating a consequentialist who would get a particularly high reward). It could also create consequentialist behavior from scratch.
  5. An outer loop could use language models to predict the consequences of many different actions and then select actions based on their consequences.

In general #1 is probably the most common ways the largest language models are used right now. It clearly generates real-world consequentialist behavior, but as long as you imitate someone aligned then it doesn't pose much safety risk.

#2, #4, and #5 can also generate real-world consequentialism and pose a classic set of risks, even if the vast majority of training compute goes into language model pre-training. We fear that models might be used in this way because it is more productive than #1 alone, especially as your model becomes superhuman. (And indeed we see plenty of examples.)

We haven't seen concerning examples of #3, but we do expect them at a large enough scale. This is worrying because it could result in deceptive alignment, i.e. models which are pursuing some goal different from next word prediction which decide to continue predicting well because doing so is instrumentally valuable. I think this is significantly more speculative than #2/4/5 (or rather, we are more unsure about when it will occur relative to transformative capabilities, especially if modest precautions are taken). However it is most worrying if it occurs, since it would tend to undermine your ability to validate safety--a deceptively aligned model may also be instrumentally motivated to perform well on validation. It's also a problem even if you apply your model even to an apparently benign task like next-word prediction (and indeed I'd expect this to be a particularly plausible if you try to do only #1 and avoid #2/4/5 for safety reasons).

The list #1-#5 is not exhaustive, even of the dynamics that we are currently aware of. Moreover, a realistic situation is likely to be much messier (e.g. involving a combination of these dynamics as well as others that are not so succinctly described). But I think these capture many of the important dynamics from a safety perspective, and that it's a good list to have in mind if thinking concretely about potential risks from large language models.

Comment by Paul_Christiano on On Deference and Yudkowsky's AI Risk Estimates · 2022-07-09T01:56:59.253Z · EA · GW

I'm not sure either of the quotes you cited by Eliezer require or suggest ridiculous overconfidence.

If I've seen some photos of a tiger in town, and I know a bunch of people in town who got eaten by an animal, and we've all seen some apparent tiger-prints near where people got eaten, I may well say "it's obvious there is a tiger in town eating people." If people used to think it was a bear, but that belief was formed based on priors when we didn't yet have any hard evidence about the tiger, I may be frustrated with people who haven't yet updated. I may say "The only question is how quickly people's views shift from bear to tiger. Those who haven't already shifted seem like they are systematically slow on the draw and we should learn from their mistakes." I don't think any of those statements imply I think there's a 99.9% chance that it's a tiger. It's more a statement rejecting the reasons why people think there is a bear, and disagreeing with those reasons, and expecting their views to predictably change over time. But I could say all that while still acknowledging some chance that the tiger is a hoax, that there is a new species of animal that's kind of like a tiger, that the animal we saw in photos is different from the one that's eating people, or whatever else. The exact smallness of the probability of "actually it wasn't the tiger after all" is not central to my claim that it's obvious or that people will come around.

I don't think it's central to this point, but I think 99% is a defensible estimate for many-worlds. I would probably go somewhat lower but certainly wouldn't run victory laps about that or treat it as damning of someone's character. The above is mostly a bad analogy explaining why I think it's pretty reasonable to say things like Eliezer did even if your all-things-considered confidence was 99% or even lower.

To get a sense for what Eliezer finds frustrating and intends to critique, you can read If many-worlds had come first (which I find quite obnoxious). I think to the extent that he's wrong it's generally by mischaracterizing the alternative position and being obnoxious about it (e.g. misunderstanding the extent to which collapse is proposed as ontologically fundamental rather than an expression of agnosticism or a framework for talking about experiments, and by slightly misunderstanding what "ontologically fundamental collapse" would actually mean). I don't think it has much to do with overconfidence directly, or speaks to the quality of Eliezer's reasoning about the physical world, though I think it is a bad recurring theme in Eliezer's reasoning about and relationships with other humans. And in fairness I do think there are a lot of people who probably deserve Eliezer's frustration on this point (e.g. who talk about how collapse is an important and poorly-understood phenomenon rather than most likely just being the most boring thing) though I mostly haven't talked with them and I think they are systematically more mediocre physicists.

Comment by Paul_Christiano on On Deference and Yudkowsky's AI Risk Estimates · 2022-07-06T17:40:45.779Z · EA · GW

I think my views about takeoff speeds are generally similar to Robin's though neither Robin nor Eliezer got at all concrete in that discussion so I can't really say. You can read this essay from 1998 with his "outside-view" guesses, which I suspect are roughly in line with what he's imagining in the FOOM debate.

I think that doc implies significant probability on a "slow" takeoff of 8, 4, 2... year doublings (more like the industrial revolution), but a broad distribution over dynamics which also puts significant probability on e.g. a relatively fast jump to a 1 month doubling time (more like the agricultural revolution). In either case, over the next few doublings he would by default expect still further acceleration. Overall I think this is basically a sensible model.

(I agree that shorter timelines generally suggest faster takeoff, but I think either Robin or Eliezer's views about timelines would be consistent with either Robin or Eliezer's views about takeoff speed.)

Comment by Paul_Christiano on On Deference and Yudkowsky's AI Risk Estimates · 2022-07-05T21:16:52.500Z · EA · GW

e.g. Paul Christiano has also said that Hanson's predictions looked particularly bad in the FOOM debate

To clarify, what I said was: 

I don't think Eliezer has an unambiguous upper hand in the FOOM debate at all

Then I listed a bunch of ways in which the world looks more like Robin's predictions, particularly regarding continuity and locality. I said Robin's predictions about AI timelines in particular looked bad. This isn't closely related to the topic of your section 3, where I mostly agree with the OP.

Comment by Paul_Christiano on On Deference and Yudkowsky's AI Risk Estimates · 2022-07-05T21:02:15.358Z · EA · GW

This doesn't feel like a track record claim to me. Nothing has changed since Eliezer wrote that; it reads as reasonably now as it did then; and we have nothing objective against which to evaluate it.

I broadly agree with Eliezer that (i) collapse seems unlikely, (ii) if the world is governed by QM as we understand it, the whole state is probably as "real" as we are, (iii) there seems to be nothing to favor the alternative interpretations other than those that make fewer claims and are therefore more robust to unknown-unknowns. So if anything I'd be inclined to give him a bit of credit on this one, given that it seems to have held up fine for readers who know much more about quantum mechanics than he did when writing the sequence.

The main way the sequence felt misleading was by moderately overstating how contrarian this take was. For example, near the end of my PhD I was talking with Scott Aaronson and my advisor Umesh Vazirani, who I considered not-very-sympathetic to many worlds. When asked why, my recollection of his objection was "What are these 'worlds' that people are talking about?  There's just the state." That is, the whole issue turned on a (reasonable) semantic objection.

However, I do think Eliezer is right that in some parts of physics collapse is still taken very seriously and there are more-than-semantic disagreements. For example, I was pretty surprised by David Griffiths' discussion of collapse in the afterword of his textbook (pdf) during undergrad. I think that Eliezer is probably right that some of these are coming from a pretty confused place. I think the actual situation with respect to consensus is a bit muddled, and e.g. I would be fairly surprised if Eliezer was able to make a better prediction about the result of any possible experiment than the physics community based on his confidence in many-worlds. But I also think that a naive-Paul perspective of "no way anyone is as confused as Eliezer is saying" would have been equally-unreasonable.

I agree that Eliezer is overconfident about the existence of the part of the wavefunction we never see. If we are deeply wrong about physics, then I think this could go either way. And it still seems quite plausible that we are deeply wrong about physics in one way or another (even if not in any particular way). So I think it's wrong to compare many-worlds to heliocentrism (as Eliezer has done). Heliocentrism is extraordinarily likely even if we are completely wrong about physics---direct observation of the solar system really is a much stronger form of evidence than a priori reasoning about the existence of other worlds. Similarly, I think it's wrong to compare many-worlds to a particular arbitrary violation of conservation of energy when top quarks collide, rather than something more like "there is a subtle way in which our thinking about conservation of energy is mistaken and the concept either doesn't apply or is only approximately true." (It sounds reasonable to compare it to the claim that spinning black holes obey conservation of angular momentum, at least if you don't yet made any astronomical observations that back up that claim.)

My understanding is this is the basic substance of Eliezer's disagreement with Scott Aaronson. My vague understanding of Scott's view (from one conversation with Scott and Eliezer about this ~10 years ago) is roughly "Many worlds is a strong prediction of our existing theories which is intuitively wild and mostly-experimentally-unconfirmed. Probably true, and would be ~the most interesting physics result ever if false, but still seems good to test and you shouldn't be as confident as you are about heliocentrism."

Comment by Paul_Christiano on Updating on Nuclear Power · 2022-04-27T16:44:07.046Z · EA · GW

And here's the initial post (which seems a bit less reasonable, since I'd spent less time learning about what was going on):

Given current trends in technology and policy, solar panels seem like the easiest way to make clean electricity (and soon the easiest way to make energy at all). I’m interested in thinking/learning about what a 100% solar grid would look like.

Here are my own guesses.

(I could easily imagine this being totally wrong because I’m a layperson who has only spent a little while looking into this. I’m not going to have “I think caveats” in front of *every* sentence but you should imagine them there.)

Overall I was surprised by how economical all-solar seems. Given 10-20 years and moderate progress on solar+storage I think it probably makes sense to use solar power for everything other than space heating, for which it seems like we should probably just continue to use natural gas. I was surprised by how serious and isolated a problem space heating seemed to be.

Other forms of power like nuclear or fusion might be even better, but it feels like all-solar will still be cheaper and easier than the status quo and won’t require any fossil fuels at all. Issues with storage would create big fluctuations in the price of electricity, which would change the way we use and think about electricity but would not change the basic cost-benefit analysis.

ETA: feels like the main problem is if there's variability in how dark winters are and some of them are quite dark. This is related to the heating issue, but might be a big problem even for non-heating needs.

1. Night vs day

It looks like the costs of overnight energy storage will eventually dominate the costs of solar, but still be low enough to easily beat out other power sources.

For example, the cost of a Tesla powerwall like $400/kWH; they can be cycled 5000 times under warranty. If you did that every night for 15 years it’s a total cost of $0.08/kWH stored. The cost of battery storage has fallen by a factor of 3 over the last 5 years and seems likely to continue to fall, and I expect utility-scale prices to also fall to keep up roughly with batteries.

Here are some cost projections of $400/kWH in 2020 falling to $200/kWH in 2030: https://www.nrel.gov/docs/fy20osti/75385.pdf. Here is a description of historical costs that finds them falling from $2150/kWH to $625/kWH in 2018: https://www.eia.gov/todayinenergy/detail.php?id=45596. Overall it looks to me like the $200/kWH looks pretty realistic

(ETA: I now think that forecast is probably pretty conservative and $100/kWH or less is more likely. But the rest of the post doesn't depend on such aggressive estimates, except for the part where I talk about heating.)

The efficiency of storage is 90%+ which is high enough to not matter much compared to the cost of storage, especially as solar prices fall.

Current electricity prices are around $0.10/kWH. So at $0.08/kWH solar couldn’t be quite competitive, but another factor of 2-4 could easily do it (especially if other costs of solar continue to fall to negligible levels at their current very rapid clip). I haven’t seen anyone projecting batter prices to plateau before hitting those levels.

Overall the trends on storage are worse than on panels themselves; it’s already the biggest cost of an all-solar grid and I think it would just become totally dominant. But they still seem low enough to make it work.

Storage is a lot cheaper if you are using some of your electricity directly from panels (as under the status quo) and need to store <100% of your power. You’d only need 100% in the worst case where all solar power arrives in a burst at noon, and the real world isn’t going to be quite that bad.

I could easily imagine cutting this down to only needing to store 50-75% of electricity, which cuts the cost with current technologies to $0.04-0.06/kWH. I think cutting costs in this way would be important in practice, but given that we’re only talking about a factor of 2 it’s not going to make a big difference unless battery costs plateau in the next few years.

Meaningful amounts of solar are only available for ~1/3 of a day (depending on latitude) so if you just used energy constantly and wasted nearly half of the solar power you’d need like 66% storage (depending a lot on latitude and season). Today we have a *lot* of appliances that use electricity a small fraction of their life and that you can run when electricity is cheap; and also most of our needs are when humans are awake and the sun is up. But if you switch to using electricity for more applications and if we industrialize further (as I’d expect) then this will get less possible. Amongst the things that can be shifted, you might as well put them at the peak hours around noon, which helps cut peak losses. (For things like cars and computers with batteries, you can either think of these as flexible appliances or as further reasons.)

That’s a complicated mess of considerations. 50-75% is a rough guess for how much you’d have to store but I’m not at all confident.

Eventually it will be unrealistic to amortize battery costs over 15 years. That said, 30 year interest rates are currently 1% and I think time horizons are changing pretty slowly, so I expect this trend to be much slower than changes in storage costs.

2. Winter vs summer

My impression is that solar panels give like 33% less power in winter than summer (obviously depending a ton on latitude, but that’s a typical value for populated places). Storing energy across seasons seems completely impractical.

That sounds like a lot but even in the worst case it only increases the cost of solar power by 50%, since you can just build 50% more panels. That doesn’t seem like enough overhead to make a fundamental difference.

Most importantly, this doesn’t increase the number of batteries you need. You will have enough batteries to store power in the winter, and then in the summer you will have a ton of extra production that you can't store and so use in some low-value way. So if batteries are the dominant cost, you don’t even care.

I think this is the main answer, and the rest of this section is gravy. But as in the last section, the “gravy” could still cut costs by 10% or more, so I think people will care a lot about it and it will change the way we relate to electricity. So also interesting to talk about.

Here are some guesses about what you could scale up in the summer:

* You can run some machines only during the summer, e.g. if the main cost of your computer was the cost of running it (rather than capital costs) then you might as well scale down your datacenters in the winter. Of course, you probably wanted to move that kind of machine towards the equator anyway where electricity prices would be lower.

* You could imagine literally migrating your machines to wherever the power is cheap (e.g. moving your datacenter to the other hemisphere for the winter). This sounds a bit crazy but I wouldn’t be at all surprised if it works for a non-negligible % of energy. Perhaps the simplest case would be moving vehicles and having people travel more during their summer.

* There are lots of small tweaks on the margin, e.g. using 10% more energy during the summer than you would if electricity was constant-price and using 10% less energy during the winter. You can do everything a bit slower and more efficient when it gets colder.

These things are interesting to think about—I like the image of a civilization that hums to life during the summer days—but it doesn’t seem like it changes the calculation for feasibility of solar at all. You could just totally ignore all of this and pay a small premium, from 0% (if batteries dominate anyway) to +50%. Some of these changes would happen from the most cost-conscious customers.

3. Other fluctuation and peaking power

In addition to winter/summer and night/day there is also some variability from weather day-to-day. My impression is that this is a smaller deal than the other two factors and doesn’t change the calculus much for a few reasons:

* As discussed in the last section, you probably want to have too many panels anyway and be bottlenecked by storage. In that case, variability in weather doesn’t matter much since you have too much capacity most days.

* It only really potentially matters in the winter months with already-low light, where you may fall short on total production (and not even be able to charge batteries).

* But once you are talking about a small fraction of the year, it’s pretty cheap for some people to just turn off the power (e.g. scaling down my datacenter) from time to time. If I’m doing that for 5% of the year it effectively increases capital costs by 5%, which is only really a problem if electricity costs are a tiny fraction of my net expenses. And there only have to be a few industries that can do that. So we only have a problem if the all-solar grid is serving every industry exceptionally well (in which case the absolute size of my problem is quite small).

There are also fluctuations in demand. In general having variable demand seems like it’s probably good, since by passing on costs appropriately you can shift demand to times when power is available. But big exogenous changes could cause trouble. This seems to be by far most troubling at the scale of days rather than hours (since you have a ton of batteries to handle within-day variability).

I think most people can avoid huge fluctuations in demand most of the time---there just aren’t that many things that bounce around from day to day where I have very little control over when they happen. The big exception I know about is climate control—people want to use a lot more power for AC during hot days and for heating during cold days (if we move away from natural gas for heating).

AC isn’t a problem, because it happens during hot summer days when you have tons of extra power anyway. So that brings us to...

4. Heating

Heating during cold periods seems like a big problem. As far as I can see it's the biggest single problem with an all-solar grid (with all-electric heating)

Unfortunately, I think heating is a giant use of energy (at least in the US). Right now I think it’s almost half of home energy use, mostly natural gas, and I’d guess something like 10-20% of total energy use in the US.

It's also the worst possible case for solar. It’s seasonal, which is already bad, and then there is huge variation in how cold it actually gets. And it’s really bad if you aren’t able to keep things heated during a cold snap. In practice you should just stop other uses of electricity when it gets cold. But with an all-solar grid you aren’t going to be putting many energy-intensive activities in places with cold winters, so you may have less cheap slop than you wanted and blackouts from cold could be quite expensive (even if you literally aren't having people freeze in their homes).

Here are some options:

* Use peaking power plants basically just for heating. It’s crazy to me to imagine the world where this is the *only* reason you actually need peaking power plants. I suspect you don’t want to do this.

* Use natural gas to heat homes. This is appealing because it’s what we currently do so doesn’t require big changes, it’s pretty clean, and you don’t need to maintain a bunch of peaking power plants with significant efficiency losses in transit. I think the main cost is maintaining infrastructure for delivering natural gas.

* Do something creative or develop new technologies. In some sense heating is an incredibly “easy” problem, since anything you do with electricity will generate heat. The problem is just getting it where you want to go. You could move more electricity-consuming appliances into homes/offices you want to heat, or do cogeneration with data centers, or something else crazy.

Here are some reasons the heating cost may not be so bad, so that you may be able to just eat the costs (for any of the above proposals).

* If we are doing a lot of electricity-intensive industry then space heating may be a much smaller fraction of costs. Honestly, I think this whole discussion is mostly relevant if we want to scale up electricity use quite a lot, but I don’t expect to scale up space heating in the same way. So I think it would be reasonable to keep meeting our heating needs in a primitive way while scaling up an all-solar grid for our increasing energy needs.

* You could massively improve insulation over the status quo if heating costs were actually a big deal. Right now heating is a huge fraction of energy but a much smaller fraction of costs. Under an all-solar grid the energy for heating would be by far the most expensive energy, and so incentives to save on heat would be much larger.

* We could generally move to hotter places. They get more appealing as AC gets cheaper / heating is more expensive, and I’m told global warming will make everywhere a bit hotter.

* We could probably modestly improve the energy efficiency of heating by using heat pumps Unfortunately it’s kind of hard to improve efficiency in any other way. And heat pumps are pretty scary since they don’t work when it gets really cold.

Overall my guess is that you should just leave natural gas infrastructure in place, especially in cold places, and use solar for everything else.

Comment by Paul_Christiano on Updating on Nuclear Power · 2022-04-27T16:40:01.453Z · EA · GW

No, sorry. Here's a copy-paste though.

Yet another post about solar! This time about land use.

— TL;DR

Suppose that you handle low solar generation winter by just building 3-6x more panels than you need in summer and wasting all the extra power.

1. The price of the required land is about 0.1 cents per kWh (2% of current electricity prices).

2. Despite the cost being low, the absolute amounts of land used are quite large. Replacing all US energy requires 8% of our land, for Japan 30%. This seems reasonably likely to be a political obstacle.

I’m not too confident in any of these numbers, corrections welcome.

— Background

I’ve been wondering about the price of an all-solar grid without any novel storage or firm generation. In my first post I proposed having enough batteries for 1-2 days, and said that buying that many batteries seemed affordable (https://www.facebook.com/paulfchristiano/posts/10226561810329293). In the second I argued that emergency natural gas you never actually use looked like it was totally affordable (https://www.facebook.com/paulfchristiano/posts/10226568532377340).

A potential drawback of the all solar plan is that you *massively* overbuild panels so that you have enough generation in the winter months. This isn’t too expensive because most of your capital cost was storage anyway. But it does mean you use a boatload of land. I wanted to understand that better. See the TL;DR above for my conclusions.

After this post, I think the biggest unresolved question for me is how variable cloud cover is during the winter—I know that large solar installations are pretty consistent at the scale of months (and can fall back to emergency natural gas in the rare cases where they aren’t). But is it the case that e.g. there is frequently a bad 4-day stretch in January where the average solar generation across Japan is significantly reduced?

My second biggest question is about the feasibility and cost of large-scale transmission, both to smooth out that kind of short-term variability and to supply power further north.

— A note on location

The feasibility of this depends a ton on where you are. I’m going to start by talking about the largest US solar farms in the southwest. I believe the situation gets about 2x worse if you move to the US northeast or northern Europe.

If you go further north it gets even more miserable---wintertime solar is much more sensitive to latitude than summer solar. I'd guess that people in the US northeast should already be importing power from sunnier places, to say nothing of Canada. I don’t know how politically realistic that is. If you didn’t have backup natural gas it sounds insane, but if everyone is just building backup natural gas anyway I think it might be OK.

— Efficiency of solar

I looked up the Topaz solar farm (info taken from wikipedia: https://en.wikipedia.org/wiki/Topaz_Solar_Farm).

Setting aside its first year while panels were still be installed, its worst month was December of 2016 were it generated about 57 million kWh.

The “overbuild panels” plan requires us to build enough panels that we’d be OK even in the deepest winter. If we pessimistically assume that all of the excess power is completely wasted, that means you get about 684 million kWh per year.

The site area is 7.3 square miles. So in total we are getting about 94 million kWh per square mile per year. (Or 145 thousand kWh per acre).

I got almost identical numbers for McCoy solar installation.

I think you could push the numbers somewhat higher, perhaps a factor of 2, by economizing more on land (check out that picture of Topaz solar farm from space, tons of room to improve density), improving panel efficiency (once panel costs are no longer a major expense you can focus on efficiency rather than price), and focusing on winter generation. When I did this calculation on paper I got numbers 2-4 higher than the practical ones.

I’m going to just round the number up to 100 million kWh to make things simple. In reality you’d probably increase density above this but may also be pushed to use worse sites, so this seems fine for the headline figures.

— How much land is needed in the US?

In 2020 the US used about 100 quadrillion BTUs of power (mostly oil and natural gas), a bit less than 3e13 kWh: https://www.eia.gov/energyexplained/us-energy-facts.

If we pretend it was always midwinter, this would require 300,000 square miles. This is about 8% of all the land in the US.

To help understand what this means, this site gives us the total breakdown of US land. I don’t trust it totally but I think it’s roughly right. https://www.visualcapitalist.com/america-land-use/

* 842,000 square miles of forest

* 750,000 square miles of shrub

* 530,000 square miles of farmland

* 530,000 square miles of grassland (I assume this breakdown was just made up?)

* 400,000 square miles of other nature

* 63,000 square miles of cities

— How expensive is that land?

Suppose that we put solar farms on cropland. The cost of 1 acre of farmland in the US is about $3000. Renting an acre of unirrigated land is about $140/year. (https://www.nass.usda.gov/.../land-values-cash-rents.pdf)

Pasture is quite a lot cheaper than that, and you’d only have to use ~50% of the US pasture to put in all this solar. So I think $140/acre/year is pretty conservative.

Above we estimated that an acre generated 145,000 kWh per year.

So even if you are renting farmland, and *throwing away all power above the amount generated in midwinter*, the price is only a tenth of a cent per kWh. That’s about 50x lower than the current price of power. So it won’t be a large part of the price until you are dropping electricity costs by 10x or more.

— What about Japan?

Japan uses about 386 million tons of oil equivalent per year, or 4.5e12 kWh. By the same calculation that would require about 45,000 square miles. (I think Japan has fewer good solar sites than the southwest US, so they’ll be leaning more on the hope that you can squeeze more density out of installations).

The area of Japan is about 145,000 square miles. So this is about 30% of the total area. Right now in Japan I believe essentially all of this would have to come from clearing forest. The cost of clearing that land isn’t significant (and it’s not any more expensive than cropland), but I expect people would be unhappy about losing 1/3 of their forest.

— Other thoughts

These proposals involving wasting 65-85% of all the generation. If you are able to use more electricity on summer days, that helps a lot, as discussed in previous posts. The most obvious way this happens is if you can synthesize fuel, and energy costs of synthesis are dominant rather than capital costs. That would be a game-changer for the all-solar grid (as well as removing the need to electrify all your cars and planes).

I’ve ignored increasing energy usage. That seems kind of reasonable because I’ve discussed the US and Japan, two countries with relatively high energy use that has been declining in recent years. But big increases in energy use would change the picture.

In the long run it does seem like floating solar over the ocean could be quite important. But I have no idea how to think about the costs for that, and especially energy transport.

Depending on the design of your panels, putting down this many could change significantly heat the earth just by absorbing sunlight. This is on the same order of magnitude as the heat generated by running appliances (e.g. the heat generated by the engine of your car and the friction of your wheels against pavement), but if your panel is 20% efficient then I think it probably ends up about 2-3x bigger. I don’t normally think about e.g. space heaters contributing to global warming by literally heating up the house. It does seem like a consideration but I’d like to better understand how it compares.

If clearing forests or pasture, it seems important not to release all that carbon into the atmosphere. My guess would have been that most of this land would be at rough equilibrium and so this isn’t going to have a CO2 effect (assuming you don’t burn the biomass or let it rot), but I’d be interested to know, and am not sure if that’s feasible.

Comment by Paul_Christiano on Updating on Nuclear Power · 2022-04-27T16:37:28.736Z · EA · GW

This does require prices going down. I think prices in many domains have gone up (a lot) over the last few years, so it doesn't seem like a lot of evidence about technological progress for solar panels. (Though some people might take it as a warning shot for long-running decay that would interfere with a wide variety of optimistic projections from the past.)

I think it's not clear whether non-technological factors get cheaper or more expensive at larger scales. Seems to me like "expected cost is below current electricity costs" is a reasonable guess, but ">75% chance of being economically feasible" is not.

My current understanding is that there are plenty of the relevant minerals (and in many cases there is a lot of flexibility about exactly what to use), and so this seems unlikely to be a major driver of cost over the very long term even if short-term supply is relatively inelastic. (Wasn't this the conclusion last time we had a thread on this?)

Comment by Paul_Christiano on Updating on Nuclear Power · 2022-04-24T16:00:07.983Z · EA · GW

I wrote a series of posts on the feasibility of an all-solar grid last year, here (it links to two prior posts).

Overall my tentative conclusion was:

  • It's economically feasible to go all solar without firm generation, at least in places at the latitude of the US (further north it becomes impossible, you'd need to import power).
  • The price of the land required for all-solar production seems very small.
  • However, the absolute amount of land required is nonetheless quite large. In the US building enough solar to supply all energy needs through a cloudy winter would be something like 8% of land; in Japan 30%+.
  • I expect this to be a serious political obstacle even if it's not an economic obstacle. (Though in extreme cases like Japan it may also become an economic obstacle since you have to move to increasingly marginal + expensive land.)
  • So in practice I expect most countries to need alternatives to solar for winter generation, at least in places at the latitude of the US  (closer to the equator it  becomes easier).
  • If you have alternatives for winter generation (or long-term storage), the land requirements fall by something like 3-5x. Winter vs summer isn't nearly as big a deal for total costs as for land use (since so much of the all-in cost is batteries and other infrastructure) (ETA: don't see where that 3-5x number came from, might be right but take this bullet with a grain of salt. I do think it's a big factor but maybe not that big?)
  • It seems like all-solar is mostly economically and technically feasible, though in addition to lots of land it requires modest further improvements in battery prices, maintaining back-up natural gas to use once a decade (which is relatively cheap), and building long-distance transmission (which is again affordable but likely to be prohibitively difficult politically).

It was interesting to me that "political feasibility" and "economic feasibility" seemed to come apart so strongly in this case.

Not sure if all of that is right, but overall it significantly changed my sense of the economics and real obstacles to renewable power.

Comment by Paul_Christiano on Is AI safety still neglected? · 2022-03-31T16:53:16.308Z · EA · GW

Regarding susceptibility to s-risk:

  • If you keep humans around, they can decide on how to respond to threats and gradually improve their policies as they figure out more (or their AIs figure out more).
  • If you build incorrigible AIs who will override human preferences (so that a threatened human has no ability to change the behavior of their AI), while themselves being resistant to threats, then you may indeed reduce the likelihood of threats being carried out.
  • But in practice all the value is coming from you solving "how do we deal with threats?" at the same time that you solved the alignment problem.
  • I don't think there's any real argument that solving CEV or ambitious value learning per se helps with these difficulties, except insofar as your AI was able to answer these questions. But in that case a corrigible AI could also answer those questions.
  • Humans may ultimately build incorrigible AI for decision-theoretic reasons, but I think the decision should do so should probably be separated from solving alignment.
  • I think the deepest coupling comes from the fact that the construction of incorrigible AI is itself an existential risk, and so it may be extremely harmful to build technology that enables that prior to having norms and culture that are able to use it responsibly.
  • Overall, I'm much less sure than you that "making it up as you go along alignment" is bad for s-risk.
Comment by Paul_Christiano on Against cash benchmarking for global development RCTs · 2022-03-21T17:11:41.279Z · EA · GW

When we eventually told the cash arm participants that we had given other households assets of the same value, most said they would have preferred the assets, “We don’t have good products to buy here”. We had also originally planned to work in 2 countries but ended up working in just 1, freeing up enough budget to pay for cash. 

I'm intuitively drawn to cash transfer arms, but "just ask the participants what they would want" also sounds very compelling for basically the same reasons. Ideally you could do that both before and after ("would you recommend other families take the cash or the asset?")

Have you done or seen systematic analysis along these lines? How do you feel about that idea?

Asking about the comparison to cash also seems like a reasonable way to do the comparison even if you were running both arms (i.e. you could ask both groups whether they'd prefer $X or asset Y, and get some correction for biases to prefer/disprefer the option they actually received).

Maybe direct comparison surveys also give you a bit more hope of handling timing issues, depending on how biased you think participants are by recency effects.  If you give someone an asset that pays off over multiple years, I do expect their "cash vs asset" answers to change over time. But still people can easily imagine getting the cash now and so if nothing else it seems like a strong sanity check if you ask asset-recipients in 2 years and confirm they prefer the asset.

At a very basic intuitive level, hearing "participants indicated strong preference for receiving our assets to receiving twice as much cash" feels more persuasive than comparing some measured outcome between the two groups (at least for this kind of asset transfer program where it seems reasonable to defer to participants about what they need/want)

Comment by Paul_Christiano on ARC is hiring alignment theory researchers · 2021-12-17T04:56:05.285Z · EA · GW

Compared to MIRI: We are trying to align AI systems trained using techniques like modern machine learning. We're looking for solutions that are (i) competitive, i.e. don't make the resulting AI systems much weaker, (ii) work no matter how far we scale up ML, (iii) work for any plausible situation we can think of, i.e. don't require empirical assumptions about what kind of thing ML systems end up learning. This forces us to confront many of the same issues at MIRI, though we are doing so in a very different style that you might describe as "algorithm-first" rather than "understanding-first." You can read a bit about our methodology in "My research methodology" or this section of our ELK writeup.

I think that most researchers at MIRI don't think that this goal is achievable, at least not without some kind of philosophical breakthrough. We don't have the same intuition (perhaps we're 50-50). Some of the reasons: it looks to us like there are a bunch of possible approaches for making progress, there aren't really any clear articulations of fundamental obstacles that will cause those approaches to fail, and there is extremely little existing work pursuing plausible worst-case algorithms. Right now it mostly seems like people just have varying intuitions, but searching for a worst-case approach seems like it's a good deal as long as there's a reasonable chance it's possible. (And if we fail we expect to learn something about why.)

Compared to everyone else: We think of a lot of possible algorithms, but we can virtually always rule it out without doing any experiments.  That means we are almost always doing theoretical research with pen and paper. It's not obvious whether a given algorithm works in practice, but it usually is obvious that there exist plausible situations where it wouldn't work, and we are searching (optimistically) for something that works in every plausible situation.

Comment by Paul_Christiano on Forecasting transformative AI: what's the burden of proof? · 2021-09-01T15:12:49.159Z · EA · GW

So I'd much rather people focus on the claim that "AI will be really, really big" than "AI will be bigger than anything else which comes afterwards".

I think AI is much more likely to make this the most important century than to be "bigger than anything else which comes afterwards." Analogously, the 1000 years after the IR are likely to be the most important millennium even though it seems basically arbitrary whether you say the IR is more or less important than AI or the agricultural revolution. In all those cases, the relevant thing is that a significant fraction of all remaining growth and technological change is likely to occur in the period, and many important events are driven by growth or tech change.

The answer to this question could change our estimate of P(this is the most important century) by an order of magnitude

I think it's more likely than not that there will be future revolutions as important TAI, but there's a good probability that AI leads to enough acceleration that a large fraction of future revolutions occur in the same century. There's room for the debate over the exact probability and timeline for such acceleration, but I think no real way to argue for anything as low as 10%.

Comment by Paul_Christiano on All Possible Views About Humanity's Future Are Wild · 2021-07-15T23:06:57.081Z · EA · GW

We were previously comparing two hypotheses:

  1. HoH-argument is mistaken
  2. Living at HoH

Now we're comparing three:

  1. "Wild times"-argument is mistaken
  2. Living at a wild time, but HoH-argument is mistaken
  3. Living at HoH

"Wild time" is almost as unlikely as HoH. Holden is trying to suggest it's comparably intuitively wild, and it has pretty similar anthropic / "base rate" force.

So if your arguments look solid,  "All futures are wild" makes hypothesis 2 look kind of lame/improbable---it has to posit a flaw in an argument, and also that you are living at a wildly improbable time. Meanwhile, hypothesis 1 merely has to posit a flaw in an argument, and hypothesis 3 merely has to a posit HoH (which is only somewhat more to swallow than a wild time).

So now if you are looking for errors, you probably want to focus for errors in the argument that we are living at a "wild time." Realistically, I think you probably need to reject the possibility that the stars are real and that it is possible for humanity to spread to them. In particular, it's not too helpful to e.g. be skeptical of some claim about AI timelines or about our ability to influence society's trajectory.

This is kind of philosophically muddled because (I think) most participants in this discussion already accept a simulation-like argument that "Most observers like us are mistaken about whether it will be possible for them to colonize the stars." If you set aside the simulation-style arguments, then I think the "all futures are wild" correction is more intuitively compelling.

(I think if you tell people "Yes, our good skeptical epistemology allows us to be pretty confident that the stars don't exist" they will have a very different reaction than if you tell them "Our good skeptical epistemology tells us that we aren't the most influential people ever.")

Comment by Paul_Christiano on Taboo "Outside View" · 2021-06-30T20:34:03.666Z · EA · GW

I do think my main impression of insect <-> simulated robot parity comes from very fuzzy evaluations of insect motor control vs simulated robot motor control (rather than from any careful analysis, of which I'm a bit more skeptical though I do think it's a relevant indicator that we are at least trying to actually figure out the answer here in a way that wasn't true historically). And I do have only a passing knowledge of insect behavior, from watching youtube videos and reading some book chapters about insect learning. So I don't think it's unfair to put it in the same reference class as Rodney Brooks' evaluations to the extent that his was intended as a serious evaluation.

Comment by Paul_Christiano on Taboo "Outside View" · 2021-06-30T17:35:55.709Z · EA · GW

The Nick Bostrom quote (from here) is:

In retrospect we know that the AI project couldn't possibly have succeeded at that stage. The hardware was simply not powerful enough. It seems that at least about 100 Tops is required for human-like performance, and possibly as much as 10^17 ops is needed. The computers in the seventies had a computing power comparable to that of insects. They also achieved approximately insect-level intelligence.

I would have guessed this is just a funny quip, in the sense that (i) it sure sounds like it's just a throw-away quip, no evidence is presented for those AI systems being competent at anything (he moves on to other topics in the next sentence), "approximately insect-level" seems appropriate as a generic and punchy stand in for "pretty dumb," (ii) in the document he is basically just thinking about AI performance on complex tasks and trying to make the point that you shouldn't be surprised by subhuman performance on those tasks, which doesn't depend much on the literal comparison to insects, (iii) the actual algorithms described in the section (neural nets and genetic algorithms) wouldn't plausibly achieve insect-level performance in the 70s since those algorithms in fact do require large training processes (and were in fact used in the 70s to train much tinier neural networks).

(Of course you could also just ask Nick.)

I also think it's worth noting that the prediction in that section looks reasonably good in hindsight. It was written right at the beginning of resurgent interest in neural networks (right before Yann LeCun's paper on MNIST with neural networks). The hypothesis "computers were too small in the past so that's why they were lame" looks like it was a great call, and Nick's tentative optimism about particular compute-heavy directions looks good. I think overall this is a significantly better take than mainstream opinions in AI. I don't think this literally affects your point, but it is relevant if the implicit claim is "And people talking about insect comparisons were lead astray by these comparisons."

I suspect you are more broadly underestimating the extent to which people used "insect-level intelligence" as a generic stand-in for "pretty dumb," though I haven't looked at the discussion in Mind Children and Moravec may be making a stronger claim. I'd be more inclined to tread carefully if some historical people tried to actually compare the behavior of their AI system to the behavior of an insect and found it comparable as in posts like this one (it's not clear to me how such an evaluation would have suggested insect-level robotics in the 90s or even today, I think the best that can be said is that today it seems compatible with insect-level robotics in simulation today). I've seen Moravec use the phrase "insect-level intelligence" to refer to the particular behaviors of "following pheromone trails" or "flying towards lights," so I might also read him as referring to those behaviors in particular. (It's possible he is underestimating the total extent of insect intelligence, e.g. discounting the complex motor control performed by insects, though I haven't seen him do that explicitly and it would be a bit off brand.)

ETA: While I don't think 1990s robotics could plausibly be described as "insect-level," I actually do think that the linked post on bee vision could plausibly have been written in the 90s and concluded that computer vision was bee-level, it's just a very hard comparison to make and the performance of the bees in the formal task is fairly unimpressive.

Comment by Paul_Christiano on Issues with Using Willingness-to-Pay as a Primary Tool for Welfare Analysis · 2021-06-28T00:21:46.741Z · EA · GW

Ironically, although cost-benefit analysts generally ignore the diminishing marginal benefit of money when they are aggregating value across people at a single date, their main case for discounting future commodities is founded on this diminishing marginal benefit. 

I think the "main" (i.e. econ 101) case for time discounting (for all policy decisions other than determining savings rates) is roughly the one given by Robin here

I don't think there is a big incongruity here. Questions about diminishing returns to wealth become relevant when trying to determine what savings rate might be socially optimal. Analogously, questions about diminishing returns to wealth become relevant when we ask about what level of redistribution might be socially optimal, even if most economists would prefer to bracket them for most other policy discussions.

Comment by Paul_Christiano on Issues with Using Willingness-to-Pay as a Primary Tool for Welfare Analysis · 2021-06-28T00:14:39.600Z · EA · GW

For governments who have the option to tax, WTP has obvious relevance as a way of comparing a policy to a benchmark of taxation+redistribution. I tentatively think that an idealized state (representing any kind of combination of its constituents' interests) ought to use a WTP analysis for almost all of its policy decisions. I wrote some opinionated thoughts here.

It's less clear if this is relevant for a realistic, state and the discussion becomes more complex. I think it depends on a question like "what is the role of cost-effectiveness analysis in contexts where it is a relatively minor input  into decision-making?" I think realistically there will be different kinds of cost-benefit analyses for different purposes.  Sometimes WTP will be appropriate but probably not most of the time. When those other analyses depend on welfare, I expect they can often be productively framed as "WTP x (utility/$)" with some reasonable estimate for utility/$. But even that abstraction will often break down in cases where WTP is hard-to-observe or beneficiaries are irrational or whatever.

I think for a philanthropist WTP isn't compelling as a metric, and should usually be combined with an explicit estimate of (utility/$). I don't think I've seen philanthropists using WTP in this way and certainly wouldn't expect to see someone suggesting that handing money to rich people is more effective since it can be done with lower overhead.

Comment by Paul_Christiano on Draft report on existential risk from power-seeking AI · 2021-05-03T19:31:49.566Z · EA · GW

A 5% probability of disaster isn't any more or less confident/extreme/radical than a 95% probability of disaster; in both cases you're sticking your neck out to make a very confident prediction.

"X happens" and "X doesn't happen" are not symmetrical once I know that X is a specific event. Most things at the level of specificity of "humans build an AI that outmaneuvers humans to permanently disempower them" just don't happen.

The reason we are even entertaining this scenario is because of a special argument that it seems very plausible. If that's all you've got---if there's no other source of evidence than the argument---then you've just got to start talking about the probability that the argument is right.

And the argument actually is a brittle and conjunctive thing. (Humans do need to be able to build such an AI by the relevant date, they do need to decide to do so, the AI they build does need to decide to disempower humans notwithstanding a prima facie incentive for humans to avoid that outcome.)

That doesn't mean this is the argument or that the argument is brittle in this way---there might be a different argument that explains in one stroke why several of these things will happen. In that case, it's going to be more productive to talk about that.

(For example, in the context of the multi-stage argument undershooting success probabilities, it's that people will be competently trying to achieve X and most of uncertainty is estimating how hard and how effectively people are trying---which is correlated across steps. So you would do better by trying to go for the throat and reason about the common cause of each success, and you will always lose if you don't see that structure.)

And of course some of those steps may really just be quite likely and one shouldn't be deterred from putting high probabilities on highly-probable things. E.g. it does seem like people have a very strong incentive to build powerful AI systems (and moreover the extrapolation suggesting that we will be able to build powerful AI systems is actually about the systems we observe in practice and already goes much of the way to suggesting that we will do so). Though I do think that the median MIRI staff-member's view is overconfident on many of these points.

Comment by Paul_Christiano on Dutch anti-trust regulator bans pro-animal welfare chicken cartel · 2021-02-25T17:41:26.548Z · EA · GW

Is your impression that if customers were willing to pay for it, then that wouldn't be sufficient cause to say that it benefited customers? (Does that mean that e.g. a standard ensuring that children's food doesn't cause discomfort also can't be protected, since it benefits customers' kids rather than customers themselves?)

Comment by Paul_Christiano on Dutch anti-trust regulator bans pro-animal welfare chicken cartel · 2021-02-24T16:37:48.881Z · EA · GW

These cases are also interesting for alignment agreements between AI labs, and it's interesting to see it playing out in practice. Cullen wrote about this here much better than I will.

Roughly speaking, if individual consumers would prefer use a riskier AI (because costs are externalized) then it seems like an agreement to make AI safer-but-more-expensive would run afoul of the same principles as this chicken-welfare agreement.

On paper, there are some reasons that the AI alignment case should be easier than the chicken-welfare case: (i) using unsafe AI hurts non-customer humans, and AI customers care more about other humans than they do about chickens, (ii) deploying unaligned AI actually likely hurts other AI customers in particular (since they will be the main ones competing with the unaligned but more sophisticated AI). So it's likely that every individual AI customer would  benefit.

Unfortunately, it seems like the same thing could be true in the chicken case---every individual customer could prefer the world with the welfare agreement---and it wouldn't change the regulator's decision.

For example, suppose that Dutch consumers eat 100 million chickens a year, 10/year for each of 10 million customers. Customer surveys discover that customers would only be willing to pay $0.01 for a chicken to have more space and a slightly longer life, but that these reforms increase chicken prices by $1. So they strike down the reform.

But with welfare standards in place, each customer pays an extra $10/year for chicken and 100 million chickens have improved lives, with a cost per chicken of less than $0.0000001/chicken, thousands of times lower than their WTP. (This is the same dynamic described here.) So every chicken consumer prefers the world where the standards are in place, despite not being willing to pay money to improve the lives of the tiny number of chickens they eat personally. This seems to be a very common reaction to discussions of animal welfare ("what difference does my consumption make? I can't change the way most chickens are treated...")

Because the number of chicken-eaters is so large, the relevant question in the survey should be "Would you prefer that someone else pay $X in order to improve chicken welfare?", making a tradeoff between two strangers. That's the relevant question for them, since the welfare standards mostly affect other people.

Analogously, if you ask AI consumers "Would you prefer have an aligned AI, or a slightly more sophisticated unaligned AI?" they could easily all say "I want the more sophisticated one," even if every single human would be better off if there were an agreement to make only aligned AI. If an anti-trust regulator used the same standard as in this case, it seems like they would throw out an alignment agreement because of that, even knowing that it would make every single human worse off.

I still think in practice AI alignment agreements would be fine for a variety of reasons. For example, I think if you ran a customer survey it's likely people would say they prefer use aligned AI even if it would disadvantage them personally because public sentiment towards AI is very different and the regulatory impulse is stronger. (Though I find it hard to believe that anything would end up hinging on such a survey, and even more strongly I think it would never come to this because there would be much less political pressure to enforce anti-trust.)

Comment by Paul_Christiano on Alternatives to donor lotteries · 2021-02-17T19:45:53.807Z · EA · GW

I guess I wouldn't recommend the donor lottery to people who wouldn't be happy entering a regular lottery for their charitable giving

Strong +1.

If I won a donor lottery, I would consider myself to have no obligation whatsoever towards the other lottery participants, and I think many other lottery participants feel the same way. So it's potentially quite bad if some participants are thinking of me  as an "allocator" of their money. To the extent there is ambiguity in the current setup, it seems important to try to eliminate that.

Comment by Paul_Christiano on [Link post] Are we approaching the singularity? · 2021-02-15T18:50:22.427Z · EA · GW
  1. I think that acceleration is autocorrelated---if things are accelerating rapidly at time T they are also more likely to be accelerating rapidly at time T+1. That's intuitively pretty likely, and it seems to show up pretty strongly in the data. Roodman makes no attempt to model it, in the interest of simplicity and analytical tractability. We are currently in a stagnant period, and so I think you should expect continuing stagnation. I'm not sure exactly how large the effect (and obviously it depends on the model) is but I think it's at least a 20-40 year delay. (There are two related angles to get a sense for the effect: one is to observe that autocorrelations seem to fade away on the timescale of a few doublings, rather than being driven by some amount of calendar time, and the other is to just look at the fact that we've had something like ~40 years of relative stagnation.)
  2. I think it's plausible that historical acceleration is driven by population growth, and that just won't really happen going forward. So at a minimum we should be uncertain betwe3en roodman's model and one that separates out population explicitly, which will tend to stagnate around the time population is limited by fertility rather than productivity.

(I agree with Max Daniel below that I don't think that Nordhaus' methodology is inherently more trustworthy. I think it's dealing with a relatively small amount of pretty short-term data, and is generally using a much more opinionated model of what technological change would look like.)

Comment by Paul_Christiano on [Link post] Are we approaching the singularity? · 2021-02-13T16:57:34.624Z · EA · GW

The relevant section is VII. Summarizing the six empirical tests:

  1. You'd expect productivity growth to accelerate as you approach the singularity, but it is slowing.
  2. The capital share should approach 100% as you approach the singularity. The share is growing, but at the slow rate of ~0.5%/year. At that rate it would take roughly 100 years to approach 100%.
  3. Capital should get very cheap as you approach the singularity. But capital costs (outside of computers) are falling relatively slowly.
  4. The total stock of capital should get large as you approach the singularity. In fact the stock of capital is slowly falling relative to output.
  5. Information should become an increasingly important part of the capital stock as you approach the singularity. This share is increasing, but will also take >100 years to become dominant.
  6. Wage grow should accelerate as you approach the singularity, but it is slowing.

I would group these into two basic classes of evidence:

  • We aren't getting much more productive, but that's what a singularity is supposed to be all about.
  • Capital and IT extrapolations are potentially compatible with a singularity, but only a timescale of 100+ years.

I'd agree that these seem like two points of evidence against singularity-soon, and I think that if I were going on outside-view economic arguments I'd probably be <50% singularity by 2100. (Though I'd still have a meaningful probability soon, and even at 100 years the prospect of a singularity would be one of the most important facts about the basic shape of the future.)

There are some more detailed aspects of the model that I don't buy, e.g. the very high share of information capital and persistent slow growth of physical capital. But I don't think they really affect the bottom line.

Comment by Paul_Christiano on Three Impacts of Machine Intelligence · 2021-02-13T02:03:15.499Z · EA · GW

If the market can't price 30-year cashflows, it can't price anything, since for any infinitely-lived asset (eg stocks!), most of the present-discounted value of future cash flows is far in the future. 

If an asset pays me far in the future,  then long-term interest rates are one factor affecting its price. But it seems to me that in most cases that factor still explains a minority of variation in prices (and because it's a slowly-varying factor it's quite hard to make money by predicting it).

For example, there is a ton of uncertainty about how much money any given company is going to make next year. We get frequent feedback signals about those predictions, and people who win bets on them immediately get returns that let them show how good they are and invest more, and so that's the kind of case where I'd be more scared of outpredicting the market.

So I guess that's saying that I expect the relative prices of stocks to be much more efficient than the absolute level.

See eg this Ralph Koijen thread and linked paper, "the first 10 years of dividends only make up ~20% of the value of the stock market. 80% is due to value of cash flows beyond 10 years"

Haven't looked at the claim but it looks kind of misleading. Dividend yield for SPY is <2% which I guess is what they are talking about? But buyback yield is a further 3%, and with a 5% yield you're getting 40% of the value in the first 10 years, which sounds more like it. So that would mean that you've gotten half of the value within 13.5 years instead of 31 years.

Technically the stock is still valued based on the future dividends, and a buyback is just decreasing outstanding shares and so increasing earnings per share. But for the purpose of pricing the stock it should make no difference whether earnings are distributed as dividends or buybacks, so the fact that buybacks push cashflows to the future can't possibly affect the difficulty of pricing stocks.

Put a different way, the value of a buyback to investors doesn't depend on the actual size of future cashflows, nor on the discount rate. Those are both cancelled out because they are factored into the price at which the company is able to buy back its shares. (E.g. if PepsiCo was making all of its earnings in the next 5 years, and ploughing them into buybacks, after which they made a steady stream of not-much-money, then PepsiCo prices would still be equal to the NPV of dividends, but the current PepsiCo price would just be an estimate of earnings over the next 5 years and would have almost no relationship to long-term interest rates.)

Even if this is right it doesn't affect your overall point too much though, since 10-20 year time horizons are practically as bad as 30-60 year time horizons.

Comment by Paul_Christiano on Three Impacts of Machine Intelligence · 2021-02-12T17:59:22.351Z · EA · GW

I I think the market just doesn't put much probability on a crazy AI boom anytime soon. If you expect such a boom then there are plenty of bets you probably want to make. (I am personally short US 30-year debt, though it's a very small part of my AI-boom portfolio.)

I think it's very hard for the market to get 30-year debt prices right because the time horizons are so long and they depend on super hard empirical questions with ~0 feedback. Prices are also determined by supply and demand across a truly huge number of traders, and making this trade locks up your money forever and can't be leveraged too much. So market forecasts are basically just a reflection of broad intellectual consensus about the future of growth (rather than views of the "smart money" or anything), and the mispricing is just a restatement of the fact that AI-boom is a contrarian position.

Comment by Paul_Christiano on AGB's Shortform · 2021-01-06T03:21:46.099Z · EA · GW

Some scattered thoughts (sorry for such a long comment!). Organized in order rather than by importance---I think the most important argument for me is the analogy to computers.

  • It's possible to write "Humanity survives the next billion years" as a conjunction of a billion events (humanity survives year 1, and year 2, and...). It's also possible to write "humanity goes extinct next year" as a conjunction of a billion events (Alice dies, and Bob dies, and...). Both of those are quite weak prima facie justifications for assigning high confidence. You could say that the second conjunction is different, because the billionth person is very likely to die once the others have died (since there has apparently been some kind of catastrophe), but the same is true for survival. In both cases there are conceivable events that would cause every term of the conjunction to be true,  and we need to address the probability of those common causes directly. Being able to write the claim as a conjunction doesn't seem to help you get to extreme probabilities without an argument about independence.
  • I feel you should be very hesitant to assign 99%+ probabilities without a good argument, and I don't think this is about anchoring to percent. The burden of proof gets stronger and stronger as you move closer to 1, and 100 is getting to be a big number. I think this is less likely to be a tractable disagreement than the other bullets but it seems worth mentioning for completeness. I'm curious if  you think there are other natural statements where the kind of heuristic you are describing (or any other similarly abstract heuristic) would justifiably get you to such high confidences. I agree with Max Daniel's point that it doesn't work for realistic versions of claims like "This coin will come up heads 30 times in a row." You say that it's not exclusive to simplified models but I think I'd be similarly skeptical of any application of this principle. (More generally, I think it's not surprising to assign very small probabilities to complex statements based on weak evidence, but that it will happen much more rarely for simple statements. It doesn't seem promising to get into that though.)
  • I think space colonization is probably possible, though getting up to probabilities like 50% for space colonization feasibility would be a much longer discussion. (I personally think >50% probability is much more reasonable than <10%.) If there is a significant probability that we colonize space, and that spreading out makes the survival of different colonists independent (as it appears it would), then it seems like we end up with some significant probability of survival. That said, I would also assign ~1/2 probability to surviving a billion years even if we were confined to Earth. I could imagine being argued down to 1/4 or even 1/8 but each successive factor of 2 seems much harder. So in some sense the disagreement isn't really about colonization.
  • Stepping back, I think the key object-level questions are something like "Is there any way to build a civilization that is very stable?" and "Will people try?" It seems to me you should have a fairly high probability on "yes" to both questions. I don't think you have to invoke super-aligned AI to justify that conclusion---it's easy to imagine organizing society in a way which drives existing extinction risks to negligible levels, and once that's done it's not clear where you'd get to 90%+ probabilities for new risks emerging that are much harder to reduce. (I'm not sure which step of this you get off the boat for---is it that you can't imagine a world that say reduced the risk of an engineered pandemic killing everyone to < 1/billion per year? Or that you think it's very likely other much harder-to-reduce risks would emerge?)
  • A lot of this is about burden of proof arguments. Is the burden of proof on someone to exhibit a risk that's very hard to reduce, or someone to argue that there exists no risk that is hard to reduce? Once we're talking about 10% or 1% probabilities it seems clear to me that the burden of proof is on the confident person. You could try to say "The claim of 'no bad risks' is a conjunction over all possible risks, so it's pretty unlikely" but I could just as well say "The claim about 'the risk is irreducible' is a conjunction over all possible reduction strategies, so it's pretty unlikely" so I don't think this gets us out of the stalemate (and the stalemate is plenty to justify uncertainty).
  • I do furthermore think that we can discuss concrete (kind of crazy) civilizations that are likely to have negligible levels of risk, given that e.g. (i) we have existence proofs for highly reliable machines over billion-year timescales, namely life, (ii) we have existence proofs for computers if you can build reliable machinery of any kind, (iii) it's easy to construct programs that appear to be morally relevant but which would manifestly keep running indefinitely.  We can't get too far with this kind of concrete argument, since any particular future we can imagine is bound to be pretty unlikely. But it's relevant to me that e.g. stable-civilization scenarios seem about as gut-level plausible to me as non-AI extinction scenarios do in the 21st century.
  • Consider the analogous question "Is it possible to build computers that successfully carry out trillions of operations without errors that corrupt the final result?" My understanding is that in the early 20th century this question was seriously debated (though that's not important to my point), and it feels very similar to your question. It's very easy for a computational error to cascade and change the final result of a computation. It's possible to take various precautions to reduce the probability of an uncorrected error, but why think that it's possible to reduce that risk to levels lower than 1 in a trillion, given that all observed computers have had fairly high error rates? Moreover, it seems that error rates are growing  as we build bigger and bigger computers, since each element has an independent failure rate, including the machinery designed to correct errors. To really settle this we need to get into engineering details, but until you've gotten into those details I think it's clearly unwise to assign very low probability to building a computer that carries out trillions of steps successfully---the space of possible designs is large and people are going to try to find one that works, so you'd need to have some good argument about why to be confident that they are going to fail.
  • You could say that computers are an exceptional example I've chosen with hindsight. But I'm left wondering if there are any valid applications of this kind of heuristic--what's the reference class of which "highly reliable computers" are exceptional rather than typical?
  • If someone said:"A billion years is a long time. Any given thing that can plausibly happen should probably be expected to happen over that time period" then I'd ask about why life survived the last billion years.
  • You could say that "a billion years" is a really long time for human civilization (given that important changes tend to happen within decades or centuries) but not a long time for intelligent life (given that important changes takes millions of years). This is similar to what happens if you appeal to current levels of extinction risk being really high. I don't buy this because life on earth is currently at a period of unprecedentedly rapid change. You should have some reasonable probability of returning to more historically typical timescales of hundreds of millions of years, which in turn gives you a reasonable overall probability on surviving for hundreds of millions of years. (Actually I think we should have >50% probabilities for reversion to lower timescales, since we can tell that the current period of rapid growth will soon be over. Over our history rapid change and rapid growth have basically coincided, so it's particularly plausible that returning to slow-growth will also return to slow-change.)
  • Applying the rule of thumb for estimating lifetimes to "the human species" rather than "intelligent life" seems like it's doing a huge amount of work. It might be reasonable to do the extrapolation using some mixture between these reference classes (and others), but in order to get extreme probabilities for extinction you'd need to have an extreme mixture. This is part of the general pattern why you don't usually end up with 99% probabilities for interesting questions without real arguments---you need to not only have a way of estimating that has very high confidence, you need to be very confident in that way of estimating.
  • You could appeal to some similar outside view to say "humanity will undergo changes similar in magnitude to those that have occurred over the last billion years;" I think that's way more plausible (though I still wouldn't believe 99%) but I don't think that it matters for claims about the expected moral value of the future.
  • The doomsday argument can plausibly arrive at very high confidences based on anthropic considerations (if you accept those anthropic principles with very high confidence). I think many long-termists would endorse the conclusion that the vast majority of observers like us do not actually live in a large and colonizable universe---not at 99.999999% but at least at 99%. Personally I would reject the inference that we probably don't live in a large universe because I reject the implicit symmetry principle. At any rate, these lines of argument go in a rather different direction than the rest of your post and I don't feel like it's what you are getting at.
Comment by Paul_Christiano on Against GDP as a metric for timelines and takeoff speeds · 2020-12-30T00:03:28.544Z · EA · GW

Scaling down all the amounts of time, here's how that situation sounds to me: US output doubles in 15 years (basically the fastest it ever has), then doubles again in 7 years. The end of the 7 year doubling is the first time that your hypothetical observer would say "OK yeah maybe we are transitioning to a new faster growth mode," and stuff started getting clearly crazy during the 7 year doubling. That scenario wouldn't be surprising to me. If that scenario sounds typical to you then it's not clear there's anything we really disagree about.

Moreover, it seems to contradict your claim that 0.14% growth was already high by historical standards.

0.14%/year growth sustained over 500 years is a doubling. If you did that between 5000BC and 1000AD then that would be 4000x growth. I think we have a lot of uncertainty about how much growth actually occurred but we're pretty sure it's not 4000x (e.g. going from 1 million people to 4 billion people). Standard kind of made-up estimates are more like 50x (e.g. those cited in Roodman's report), half that fast.

There is lots of variance in growth rates, and it would temporarily be above that level given that populations would grow way faster than that when they have enough resources. That makes it harder to tell what's going on but I think you should still be surprised to see such high growth rates sustained for many centuries.

(assuming you discount 1350 as I do as an artefact of recovering from various disasters

This doesn't seem to work, especially if you look at the UK. Just consider a long enough period of time (like 1000AD to 1500AD) to include both the disasters and the recovery. At that point, disasters should if anything decrease growth rates. Yet this period saw historically atypically fast growth.

Comment by Paul_Christiano on Against GDP as a metric for timelines and takeoff speeds · 2020-12-29T19:42:34.552Z · EA · GW

Some thoughts on the historical analogy:

If you look at the graph at the 1700 mark, GWP is seemingly on the same trend it had been on since antiquity. The industrial revolution is said to have started in 1760, and GWP growth really started to pick up steam around 1850. But by 1700 most of the Americas, the Philippines and the East Indies were directly ruled by European powers

I think European GDP was already pretty crazy by 1700. There's been a lot of recent arguing about the particular numbers and I am definitely open to just being wrong about this, but so far nothing has changed my basic picture.

After a minute of thinking my best guess for finding the most reliable time series was from the Maddison project. I pulled their dataset from here.

Here's UK population:

  • 1000AD: 2 million
  • 1500AD: 3.9 million (0.14%/year growth)
  • 1700AD: 8.6 million (0.39%)
  • 1820AD: 21.2 million (0.76%)

A 0.14%/year growth rate was already very fast by historical standards, and by 1700 things seemed really crazy.

Here's population in Spain:

  • 1000AD: 4 million
  • 1500AD: 6.8 million (0.11%)
  • 1700AD: 8.8 million (0.13%)
  • 1820AD: 12.2 million (0.28%)

The 1500-1700 acceleration is less marked here but still seems like growth was fast.

Here's the world using the data we've all been using in the past (which I think is much more uncertain):

  • 10000BC: 4 million
  • 3000BC: 14 million (0.02%)
  • 1000BC: 50 million (0.06%)
  • 1000AD: 265 million (0.08%)
  • 1500AD: 425 million (0.09%)
  • 1700AD: 610 million (0.18%)
  • 1820AD: 1 billion (0.41%)

This puts the 0.14%/year growth in the UK in context, and also suggests that things were generally blowing up by 1700AD.

I think that looking at the country-level data is probably better since it's more robust, unless your objection is "GWP isn't what matters because some countries' GDP will be growing much faster."

Comment by Paul_Christiano on Some thoughts on the EA Munich // Robin Hanson incident · 2020-10-17T17:18:33.621Z · EA · GW

I'm not sure what difference in prioritization this would imply or if we have remaining quantitative disagreements. I agree that it is bad for important institutions to become illiberal or collapse and so erosion of liberal norms is worthwhile for some people to think about. I further agree that it is bad for me or my perspective to be pushed out of important institutions (though much less bad to be pushed out of EA than out of Hollywood or academia).

It doesn't currently seem like thinking or working on this issue should be a priority for me (even within EA other people seem to have clear comparative advantage over me). I would feel differently if this was an existential issue or had a high enough impact, and I mostly dropped the conversation when it no longer seemed like that was at issue / it seemed in the quantitative reference class of other kinds of political maneuvering. I generally have a stance of just doing my thing rather than trying to play expensive political games, knowing that this will often involve losing political influence.

It does feel like your estimates for the expected harms are higher than mine, which I'm happy enough to discuss, but I'm not sure there's a big disagreement (and it would have to be quite big to change my bottom line).

I was trying to get at possible quantitative disagreements by asking things like "what's the probability that making pro-speech comments would itself be a significant political liability at some point in the future?" I think I have a probability of perhaps 2-5% on "meta-level pro-speech comments like this one eventually become a big political liability and participating in such discussions causes Paul to miss out on at least one significant opportunity to do good or have influence."

I'm always interested in useful thoughts about cost-effective things to do. I could also imagine someone making the case that "think about it more" is cost-effective for me, but I'm more skeptical of that (I expect they'd instead just actually do that thinking and tell me what they think I should do differently as a result, since the case for them thinking will likely be much better than the case for me doing it). I think your earlier comments make sense from the perspective of trying to convince other folks here to think about these issues and I didn't intend for the grandparent to be pushing against that.

For me it seems like one easy and probably-worthwhile intervention is to (mostly) behave according to a set of liberal norms that I like (and I think remain very popular) and to be willing to pay costs if some people eventually reject that behavior (confident that there will be other communities that have similar liberal norms). Being happy to talk openly about "cancel culture" is part of that easy approach, and if that led to serious negative consequences then it would be a sign that the issue is much more severe than I currently believe and it's more likely I should do something. In that case I do think it's clear there is going to be a lot of damage, though again I think we differ a bit in that I'm more scared about the health of our institutions than people like me losing influence.

Comment by Paul_Christiano on Hiring engineers and researchers to help align GPT-3 · 2020-10-09T16:37:46.088Z · EA · GW

My process was to check the "About the forum" link on the left hand side, see that there was a section on "What we discourage" that made no mention of hiring, then search for a few job ads posted on the forum and check that no disapproval was expressed in the comments of those posts.

Comment by Paul_Christiano on Hiring engineers and researchers to help align GPT-3 · 2020-10-05T19:43:28.252Z · EA · GW

I think that a scaled up version of GPT-3 can be directly applied to problems like "Here's a situation. Here's the desired result. What action will achieve that result?" (E.g. you can already use it to get answers like "What copy will get the user to subscribe to our newsletter?" and we can improve performance by fine-tuning on data about actual customer behavior or by combining GPT-3 with very simple search algorithms.)

I think that if GPT-3 was more powerful then many people would apply it to problems like that. I'm concerned that such systems will then be much better at steering the future than humans are, and that none of these systems will be actually trying to help people get what they want.

A bunch of people have written about this scenario and whether/how it could be risky. I wish that I had better writing to refer people to. Here's a post I wrote last year to try to communicate what I'm concerned about.

Comment by Paul_Christiano on Hiring engineers and researchers to help align GPT-3 · 2020-10-05T19:34:57.666Z · EA · GW

Hires would need to be able to move to the US.

Comment by Paul_Christiano on Hiring engineers and researchers to help align GPT-3 · 2020-10-05T19:34:29.258Z · EA · GW

No, I'm talking somewhat narrowly about intent alignment, i.e. ensuring that our AI system is "trying" to do what we want. We are a relatively focused technical team, and a minority of the organization's investment in safety and preparedness.

The policy team works on identifying misuses and developing countermeasures, and the applied team thinks about those issues as they arise today.

Comment by Paul_Christiano on Some thoughts on the EA Munich // Robin Hanson incident · 2020-09-26T01:59:48.704Z · EA · GW
The conclusion I draw from this is that many EAs are probably worried about CC but are afraid to talk about it publicly because in CC you can get canceled for talking about CC, except of course to claim that it doesn't exist. (Maybe they won't be canceled right away, but it will make them targets when cancel culture gets stronger in the future.) I believe that the social dynamics leading to development of CC do not depend on the balance of opinions favoring CC, and only require that those who are against it are afraid to speak up honestly and publicly (c.f. "preference falsification"). That seems to already be the situation today.

It seems possible to me that many institutions (e.g. EA orgs, academic fields, big employers, all manner of random FB groups...) will become increasingly hostile to speech or (less likely) that they will collapse altogether.

That does seem important. I mostly don't think about this issue because it's not my wheelhouse (and lots of people talk about it already). Overall my attitude towards it is pretty similar to other hypotheses about institutional decline. I think people at EA orgs have way more reasons to think about this issue than I do, but it may be difficult for them to do so productively.

If someone convinced me to get more pessimistic about "cancel culture" then I'd definitely think about it more. I'd be interested in concrete forecasts if you have any. For example, what's the probability that making pro-speech comments would itself be a significant political liability at some point in the future? Will there be a time when a comment like this one would be a problem?

Looking beyond the health of existing institutions, it seems like most people I interact with are still quite liberal about speech, including a majority of people who I'd want to work with, socialize with, or take funding from. So hopefully the endgame boils down to freedom of association. Some people will run a strategy like "Censure those who don't censure others for not censuring others for problematic speech" and take that to its extreme, but the rest of the world will get along fine without them and it's not clear to me that the anti-speech minority has anything to do other than exclude people they dislike (e.g. it doesn't look like they will win elections).

in CC you can get canceled for talking about CC, except of course to claim that it doesn't exist. (Maybe they won't be canceled right away, but it will make them targets when cancel culture gets stronger in the future.)

I don't feel that way. I think that "exclude people who talk openly about the conditions under which we exclude people" is a deeply pernicious norm and I'm happy to keep blithely violating it. If a group excludes me for doing so, then I think it's a good sign that the time had come to jump ship anyway. (Similarly if there was pressure for me to enforce a norm I disagreed with strongly.)

I'm generally supportive of pro-speech arguments and efforts and I was glad to see the Harper's letter. If this is eventually considered cause for exclusion from some communities and institutions then I think enough people will be on the pro-speech side that it will be fine for all of us.

I generally try to state my mind if I believe it's important, don't talk about toxic topics that are unimportant, and am open about the fact that there are plenty of topics I avoid. If eventually there are important topics that I feel I can't discuss in public then my intention is to discuss them.

I would only intend to join an internet discussion about "cancellation" in particularly extreme cases (whether in terms of who is being canceled, severe object-level consequences of the cancellation, or the coercive rather than plausibly-freedom-of-association nature of the cancellation).

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-09T20:03:38.530Z · EA · GW

Thanks, super helpful.

(I don't really buy an overall take like "It seems unlikely" but it doesn't feel that mysterious to me where the difference in take comes from. From the super zoomed out perspective 1200 AD is just yesterday from 1700AD, it seems like random fluctuations over 500 years are super normal and so my money would still be on "in 500 years there's a good chance that China would have again been innovating and growing rapidly, and if not then in another 500 years it's reasonably likely..." It makes sense to describe that situation as "nowhere close to IR" though. And it does sound like the super fast growth is a blip.)

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-09T14:56:59.169Z · EA · GW

I took numbers from Wikipedia but have seen different numbers that seem to tell the same story although their quantitative estimates disagree a ton.

The first two numbers are all higher than growth rates could have plausibly been in a sustained way during any previous part of history (and the 0-1000AD one probably is as well), and they seem to be accelerating rather than returning to a lower mean (as must have happened during any historical period of similar growth).

My current view is that China was also historically unprecedented at that time and probably would have had an IR shortly after Europe. I totally agree that there is going to be some mechanistic explanation for why europe caught up with and then overtook china, but from the perspective of the kind of modeling we are discussing I feel super comfortable calling it noise (and expecting similar "random" fluctuations going forward that also have super messy contingent explanations).

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-09T14:47:20.702Z · EA · GW

If one believed the numbers on wikipedia, it seems like Chinese growth was also accelerating a ton and it was not really far behind on the IR, such that I wouldn't except to be able to easily eyeball the differences.

If you are trying to model things at the level that Roodman or I are, the difference between 1400 and 1600 just isn't a big deal, the noise terms are on the order of 500 years at that point.

So maybe the interesting question is if and why scholars think that China wouldn't have had an IR shortly after Europe (i.e. within a few centuries, a gap small enough that it feels like you'd have to have an incredibly precise model to be justifiably super surprised).

Maybe particularly relevant: is the claimed population growth from 1700-1800 just catch-up growth to Europe? (more than doubling in 100 years! And over the surrounding time period the observed growth seems very rapid even if there are moderate errors in the numbers) If it is, how does that work given claims that Europe wasn't so far ahead by 1700? If it isn't, then how does the that not very strongly suggest incredible acceleration in China, given that it had very recently had some of the fastest growth in history and is then experience even more unprecedented growth? Is it a sequence of measurement problems that just happen to suggest acceleration?

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-08T15:37:53.495Z · EA · GW
My model is that most industries start with fast s-curve like growth, then plateau, then often decline

I don't know exactly what this means, but it seems like most industries in the modern world are characterized by relatively continuous productivity improvements over periods of decades or centuries. The obvious examples to me are semiconductors and AI since I deal most with those. But it also seems true of e.g. manufacturing, agricultural productivity, batteries, construction costs. It seems like industries where the productivity vs time curve is a "fast S-curve" are exceptional, which I assume means we are somehow reading the same data differently. What kind of industries would you characterize this way?

(I agree that e.g. "adoption" is more likely to be an s-curve given that it's bounded, but productivity seems like the analogy for growth rates.)

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-08T15:24:51.649Z · EA · GW

It feels like you are drawing some distinction between "contingent and complicated" and "noise." Here are some possible distinctions that seem relevant to me but don't actually seem like disagreements between us:

  • If something is contingent and complicated, you can expect to learn about it with more reasoning/evidence, whereas if it's noise maybe you should just throw up your hands. Evidently I'm in the "learn about it by reasoning" category since I spend a bunch of time thinking about AI forecasting.
  • If something is contingent and complicated, you shouldn't count on e.g. the long-run statistics matching the noise distribution---there are unmodeled correlations (both real and subjective). I agree with this and think that e.g. the singularity date distributions (and singularity probability) you get out of Roodman's model are not trustworthy in light of that (as does Roodman).

So it's not super clear there's a non-aesthetic difference here.

If I was saying "Growth models imply a very high probability of takeoff soon" then I can see why your doc would affect my forecasts. But where I'm at from historical extrapolations is more like "maybe, maybe not"; it doesn't feel like any of this should change that bottom line (and it's not clear how it would change that bottom line) even if I changed my mind everywhere that we disagree.

"Maybe, maybe not" is still a super important update from the strong "the future will be like the recent past" prior that many people implicitly have and I might otherwise take very seriously. It also leads me to mostly dismiss arguments like "this is obviously not the most important century since most aren't." But it mostly means that I'm actually looking at what is happening technologically.

You may be responding to writing like this short post where I say "We have been in a period of slowing growth for the last forty years. That’s a long time, but looking over the broad sweep of history I still think the smart money is on acceleration eventually continuing, and seeing something like [hyperbolic growth]...". I stand by the claim that this is something like the modal guess---we've had enough acceleration that the smart money is on it continuing, and this seems equally true on the revolutions model. I totally agree that any specific thing is not very likely to happen, though I think it's my subjective mode. I feel fine with that post but totally agree it's imprecise and this is what you get for being short.

The story with fossil fuels is typically that there was a pre-existing economic efflorescence that supported England's transition out of an 'organic economy.' So it's typically a sort of tipping point story, where other factors play an important role in getting the economy to the tipping point.

OK, but if those prior conditions led to a great acceleration before the purported tipping point, then I feel like that's mostly what I want to know about and forecast.

Supposing we had accurate data, it seems like the best approach is running a regression that can accommodate either hyperbolic or exponential growth — plus noise — and then seeing whether we can object the exponential hypothesis. Just noting that the growth rate must have been substantially higher than average within one particular millennium doesn’t necessarily tell us enough; there’s still the question of whether this is plausibly noise.

I don't think that's what I want to do. My question is, given a moment in history, what's the best way to guess whether and in how long there will be significant acceleration? If I'm testing the hypothesis "The amount of time before significant acceleration tends to be a small multiple of the current doubling time" then I want to look a few doublings ahead and see if things have accelerated, averaging over a doubling (etc. etc.), rather than do a regression that would indirectly test that hypothesis by making additional structural assumptions + would add a ton of sensitivity to noise.

You don’t need a story about why they changed at roughly the same time to believe that they did change at roughly the same time (i.e. over the same few century period). And my impression is that that, empirically, they did change at roughly the same time. At least, this seems to be commonly believed.
I don’t think we can reasonably assume they’re independent. Economic histories do tend to draw casual arrows between several of these differences, sometimes suggesting a sort of chain reaction, although these narrative causal diagrams are admittedly never all that satisfying; there’s still something mysterious here. On the other hand, higher population levels strike me as a fairly unsatisfying underlying cause.

It looked like you were listing those things to help explain why you have a high prior in favor of discontinuities between industrial and agricultural societies. "We don't know why those things change together discontinuously, they just do" seems super reasonable (though whether that's true is precisely what's at issue). But it does mean that listing out those factors adds nothing to the a priori argument for discontinuity.

Indeed, if you think that all of those are relevant drivers of growth rates then all else equal I'd think you'd expect more continuous progress, since all you've done is rule out one obvious way that you could have had discontinuous progress (namely by having the difference be driven by something that had a good prima facie reason to change discontinuously, as in the case of the agricultural revolution) and now you'll have to posit something mysterious to get to your discontinuous change.

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-08T15:09:35.106Z · EA · GW

I think Roodman's model implies a standard deviation of around 500-1000 years for IR timing starting from 1000AD, but I haven't checked. In general for models of this type it seems like the expected time to singularity is a small multiple of the current doubling time, with noise also being on the order of the doubling time.

The model clearly underestimates correlations and hence the variance here---regardless of whether we go in for "2 revolutions" or "randomly spread out" we can all agree that a stagnant doubling is more likely to be followed by another stagnant doubling and vice versa, but the model treats them as independent.

(As one particular contingency you mention: It seems super plausible to me, especially, that if the Americas didn't turn out to exist, then the Industrial Revolution would have happened much later. But this seems like a pretty random/out-of-model fact about the world.)

This seems to suggest there are lots of civilizations like Europe-in-1700. But it seems to me that by this time (and so I believe before the Americas had any real effect) Europe's state of technological development was already pretty unprecedented. This is lot of what makes many of the claims about "here's why the IR happened" seem dubious to me.

My sense of that comes from: (i) in growth numbers people usually cite, Europe's growth was absurdly fast from 1000AD - 1700AD (though you may think those numbers are wrong enough to bring growth back to a normal level) (ii) it seems like Europe was technologically quite far ahead of other IR competitors.

I'm curious about your take. Is it that:

  • The world wasn't yet historically exceptional by 1700, there have been other comparable periods of rapid progress. (What are the historical analogies and how analogous do you think they are? Is my impression of technological sophistication wrong?)
  • 1700s Europe is quantitatively exceptional by virtue of being the furthest along example, but nevertheless there is a mystery to be explained about why it became even more exceptional rather than regressing to the mean (as historical exceptional-for-their-times civilizations had in the past). I don't currently see a mystery about this (given the level of noise in Roodman's model, which seems like it's going to be in the same ballpark as other reasonable models), but it may be because I'm not informed enough about those historical analogies.
  • Actually the IR may have been inevitable in 1700s Europe but the exact pace seems contingent. (This doesn't seem like a real tension with a continuous acceleration model.)
  • Actually the contingencies you have in mind were already driving the exceptional situation in 1700.
Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-07T23:01:09.249Z · EA · GW
I think that Hanson's "series of 3 exponentials" is the neatest alternative, although I also think it's possible that pre-modern growth looked pretty different from clean exponentials (even on average / beneath the noise). There's also a semi-common narrative in which the two previous periods exhibited (on average) declining growth rates, until there was some 'breakthrough' that allowed the growth rate to surge: I suppose this would be a "three s-curve" model. Then there's the possibility that the growth pattern in each previous era was basically a hard-to-characterize mess, but was constrained by a rough upper bound on the maximum achievable growth rate. This last possibility is the one I personally find most likely, of the non-hyperbolic possibilities.

It seems almost guaranteed that the data is a mess, it just seems like the only difference between the perspectives is "is acceleration fundamentally concentrated into big revolutions or is it just random and we can draw boundaries around periods of high-growth and call those revolutions?"

There may also be some fundamental meta-prior that matters, here, about the relative weight one ought to give to simple unified models vs. complex qualitative/multifactoral stories.

Which growth model corresponds to which perspective? I normally think of "'industry' is what changed and is not contiguous with what came before" as the single-factor model, and multifactor growth models tending more towards continuous growth.

A lot of my prior comes down to my impression that the dynamics of growth just *seem * very different to me for forager societies, agricultural/organic, and industrial/fossil-fuel societies.

I'm definitely much more sympathetic to the forager vs agricultural distinction.

Does a discontinuous change from fossil-fuel use even fit the data? It doesn't seem to add up at all to me (e.g. doesn't match the timing of acceleration, there are lots of industries that seemed to accelerate without reliance on fossil fuels, etc.), but would only consider a deep dive if someone actually wanted to stake something on that.

I don’t think the post-1500 data is too helpful help for distinguishing between the ‘long run trend’ and ‘few hundred year phase transition’ perspectives.
If there was something like a phase transition, from pre-modern agricultural societies to modern industrial societies, I don’t see any particular reason to expect the growth curve during the transition to look like the sum of two exponentials. (I especially don’t expect this at the global level, since diffusion dynamics are so messy.)

It feels to me like I'm saying: acceleration happens kind of randomly on a timescale roughly determined by the current growth rate. We should use the base rate of acceleration to make forecasts about the future, i.e. have a significant probability of acceleration during each doubling of output. (Though obviously the real model is more complicated and we can start deviating from that baseline, e.g. sure looks like we should have a higher probability of stagnation now given that we'e had decades of it.)

It feels to me like you are saying "No, we can have a richer model of historical acceleration that assigns significantly lower probability to rapid acceleration over the coming decades / singularity."

So to me it feels like as we add random stuff like "yeah there are revolutions but we don't have any prediction about what they will look like" makes the richer model less compelling. It moves me more towards the ignorant perspective of "sometimes acceleration happens, maybe it will happen soon?", which is what you get in the limit of adding infinitely many ex ante unknown bells and whistles to your model.

The papers typically suggest that the thing kicking off the growth surge, within a particular millennium, is the beginning of intensive agriculture in that region — so I don’t think the pivotal triggering event is really different.

Is "intensive agriculture" a well-defined thing? (Not rhetorical.) It didn't look like "the beginning of intensive agriculture" corresponds to any fixed technological/social/environmental event (e.g. in most cases there was earlier agriculture and no story was given about why this particular moment would be the moment), it just looked like it was drawn based on when output started rising faster.

I wouldn't necessarily say they were significantly faster. It depends a bit on exactly how you run this test, but, when I run a regression for "(dP/dt)/P = a*P^b" (where P is population) on the dataset up until 1700AD, I find that the b parameter is not significantly greater than 0. (The confidence interval is roughly -.2 to .5, with zero corresponding to exponential growth.)

I mean that if you have 5x growth from 0AD to 1700AD, and growth was at the same rate from 10000BC to 0AD, then you would expect 5^(10,000/1700) = 13,000-fold growth over that period. We have uncertainty about exactly how much growth there was in the prior period, but we don't have anywhere near that much uncertainty.

Doing a regression on yearly growth rates seems like a bad way to approach this. It seems like the key question is: did growth speed up a lot in between the agricultural and industrial revolutions? It seems like the way to pick that is to try to use points that are as spaced out as possible to compare growth rates in the beginning and late part of the interval from 10000BC to 1500AD. (The industrial revolution is usually marked much later, but for the purpose of the "2 revolutions" view I think you definitely need it to start by then.)

So almost all of the important measurement error is going to be in the bit of growth in the 0AD to 1500AD phase. If in fact there was only 2x growth in that period (say because the 0AD number was off by 50%) then that would only predict 100-fold growth from 10,000BC to 0AD, which is way more plausible.

The industrial era is, in comparison, less obviously different from the farming era, but it also seems pretty different. My list of pretty distinct features of pre-modern agricultural economies is: (a) the agricultural sector constituted the majority of the economy; (b) production and (to a large extent) transportation were limited by the availability of agricultural or otherwise ‘organic’ sources of energy (plants to power muscles and produce fertiliser); (c) transportation and information transmission speeds were largely limited by windspeed and the speed of animals; (d) nearly everyone was uneducated, poor, and largely unfree; (e) many modern financial, legal, and political institutions did not exist; (f) certain cultural attitudes (such as hatred of commerce and lack of belief in the possibility of progress) were much more common; and (g) scientifically-minded research and development projects played virtually no role in the growth process.

If you just keep listing things, it stops being a plausible source of a discontinuity---you then need to give some story for why your 7 factors all change at the same time. If they don't, e.g. if they just vary randomly, then you are going to get back to continuous change.

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-07T20:31:05.913Z · EA · GW
because I have a bunch of very concrete, reasonably compelling sounding stories of specific things that caused the relevant shifts

Be careful that you don't have too many stories, or it starts to get continuous again.

More seriously, I don't know what the small # of factors are for the industrial revolution, and my current sense is that the story can only seem simple for the agricultural revolution because we are so far away and ignoring almost all the details.

It seems like the only factor that looks a priori like it should cause a discontinuity is the transition from hunting+gathering to farming, i.e. if you imagine "total food" as the sum of "food we make" and "food we find" then there could be a discontinuous change in growth rates as "food we make" starts to become large relative to "food we find" (which bounces around randomly but is maybe not really changing). This is blurred because of complementarity between your technology and finding food, but certainly I'm on board with an in-principle argument for a discontinuity as the new mode overtakes the old one.

For the last 10k years my impression is that no one has a very compelling story for discontinuities (put differently: they have way too many stories) and it's mostly a stylized empirical fact that the IR is kind of discontinuous. But I'm provisionally on board with Ben's basic point that we don't really have good enough data to know whether growth had been accelerating a bunch in the run-up to the IR.

To the extent things are discontinuous, I'd guess that it's basically from something similar to the agricultural case---there is continuous growth and random variation, and you see "discontinuities" in the aggregate if a smaller group is significantly outpacing the world, so that by the time they become a large part of the world they are growing significantly faster.

I think this is also reasonably plausible in the AI case (e.g. there is an automated part of the economy doubling every 1-2 years, by the time it gets to be 10% of the economy it's driving +5%/year growth, 1-2 years later it's driving +10% growth). But I think quantitatively given the numbers involved and the actual degree of complementarity, this is still unlikely to give you a fast takeoff as I operationalized it. I think if we're having a serious discussion about "takeoff" that's probably where the action is, not in any of the kinds of arguments that I dismiss in that post.

I find the "but X has fewer parameters" argument only mildly compelling, because I feel like other evidence about similar systems that we've observed should easily give us enough evidence to overcome the difference in complexity. 

I mean something much more basic. If you have more parameters then you need to have uncertainty about every parameter. So you can't just look at how well the best "3 exponentials" hypothesis fits the data, you need to adjust for the fact that this particular "3 exponentials" model has lower prior probability. That is, even if you thought "3 exponentials" was a priori equally likely to a model with fewer parameters, every particular instance of 3 exponentials needs to be less probable than every particular model with fewer parameters.

The thing that on the margin would feel most compelling to me for the continuous view is something like a concrete zoomed in story of how you get continuous growth from a bunch of humans talking to each other and working with each other over a few generations, that doesn't immediately abstract things away into high-level concepts like "knowledge" and "capital". 

As far as I can tell this is how basically all industries (and scientific domains) work---people learn by doing and talk to each other and they get continuously better, mostly by using and then improving on technologies inherited from other people.

It's not clear to me whether you are drawing a distinction between modern economic activity and historical cultural accumulation, or whether you feel like you need to see a zoomed-in version of this story for modern economic activity as well, or whether this is a more subtle point about continuous technological progress vs continuous changes in the rate of tech progress, or something else.

Comment by Paul_Christiano on Does Economic History Point Toward a Singularity? · 2020-09-07T16:27:31.906Z · EA · GW

This would be an important update for me, so I'm excited to see people looking into it and to spend more time thinking about it myself.

High-level summary of my current take on your document:

  • I agree that the 1AD-1500AD population data seems super noisy.
  • Removing that data removes one of the datapoints supporting continuous acceleration (the acceleration between 10kBC - 1AD and 1AD-1500AD) and should make us more uncertain in general.
  • It doesn't have much net effect on my attitude towards continuous acceleration vs discontinuous jumps, this mostly pushes us back towards our prior.
  • I'm not very moved by the other evidence/arguments in your doc.

Here's how I would summarize the evidence in your document:

  • Much historical data is made up (often informed by the author's models of population dynamics), so we can't use it to estimate historical growth. This seems like the key point.
  • In particular, although standard estimates of growth from 1AD to 1500AD are significantly faster than growth between 10kBC and 1AD, those estimates are sensitive to factor-of-1.5 error in estimates of 1AD population, and real errors could easily be much larger than that.
  • Population levels are very noisy (in addition to population measurement being noisy) making it even harder to estimate rates.
  • Radiographic data often displays isolated periods of rapid growth from 10,000BC to 1AD and it's possible that average growth rates were something like 2000 year doubling. So even if 500-2000 year doubling times are accurate from 1AD to 1500, those may not be a deviation from the preceding period.
  • You haven't looked into the claims people have made about growth from 100kya to 10kya, but given what we know about measurement error from 10kya to now, it seems like the 100kya-10kya data is likely to be way too noisy to say anything about.

Here's my take in more detail:

  • You are basically comparing "Series of 3 exponentials" to a hyperbolic growth model. I think our default simple hyperbolic growth model should be the one in David Roodman's report (blog post), so I'm going to think about this argument as comparing Roodman's model to a series of 3 noisy exponentials. In your doc you often dunk on an extremely low-noise version of hyperbolic growth but I'm mostly ignoring that because I absolutely agree that population dynamics are very noisy.
  • It feels like you think 3 exponentials is the higher prior model. But this model has many more parameters to fit the data, and even ignoring that "X changes in 2 discontinuous jumps" doesn't seem like it has a higher prior than "X goes up continuously but stochastically." I think the only reason we are taking 3 exponentials seriously is because of the same kind of guesswork you are dismissive of, namely that people have a folk sense that the industrial revolution and agricultural revolutions were discrete changes. If we think those folk senses are unreliable, I think that continuous acceleration has the better prior. And at the very least we need to be careful about using all the extra parameters in the 3-exponentials model, since a model with 2x more parameters should fit the data much better.
  • On top of that, the post-1500 data is fit terribly by the "3 exponentials" model. Given that continuous acceleration very clearly applies in the only regime where we have data you consider reliable, and given that it already seemed simpler and more motivated, it seems pretty clear to me that it should have the higher prior, and the only reason to doubt that is because of growth folklore. You can't have it both ways in using growth folklore to promote this hypothesis to attention and then dismissing the evidence from growth folklore because it's folklore.
  • On the acceleration model, the periods from 1500-2000, 10kBC-1500, and "the beginning of history to 10kBC" are roughly equally important data (and if that hypothesis has higher prior I don't think you can reject that framing). Changes within 10kBC - 1500 are maybe 1/6th of the evidence, and 1/3 of the relevant evidence for comparing "continuous acceleration" to "3 exponentials." I still think it's great to dig into one of these periods, but I don't think it's misleading to present this period as only 1/3 of the data on a graph.
  • (Enough about priors, onto the data.)
  • I think that the key claim is that the 1AD-1500AD data is mostly unreliable. Without this data, we have very little information about acceleration from 10kBC - 1500AD, since the main thing we actually knew was that 1AD-1500AD must have been faster than the preceding 10k years. I'd like to look into that more, but it looks super plausible to me that the noise is 2x or more for 1AD which is enough to totally kill any inference about growth rates. So provisionally I'm inclined to accept your view there.
  • That basically removes 1 datapoint for the continuous acceleration story and I totally agree it should leave us more uncertain about what's going on. That said, throwing out all the numbers from that period also removes one of the main quantitative datapoints against continuous acceleration [ETA: the other big one being the modern "great stagnation," both of these are in the tails of the continuous acceleration story and are just in the middle of the constant exponentials in the 3-exponential story, though see Robin Hanson's writeup to get a sense for what the series of exponentials view actually ends up looking like---it's still surprised by the great stagnation], and comes much closer to leaving us with our priors + the obvious acceleration over longer periods + the obvious acceleration during the shorter period where we actually have data, which seem to all basically point in the same direction.
  • Even taking the radiocarbon data as given I don't agree with the conclusions you are drawing from that data. It feels like in each case you are saying "a 2-exponential model fits fine" but the 2 exponentials are always different. The actual events (either technological developments or climate change or population dynamics) that are being pointed to as pivotal aren't the same across the different time series and so I think we should just be analyzing these without reference to those events (no suggestive dotted lines :) ). I spent some time doing this kind of curve fitting to various stochastic growth models and this basically looks to me like what individual realizations look like from such models--the extra parameters in "splice together two unrelated curves" let you get fine-looking fits even when we know that the underlying dynamics are continuous+stochastic.
  • I currently don't trust the population data coming from the radiocarbon dating. My current expectation is that after a deep dive I would not end up trusting the radiocarbon dating at all for tracking changes in the rate of population growth when the populations in question are changing how they live and what kinds of artifacts they make (from my perspective, that's what happened with the genetics data, which wasn't caveated so aggressively in the initial draft I reviewed). I'd love to hear from someone who actually knows about these techniques or has done a deep dive on these papers though.
  • I think the only dataset that you should expect to provide evidence on its own is the China population time series. But even there if you just take rolling averages and allow for a reasonable level of noise I think the continuous acceleration story looks fine. E.g. I think if you compare David Roodman's model with the piecewise exponential model (both augmented with measurement noise, and allowing you to choose noisy dynamics however you want for the exponential model), Roodman's model is going to fit the data better despite having fewer free parameters. If that's the case, I don't think this time series can be construed as evidence against that model.
  • I agree with the point that if growth is 0 before the agricultural revolution, rather than "small," then that would undermine the continuous acceleration story. I think prior growth was probably slow but non-zero, and this document didn't really update my view on that question.
Comment by Paul_Christiano on How Much Leverage Should Altruists Use? · 2020-05-16T16:30:18.425Z · EA · GW
This is only 2.4 standard deviations assuming returns follow a normal distribution, which they don't.

No, 2.4 standard deviations is 2.4 standard deviations.

It's possible to have distributions for which what's more or less surprising.

For a normal distribution, this happens about one every 200 periods. I totally agree that this isn't a factor of 200 evidence against your view. So maybe saying "falsifies" was too strong.

But no distribution is 2.35 standard deviations below its mean with probability more than 18%. That's literally impossible. And no distribution is 4 standard deviations below its mean with probability >6%. (I'm just adopting your variance estimates here, so I don't think you can really object.)

This is not directly relevant to the investment strategies I talked about above, but if you use the really simple (and well-supported) expected return model of earnings growth plus dividends plus P/E mean reversion and plug in the current numbers for emerging markets, you get 9-11% real return (Research Affiliates gives 9%, I've seen other sources give 11%). This is not a highly concentrated investment of 50 stocks—it's an entire asset class. So I don't think expecting a 9% return is insane.

Have you looked at backtests of this kind of reasoning for emerging markets? Not of total return, I agree that is super noisy, but just the basic return model? I was briefly very optimistic about EM when I started investing, based on arguments like this one, but then when I looked at the data it just seems like it doesn't work out, and there are tons of ways that emerging market companies could be less appealing for investors that could explain a failure of the model. So I ended up just following the market portfolio, and using much more pessimistic returns estimates.

I didn't look into it super deeply. Here's some even more superficial discussion using numbers I pulled while writing this comment.

Over the decade before this crisis, it seems like EM earnings yields were roughly flat around 8%. Dividend yield was <2%. Real dividends were basically flat. Real price return was slightly negative. And I think on top of all of that the volatility was significantly higher than US markets.

Why expect P/E mean reversion to rescue future returns in this case? It seems like EM companies have lots of on-paper earnings, but they neither distribute those to investors (whether as buybacks or dividends) nor use them to grow future earnings. So their current P/E ratios seem justified, and expecting +5%/year returns from P/E mean reversion seems pretty optimistic.

Like I said, I haven't looked into this deeply, so I'm totally open to someone pointing out that actually the naive return model has worked OK in emerging markets after correcting for some important non-obvious stuff (or even just walking through the above analysis more carefully), and so we should just take the last 10 years of underperformance as evidence that now is a particularly good time to get in. But right now that's not my best guess, much less strongly supported enough that I want to take a big anti-EMH position on it (not to mention that betting against beta is one of the factors that seems most plausible to me and seems best documented, and EM is on the other side of that trade).

which explain why the authors believe their particular implementations of momentum and value have (slightly) better expected return.

I'm willing to believe that, though I'm skeptical that they get enough to pay for their +2% fees.

I don't overly trust backtests, but I trust the process behind VMOT, which is (part of the) reason to believe the cited backtest is reflective of the strategy's long-term performance.[2] VMOT projected returns were based on a 20-year backtest, but you can find similar numbers by looking at much longer data series

The markets today are a lot different from the markets 20 years ago. The problem isn't just that the backtests are typically underpowered, it's that markets become more sophisticated, and everyone gets to see that data. You write:

RAFI believes the value and momentum premia will work as well in the future as they have in the past, and some of the papers I linked above make similar claims. They offer good support for this claim, but in the interest of conservatism, we could justifiably subtract a couple of percentage points from expected return to account for premium degradation.

Having a good argument is one thing---I haven't seen one but also haven't looked that hard, and I'm totally willing to believe that one exists and I think it's reasonable to invest on the basis of such arguments. I also believe that premia won't completely dry up because smart investors won't want the extra volatility if the returns aren't there (and lots of people chasing a premium will add premium-specific volatility).

But without a good argument, subtracting a few percentage points from backtested return isn't conservative. That's probably what you should do with a good argument.

Comment by Paul_Christiano on How Much Leverage Should Altruists Use? · 2020-04-25T03:07:22.176Z · EA · GW

I haven't done a deep dive on this but I think futures are better than this analysis makes them look.

Suppose that I'm in the top bracket and pay 23% taxes on futures, and that my ideal position is 2x SPY.

In a tax-free account I could buy SPY and 1x SPY futures, to get (2x SPY - 1x interest).

In a taxable account I can buy 1x SPY and 1.3x SPY futures. Then my after-tax expected return is again (2x SPY - 1x interest).

The catch is that if I lose money, some of my wealth will take the form of taxable losses that I can use to offset gains in future years. This has a small problem and a bigger problem:

  • Small problem: it may be some years before I can use up those taxable losses. So I'll effectively pay interest on the money over those years. If real rates were 2% and I had to wait 5 years on average to return to my high-water mark, then this would be an effective tax rate of (2% * 5 years) * (23%) ~ 2.3%. I think that's conservative, and this is mostly negligible.
  • Large problem: if the market goes down enough, I could be left totally broke, and my taxable losses won't do me any good. In particular, if the market went down 52%, then my 2x leveraged portfolio should be down to around 23% of my original net worth, but that will entirely be in the form of taxable losses (losing $100 is like getting a $23 grant, to be redeemed only once I've made enough taxable gains).

So I can't just treat my taxable losses as wealth for the purpose of computing leverage. I don't know exactly what the right strategy is, it's probably quite complicated.

The simplest solution is to just ignore them when setting my desired level of leverage. If you do that, and are careful about rebalancing, it seems like you shouldn't lose very much to taxes in log-expectation (e.g. if the market is down 50%, I think you'd end up with about half of your desired leverage, which is similar to a 25% tax rate). But I'd like to work it out, since other than this futures seem appealing.

Comment by Paul_Christiano on How Much Leverage Should Altruists Use? · 2020-04-23T21:23:24.989Z · EA · GW

I'm surprised by (and suspicious of) the claim about so many more international shares being non-tradeable, but it would change my view.

I would guess the savings rate thing is relatively small compared to the fact that a much larger fraction of US GDP is inevestable in the stock market---the US is 20-25% of GDP, but the US is 40% of total stock market capitalization and I think US corporate profits are also ballpark 40% of all publicly traded corporate profits. So if everyone saved the same amount and invested in their home country, US equities would be too cheap.

I agree that under EMH the two bonds A and B are basically the same, so it's neutral. But it's a prima facie reason that A is going to perform worse (not a prima facie reason it will perform better) and it's now pretty murky whether the market is going to err one way or the other.

I'm still pretty skeptical of US equities outperforming, but I'll think about it more.

I haven't thought about the diversification point that much. I don't think that you can just use the empirical daily correlations for the purpose of estimating this, but maybe you can (until you observe them coming apart). It's hard to see how you can be so uncertain about the relative performance of A and B, but still think they are virtually perfectly correlated (but again, that may just be a misleading intuition). I'm going to spend a bit of time with historical data to get a feel for this sometime and will postpone judgment until after doing that.

Comment by Paul_Christiano on How Much Leverage Should Altruists Use? · 2020-04-23T21:12:58.317Z · EA · GW

I also like GMP, and find the paper kind of surprising. I checked the endpoints stuff a bit and it seems like it can explain a small effect but not a huge one. My best guess is that going from equities to GMP is worth like +1-2% risk-free returns.