How impactful is free and open source software development?post by Erich_Grunewald · 2021-10-09T10:15:44.105Z · EA · GW · 7 comments
This is a question post.
I originally posted this on my blog, but envisioned it as a question for all of you here.
It's fairly clear that one cannot code from the ground up in bazaar style. One can test, debug and improve in bazaar style, but it would be very hard to originate a project in bazaar mode. Linus didn't try it. I didn't either.
– Eric S. Raymond
It seems likely to me that much of our future prosperity (or much of our future, at any rate) will be built on software, including free and open source (FOSS) software. Software already pervades all the other major branches of technology – it is used in agriculture, medicine, manufacturing, construction, energy, transportation, accounting, electronics and more. It constitutes the most common means of collecting, storing, retrieving and processing information at scale. It has enabled the Global Positioning System, commercial mobile phones, near-instantaneous cash transfers, cryptocurrencies, spreadsheets, 3D printing, remote work, advanced statistical modelling, forecasting platforms and the internet among much else. Few of today's great scientific discoveries could have been made without it.
There are inventions that result in new devices or processes and no more. Then there are those inventions that result in new devices or processes that are useful in coming up with new inventions. In this second category we can include human languages and writing, the wheel, the microscope, the steam engine, electricity and the ball bearing. Often these inventions are instances of what economists call general-purpose technology, that is technology that leads to productivity increases across all sectors of a region's economy.
Software is a strong candidate for a general-purpose technology. And as software is a general-purpose technology, so are there general-purpose software technologies. I mean things like programming languages, database systems, Reactive Extensions, data structure and processing libraries like NumPy and Pandas, graphics frameworks like OpenGL and distributed version control systems like git and Mercurial – in other words, the substructure and the infrastructure of software. These are very often released under FOSS licenses, for everyone to use, inspect and modify as they wish, and are consequently often used by vast numbers of people.
Austin Vernon is more bearish on productivity gains from software:
The current status quo means we don't get productivity growth until these software-driven companies become behemoths. Amazon was founded in 1994, almost thirty years ago. In 2020, it was still less than 10% of total retail sales. Is it any wonder that we haven't seen robust productivity gains? [...] In many industries, assembly lines builders are just getting started. If these companies maintain rapid growth as their market share increases, [total-factor productivity] should finally show some life. [...] Since software is a multiplier, any increase in input quality or quantity is even more beneficial. Software is more impactful in a universe with 1 trillion humans than one with 10 billion humans.
In Evolution of Programming Language Traits, I wrote about research showing that innovation rates are largely driven by a group of people's interconnectedness. I wrote:
[W]hat greatly affects a population's rate of innovation is its interconnectedness, in other words how widely ideas, information and tools are shared. In a culture that is deeply interconnected, where information is widely shared, innovations are observable and shared tools and standards mean that innovations are also more likely to be compatible. Most importantly, interconnectedness provides each individual with a large pool of ideas from which they can select the most attractive to modify, recombine, extend and spread in turn.
I then went on to quote Richard Stallman:
Software development used to be an evolutionary process, where a person would take an existing program and rewrite parts of it for one new feature, and then another person would rewrite parts to add another feature; in some cases, this continued over a period of twenty years. Meanwhile, parts of the program would be "cannibalized" to form the beginnings of other programs.
The existence of owners prevents this kind of evolution, making it necessary to start from scratch when developing a program. It also prevents new practitioners from studying existing programs to learn useful techniques or even how large programs can be structured.
In other words, the free and open source philosophies really allow us to share tools and information in a way that wasn't possible before. So supposing, as I do, that technology is on the whole good for the world, creating free and open source software, especially general-purpose software, seems good for the world.
Okay, but how much marginal impact can a single programmer expect to have? I think that is hard to estimate, but I will note, as the epigraph suggests, that many important FOSS projects have been created basically by a single person, alone. Linus Torvalds created the Linux kernel, various distributions of which has the majority of the marketshare in smartphones, supercomputers, web servers and embedded systems. Then, for good measure, he wrote git which, though it was not exactly the first of its kind, has fairly revolutionised software development and code sharing. Richard Stallman wrote the text-editor Emacs, which is being used today, nearly 40 years after it was created, and he also laid the foundations for the GNU Compiler Collection. Carsten Dominik created the widely used and hugely influential information organising tool org-mode. Satoshi Nakamoto, building on previous work by Wei Dai and Adam Back, conceptualised and implemented Bitcoin and the blockchain. Ryan Dahl created Node.js; David Heinemeier Hansson created Ruby on Rails; Salvatore Sanfilippo created Redis; Igor Sysoev created nginx. And so on.
Of course, I would say all this. As a programmer, a part of me wants FOSS to be important and impactful, and so I don't entirely trust myself when I say that it seems pretty important and impactful to me. I, too, would be sceptic if I read an accountant explaining how good pro-bono accounting is for the world. Maybe software is not as good for humanity as I think. Maybe the free and open source philosophies are not as good for software as I think. Maybe a single programmer, talented and dedicated though they may be, is just extremely unlikely to ever write a software program that is a significant improvement for the world on the margin. Software does pervade all the other major branches of technology, but so does pretty much every other major branch of technology!
How impactful is FOSS programming? 80,000 Hours, when writing about software engineering, don't seem to consider FOSS at all:
Overall, [software engineering] is good for earning to give. Software engineers can also make some direct contributions to innovation and economic productivity, especially if working for a good company. Advocacy potential is moderately good because you can be involved in the influential and innovative tech community.
I reckon that FOSS development is good for the world, likely more impactful than whatever direct proprietary contributions one can make working for a company, but I think this is really hard to quantify – certainly much harder than quantifying charitable donations – and reckon further that this is one reason why it is talked about less than those other things. But I would like to hear other opinions.
Raymond, E. (1999). The cathedral and the bazaar. Knowledge, Technology & Policy, 12(3), 23-49. ↩︎
Basu, S., & Fernald, J. (2007). Information and communications technology as a general-purpose technology: Evidence from US industry data. German Economic Review, 8(2), 146-173. ↩︎
Naughton, J. (2016). The evolution of the Internet: from military experiment to General Purpose Technology. Journal of Cyber Policy, 1(1), 5-28. ↩︎
By general purpose, I mean only that they can be used in a wide variety of different applications. ↩︎
Brien, M. & Shennan, S. (2010). Innovation in cultural systems : contributions from evolutionary anthropology. Cambridge, Mass: MIT Press. ↩︎
Of course many other people have worked to extend, maintain and test these creations since their conceptions. I do not mean to diminish that work, which is also important. ↩︎
It is not a proven fact that he is a single person, but it seems likely to me. ↩︎
Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. Decentralized Business Review, 21260. ↩︎
It seems doubtful to me that FOSS development in general would be an amazingly impactful way to help the world. But maybe there are ways we can identify specific development projects that are much more impactful than others. A couple of rambling thoughts below. (Disclaimer! although I do write lots of C++ code for my aerospace engineering job, I am basically a rando and I don't know that much about the FOSS movement. Take everything I say with a big grain of salt.)
For every example of someone making a great outsized contribution (Bitcoin, Linux), aren't there thousands of examples of people starting small projects that never get finished or attain wide adoption? Also, aren't there many examples of private software projects becoming equally large and important (Microsoft, Google, etc)?
For an interesting perspective that is FOSS-adjacent but distinct in some major ways, you might be interested in the ideas of Jonathan Blow, a game developer who is currently developing a new programming language called Jai, in order to... do something like create a tool to help preserve the level of focus and high-level programming skill that he thinks is important for the long-term future of tech development as a field.
It does seem likely that contributing to a well-chosen software project is probably a great way to advance economic growth. But there is so much potential software out there -- FOSS vs not-FOSS is only one choice, and I would want to try and choose wisely among many dimensions. What kind of software project could be the most useful to the most people (and thus unlock the highest amount of economic growth?), but hasn't already been created (perhaps because there is no private/financial benefit to the developer, thus no incentive to create it)? This kind of question was probably on the minds of many of the most influential FOSS developers.
FYI I don't think that Austin Vernon's perspective is relevant for this choice. Yes, I agree that our civilization is probably bottlenecked more by lack of good social institutions by technology, and in the world of technology we are bottlenecked more by lack of innovation in the world of "atoms" than in the world of "bits". Nevertheless (indeed because of those factors), software is a high-productivity high-growth part of our civilization, so doing object-level work in software is probably more effective for the average worker vs bashing one's head against obstacles trying to do object-level work in the overregulated world of "atoms"? Especially if you are already specialized in software development, I think there are plenty of good opportunities in the world of "bits" -- the trick is just to choose the right project!
Overall economic growth is important (see the 80,000 Hours episode with Tyler Cowen for a comparison/debate of economic growth vs other EA causes). But other EA causes also need software! Most notably, of course, AI safety projects are extremely software-development-intensive. There are also software projects in other areas, like the startup Wave which makes it easier to send remittances abroad, or projects to imporve the cybersecurity of Biodefense and AI research efforts (whose sensitive and dangerous secrets might otherwise get stolen by malicious actors). I don't have a great overview of this space, but there are a lot of programmers in EA, so I'm sure somebody around here could provide a really great summary of some of the most crucial projects being worked on right now, some of which are probably FOSS and others which might not be.
Other than software development in support of specific EA causes, I maybe have another idea for where to look for especially impactful projects. Why does the FOSS movement arise in the first place? (Disclaimer: I am completely spitballing here and have no knowledge of the movement's history, but nevertheless...) Modular software can build upon other pieces of modular software, like knowledge, so it would be socially beneficial if almost all software was FOSS. But there are usually few private incentives for software to be FOSS, since you could always try to make a buck by having the software be proprietary and charging for it. The FOSS movement seems to have evolved as a way to partially correct for that imbalance -- to provide social incentives that make up for the lack of financial compensation intrinsic to FOSS development. In this sense, the FOSS movement is like EA -- both are social movements that encourage the creation of public goods that would otherwise go tragically un-incentivized. At the same time, the FOSS movement has been around for a while, and it has not taken over the world (I suspect most software is still developed privately), so it must be naturally limited in some ways. (Cross our fingers that the same does not happen to EA as well!! Unfortunately it probably will.)
But with software moreso than with EA, I am optimistic that, in addition to contributing to the existing pro-public-goods social movement, we can also find other ways to encourage and incentivize more public goods production. To some extent, crypto represents a big advance here -- crypto protocols like Bitcoin are technically FOSS, but unlike most FOSS projects, you can get really rich if you create a useful crypto project! Thus, projects in the crypto space are properly incentivized, fueling a boom in development activity much bigger than the ideologically and socially driven "cypherpunks" movement was able to manage earlier. Perhaps some extension of crypto ideas could help incentivize other, non-crypto software projects? (Maybe using some crypto tool, each new FOSS project could easily allow users of the software to pay to establish bug-bounties or to request new features?) I am particularly interested in incentive design systems like Gitcoin, which seek to systematically fund the socially-optimal level of public goods provision. Gitcoin's quadratic funding mechanism is interesting because it could potentially scale up far beyond software development, to become a general governance technique for promoting public goods in many areas of society.
Similarly, it's also conceivable that certain software projects could get leverage by making it easier for other developers to collaborate or work more efficiently. Git is probably the best example here, but this could also include stuff that (for example) makes it easier for people in India or Latin America to work remotely and get paid western wages for working on software projects, as a kind of virtual immigration.
...Alright, hopefully something in this incoherent brain-dump was helpful! I guess my overall theme here is that FOSS vs non-FOSS is just one of many considerations I'd want to think hard about when trying to guess at what software projects have the best long-term impact on the world. Working on EA-specific projects is obviously promising, as perhaps is Gitcoin-y incentive-design stuff that could solve the whole problem of needing a big social movement to encourage public goods production. And I'm sure there's a lot of other promising leads that I know nothing about! Best of luck to you in that difficult prioritization & forecasting endeavor, which is so core to the project of EA!
↑ comment by Erich_Grunewald · 2021-10-09T21:35:53.399Z · EA(p) · GW(p)
thanks, these are some really interesting thoughts. some comments:
For every example of someone making a great outsized contribution (Bitcoin, Linux), aren't there thousands of examples of people starting small projects that never get finished or attain wide adoption?
i think this is true. but i think it's true for most career choices / endeavours that they only make sense for people who have the right skill set, drive, personality and so on.
i found some estimates for the eu (pdf) -- around 8% of around 3.1 million programmers are involved in open source, however spending on average less than 10% of their time on open source. i'm guessing this more or less follows something like the pareto principle, where only a fairly small number of these spend significant time and effort on open source (the sort of time and effort that's needed to create a project of significant ambition from scratch).
You seem to be saying that FOSS software development is helpful because it will advance overall economic growth (moreso than software development at a private corporation).
i think i think it's good more because it enables technological development and innovation than because it aids economic growth.
that same eu report seems to think it can be both:
Starting at the generic level of the impact on GDP in the EU including the UK, a significant and large impact of up to €100 billion per year by OSS contributions has been calculated [...] OSS can be seen [...] as a public infrastructure generating massive positive network externalities supported by the high rating by the stakeholders of the benefits of open standards and interoperability provided by the use of and the contributions to OSS. Therefore, OSS has also a further growth enhancing impact in the dimension of the previously quantified impact of the stock of technical standards. Furthermore, OSS contributions are increasing the labour productivity in the EU, which can be explained by their labour cost saving effect reported as major benefits in different surveys among companies, but also being rated as very relevant in the stakeholder survey.
next, you write:
Also, aren't there many examples of private software projects becoming equally large and important (Microsoft, Google, etc)? [...] You seem to be saying that FOSS software development is helpful because it will advance overall economic growth (more so than software development at a private corporation). This is certainly plausible, although I'm not sure -- I feel like the most of the software that I use was created by private companies rather than FOSS efforts.
this is really surprising to me -- in all the projects i've worked, i would say the majority of the infrastructure -- everything from programming languages, frameworks, libraries and so on -- that we've used are open source. (i think platforms and tools are somewhat more likely to be proprietary, but even those are often open source -- think git, unix shell programs, editors, etc.) it's likely that aerospace engineering is different (more conservative?) from web/mobile/crypto which is where i've worked.
(i am talking about the non-surface layers of the tech stack of course -- the stuff that's built on top is obviously proprietary.)
At the same time, the FOSS movement has been around for a while, and it has not taken over the world (I suspect most software is still developed privately), so it must be naturally limited in some ways.
here is the introduction of an ieee software issue that literally opens with the sentence "Open source software has conquered the software world.":
You can see it nearly everywhere, from Internet infrastructure to mobile phones to the desktop. In addition to that, although many OSS practices were viewed with skepticism 20 years ago, several have become mainstream in software engineering today: from development tools such as Git to practices such as modern code reviews. In the programmer community, OSS has become so prevalent that some companies now expect potential employees to have an active GitHub profile that showcases their OSS contributions. [...] OSS has spread its philosophy well beyond software engineering and inspired many other movements and initiatives, such as open innovation, open hardware, open government, open content (e.g., Wikipedia and OpenStreetMap), and open educational resources. Even the way researchers publish their research has changed, with many attempting to have their publications available under open access and following open science principles.
though to be clear, i agree that foss is limited -- there is certainly space for proprietary software, mainly in creating (non-programmer-)user-facing applications, where for-profit firms are better at identifying and fulfilling customer needs.
Why does the FOSS movement arise in the first place? [...] Modular software can build upon other pieces of modular software, like knowledge, so it would be socially beneficial if almost all software was FOSS. But there are usually few private incentives for software to be FOSS, since you could always try to make a buck by having the software be proprietary and charging for it. The FOSS movement seems to have evolved as a way to partially correct for that imbalance -- to provide social incentives that make up for the lack of financial compensation intrinsic to FOSS development.
that is interesting -- i think there's probably a competitive advantage to foss, where people are much more likely to use it all other things equal, it being free, introspectable, extensible, etc.
↑ comment by Aaron Gertler (aarongertler) · 2021-10-13T09:14:08.700Z · EA(p) · GW(p)
This should probably be a post! I'd love to share it in the Forum Digest and elsewhere — I can technically do this even when it's a comment, but I'd love to see it get a title, some tags, its own full comment section, etc.Replies from: Jackson Wagner
↑ comment by Jackson Wagner · 2021-10-13T09:25:53.682Z · EA(p) · GW(p)
Maybe it should be an adversarial collaboration between me and somebody like Erich who knows a little more about programming! But sure; I will try to get a draft together and post it soon.Replies from: Erich_Grunewald, Oliver Sourbut
↑ comment by Erich_Grunewald · 2021-10-13T10:56:09.734Z · EA(p) · GW(p)
i would be happy to review / comment on a draft or something like that!
↑ comment by Oliver Sourbut · 2021-10-14T09:32:04.697Z · EA(p) · GW(p)
Hey, let me know if you'd like another reviewer. I'm a medium-experienced senior software engineer whose professional work and side-projects use various proportions of open-source and proprietary software. And I enjoy reviewing/proof-reading :)
Comments sorted by top scores.
comment by Chris Kerr (frenbaim) · 2022-02-16T19:41:17.658Z · EA(p) · GW(p)
Thanks for writing this! Writing a post on FOSS as an EA intervention has been on my to-do list ever since I noticed about six months ago that nobody had written one yet.
One additional benefit of FOSS that you appear to have missed is that, by offering a competitive alternative to proprietary software (and importantly, an alternative which can never be bought out in a corporate takeover), FOSS helps prevent the vendors selling proprietary software from exploiting their users quite as much as they otherwise would.
I definitely started using FOSS to save money while I was still at school and university, but now I would pay thousands to avoid the pervasive anti-features in all the (now often free-as-in-beer) proprietary ecosystems.