The Cathedral and the Bizarre a critique of twenty years of open source © Dr Mark Tarver, 2019

This was the bazaar model that Raymond in which believed.  It was to later power the development of cooperative project sharing platforms like Sourceforge and Github.

As opposed to the bazaar was the cathedral method; the traditional in-house method of closed source projects used by corporations under the close direction of project managers supervising small teams of 9-5 workers.  Raymond believed that this model would be out-competed by open source.

Perhaps in the end the open-source culture will triumph not because cooperation is morally right or software ‘‘hoarding’’ is morally wrong (assuming you believe the latter, which neither Linus nor I do), but simply because the closed-source world cannot win an evolutionary arms race with open-source communities that can put orders of magnitude more skilled time into a problem.

The Cathedral and the Bazaar p 54

The bazaar model would issue in a new era of freedom as Bob Young explains in his foreword to Raymond's book.

The success of any industry is almost directly related to the degree of freedom the suppliers and the customers of that industry enjoy......  Open-source software brings to the computer software industry even greater freedom than the hardware manufacturers and consumers have enjoyed.

The Cathedral and the Bazaar p ix

Young explains that this depends crucially on removing the constraints that allow exploitation of code and adopting open source licenses.

Legally restricting access to knowledge of the infrastructure that our society increasingly relies on (via the proprietary binary-only software licenses our industry historically has used) results in less freedom and slower innovation.

The Cathedral and the Bazaar p x

Programmers would be liberated to work in the way they wanted.

Hackers solve problems and build things, and they believe in freedom and voluntary mutual help. Hackers are naturally anti-authoritarian. Anyone who can give you orders can stop you from solving whatever problem you’re being fascinated by—and, given the way authoritarian minds work, will generally find some appallingly stupid reason to do so. So the authoritarian attitude has to be fought wherever you find it, lest it smother you and other hackers.

The Cathedral and the Bazaar p 197

Because open source code would be freely shared over a large community, bugs would be far less common.

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, ‘‘Given enough eyeballs, all bugs are shallow.’’

The Cathedral and the Bazaar p 30

Since code would be shared, the reduplication of code between competitors that came from hoarding would mean that programmers would quickly zero in on optimal solutions.  Darwinian competition would ensure that only one superfit solution to any category of problem would emerge; these programs Raymond referred to as 'category killers'.

Hence The Cathedral and the Bazaar offered an intoxicating revolutionary vision;  freedom for programmers, accelerated innovation, superfit solutions, a challenge to corporate control and more reliable software.   What's not to like?  But twenty years on, where has this revolution taken us in relation to the promises that were made?

Quality and Financial Deficiency Disease (FDD)

Now here is an unpalatable truth, twenty years on: most open source code is poor or unusable. A search through open source repositories like Sourceforge or Github will convince you of that.  If you haven't (as I have done) tried to piece together code from a repository armed only with few pages of code comments and virtually no documentation, you have not lived the Github experience.  In fact an article puts the abandonment rate of open source projects on Github at about 98% - meaning that there is no activity on 98% of projects after a year.  This has coined a phrase - abandonware.

This lesser-known fact is masked by the isolated points fallacy.  The isolated points fallacy consists in taking the high scoring points on a graph and plotting your line on the basis of them. Hence open source champions wheel out the standard examples of success - Open Office, Wordpress and Red Hat Linux (we'll look at why some of these have succeeded later) - ignoring the vast sea of floating half submerged buggy and abandoned projects (over 120,000) that litter Github.  It is the sort of technique Mugabe would have used for TV. If you're accused of starving the country, wheel out a handful of well nourished kids for people to see. 'Look, our country is fine; see how healthy these kids are'. Out in the slums the less fortunate die of cholera. 

But it isn't a physical disease like cholera that a lot of these projects are dying from - it is financial deficiency disease (FDD) due to lack of funding.

Not just the obscure projects buried on Github, but much bigger open source projects such as OpenBSD and OpenSSL have suffered from FDD. It was the HeartBleed bug that exposed the fact that OpenSSL had FDD (see HeartBleed exposes a Problem with Open Source). OpenSSL, an open source encryption programs used by millions of people to protect their credit card details over the Internet had a serious security leak which went undetected for two years.  The truth revealed that OpenSSL was seriously underfinanced with only one full time operative working on a code base of hundreds of thousands of lines of C.

 Marquess, in charge of OpenSSL, spoke about the problems of running an OS project used by millions of people.

"There should be at least a half dozen full time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you’re a corporate or government decision maker in a position to do something about it, give it some thought. Please. I’m getting old and weary and I’d like to retire someday."

After Heartbleed, OpenSSL finally got more of the funding it needed—at least for now. They currently have enough money to pay four full-time employees for three years. But a year and a half into that funding, Marquess isn’t sure what will come next.

The Ford Report on Open Source

Financial deficiency disease is a deficiency disease of open source projects analogous to scurvy, rickets or beri-beri in human beings.  It does not manifest in bleeding gums or curved bones, but in abandoned software, buggy code, poor documentation and missing support.  It is a project killer that is endemic to open source projects.

The austere facts of open source economics poke through like the bones of an undernourished cadaver when you look at some famous open source projects. In January 2014, OpenBSD entered financial crisis when OpenBSD struggled to meet the electricity bill.  After an extraordinary appeal OpenBSD was rescued by a bailout of $100,000. But the total annual revenue of this open source leader in 2015 was actually only that of a single associate professor; not exactly big potatoes. OpenBSD have recently been rescued by Microsoft.

A look at Linux Mint, a well-known Linux distro, shows that the total income from donations around 2015 was in the region of $50,000-60,000. In other words, the income by donation of two top rated open source companies pushing software used by many thousands is only scratching the income of a middle income American.

Charging Support

Raymond did see that if programmers relinquish creative control of their work and gave the sources for free, then they would be challenged by the problem of how to monetise their work. The Cathedral and the Bazaar suggests that open source programmers make money by selling support.  The model is to achieve mind-share (i.e. market dominance and general acceptance) by giving away code as open source and then use that market as a platform to offer services and support. 

Red Hat Linux is offered as a poster child of this approach.    Red Hat, for those who do not know it,  is a company that specialises in selling the operating system Linux, to people running servers that cater for many users (e.g. a web server supporting many sites). Red Hat's market is server administration and its penetration into the desktop market remains very small. It is precisely because Linux is complex, demanding and sometimes quirky that there exists a market for Red Hat to administer it at the server end. Programmers can sell services to businesses if what they produce is sufficiently complex or difficult that people cannot use it easily. But even so, despite being launched on the wave of the dot com book, after its quarter century, the Linux provider Red Hat was still 1/40th of the size of Microsoft and was recently bought out by IBM.

But if the product is highly useful, easy to use, intuitive, reliable and well documented, then giving it away as open source is commercial suicide because there is little or no market value for services: the market value is in the product. And here is the irony, because software that is useful, easy to use, intuitive, reliable and well documented is precisely the paradigm of what software should be.  Good software is properly documented, does not break and does not require hand-holding to use it.

John Gruber made the same point.

Talented programmers who work long full-time hours crafting software need to be paid. That means selling software. Remember the old open source magic formula — that one could make money giving away software by selling “services and support”? That hasn’t happened — in terms of producing well-designed end user software — and it’s no wonder why. In Raymond’s own words, the goal is "software that works so well, and is so discoverable to even novice users, that they don’t have to read documentation or spend time and mental effort to learn about it." [quote from Eric S. Raymond].

It’s pretty hard to sell “services and support” for software that fits that bill. The model that actually works is selling the software itself. This is politically distasteful to open source zealots, but it’s true — and it explains the poor state of usability in open source software.

 Ronco Spray-On Usability

This was again written in 2004.  Here we are in 2015.  

The current business model is recipe for failure.  That's the conclusion of Peter Levine, a partner at Andreessen Horowitz, the Silicon Valley venture capital firm that backed Facebook, Skype, Twitter and Box as startups.....Levine says the conventional open source business model is flawed: Open source companies that charge for maintenance, support, warranties and indemnities for an application or operating system that is available for free simply can't generate enough revenue.

Why the Open Source Business Model is a Failure,

Consequently the standard open source economic model does not favour good, easy-to-use, well-documented software as popularly claimed. What it does favour is technically complex software that needs support or buggy software that gets dropped if the developer loses interest and is not paid.  The fact that an old article from 2004 is still relevant in 2015 and today is an indictment of the inability of the open source movement to progress past Raymond's original idea. 

Forks and Abandonware

Most open source is barely usable and empirical inspection of Github will show that to be true.  Open source users will admit that a lot of open source is buggy abandonware. However they argue that this really does not matter since some small significant fraction is really quite good and that's the stuff we should use. Hence the argument is 'Yes; a lot of open source is awful but that's not important because you don't have to use it.'

However the problem is that the open source user may not stumble on this magical fraction and the invisible iceberg of buggy, ill-conceived open source lies submerged ready to rip out the bottom of your leisure time and send that lazy weekend to the bottom. In fact, open source uses massive amounts of user time trawling through defunct and buggy applications and posting to forums in search of patches and bugfixes. open source zealots tend to be blind to this. They treat the sunk costs of their learning through hard experience as zero, which is wrong. The cost of having to deal with software which should never have been issued is significant - even if you finally junk it. Bad software will injure your leisure time and your pocket. 

But above all this is the sheer waste of human effort in terms of the production of rotting software in repositories.  Github, as anybody who has toured it knows, is a graveyard for software projects, many duplicating the efforts of each other.  Many of these projects died of FDD.  It certainly wasn't supposed to be like that.  Eric .S. Raymond envisaged that open source would liberate programmers from the toil of reproducing each others work because code would be shared.

Imagine no longer having to spend your internal staff’s time and salaries on rewriting, testing, and distributing new binaries for each new kernel as it comes out. You certainly have better things to do with all that skill.

The Cathedral and the Bazaar p 165

But anybody who has kept pace with the history of Linux (and particularly the sad story of their audio systems) knows that this is exactly what goes on in Linux.  Linux is beset by forks and reduplication of effort beyond that which any reputable closed source company would find acceptable.

But let's look at the success stories.

Corporation and Tax Money

Open source weaknesses are hidden behind poster stories like Red Hat.  There are others like OpenOffice, Emacs, Linux, and SBCL. A lot of latent quality comes from these funded ventures.  So let's look at them.

Open Office was derived from Star Office which was the product of StarDivision and Sun Microsystems. It was not put together by a hacker living in his mom’s spare bedroom, but by a team of professionally qualified, highly paid and very able software engineers who were working for a company that made and sold products according to a classic capitalist model. Without the input from that team and the funding from that model there would be no Star Office. Star Office became open source and thus Open Office purely because Sun were willing to support a loss leader in order to acquire part of the Microsoft market. The open source model never supported Star Office and I doubt that it would have ever done so. 

Emacs was supported financially by people working at the MIT AI Lab, which means that it was funded by Uncle Sam. It was not invented by Richard Stallman contrary to popular myth, although he did grab the sources and improved them and tried successfully to claim as much credit as he could. It’s real cost in market terms was effectively many thousands of tax dollars.

SBCL is at the moment the leading open source Common Lisp platform. But it was, and is, deeply indebted to CMU CL from which it was a fork. CMU CL was again an Uncle Sam project being funded originally by DARPA and the guys who developed it were top class professionals who were paid a lot of money to do it. Sans CMU CL, SBCL would not have got off the ground.

Linux is of course, mostly a copy of Unix, it is deeply unoriginal, being based on ideas going back to the time of the Vietnam War. These ideas were in turn evolved within Bell Labs by its creators who were also well-paid professionals. Linus Torvalds copied an idea whose basis had been funded by university and corporation money and without that basis there would have been no Linux. Early Linuxes were dreadful.  My Ubuntu version of 2005 was an absolute crock that wasted the plastic on which it was distributed. Ubuntu was itself a loss-making personal hobby of a entrepreneur who had so many millions that he could afford to run the parent company, Canonical, at a loss for years.  The situation in 2019 is better than 2005, but the Linux desktop still lags behind Windows and the interface looks stuck in the 90s. 

These implementations owe their existence to models of funding and development to which the open source community are either allergic or indifferent.

Crowd Funding

Crowd funding has recently emerged as a potential solution to funding projects.  Does crowd funding solve the economic problems of open source? Not really; certainly not for most projects.

Most funded projects on Kickstarter fall into the category of gadgets and games. There are a few software projects that have attracted significant funding (like Light Table) but not that many. The average successful Kickstarter funding of about $5,000 will not carry any business much beyond the first quarter of its first year.  The clue is the title - a kickstart is there to get a project started, not to sustain it.  Crowd funding is not an adequate long-term income model.

Free Open Source is often Conformist

Though a lot of fuss is made about how open source is innovative, mostly it isn't.  An awful lot of popular open source is inferior reverse-engineered copies of existing commercial software (Gimp, OpenOffice etc). That's not an accident.  The quickest way to achieve popularity in open source is to copy some successful closed-source application.  Innovation is hard; it requires time and brains.  Stanislav notes that open source has this narrowing effect of reproducing accepted ideas.

I predict that no tool of any kind which too greatly amplifies the productivity of an individual will ever be permitted to most developers.  In this they shall follow in the maximally deskilled assembly-line footsteps of their grandparents.  “They’ll time your every breath.” As for the “free software” world, it eagerly opposes industrial dogmas in rhetoric but not at all in practice.  No concept shunned by cube farm hells has ever gained real traction among the amateur masses.

Consider Linux: the poster child of successful free software.  It is a knockoff of a 1970s operating system well past its sell-by date. This is because a herd simply cannot innovate, whether for fun or for profit. Every innovative work of mankind has been the product of one – sometimes two, rarely three – minds.  And never the work of a herd.  No mathematical theorem, no enjoyable novel, no work of art of any importance, have ever been produced by a herd.   I fail to see why innovative software ought to play by a different set of rules.

Where Lisp Fails: at Turning People into Fungible Cogs

If open source programmers do innovate and their innovation is good then its just as likely to be swept away from them by the corporations who have the capital to exploit it.

Open and Weak vs Closed and Strong

This was first observed as long ago as 2004 by Matthew Thomas.

Proprietary software vendors typically make money by producing software that people want to use. This is a strong incentive to make it more usable. (It doesn’t always work: for example, Microsoft, Apple, and Adobe software sometimes becomes worse but remains dominant through network effects. But it works most of the time.)

With volunteer projects, though, any incentive is much weaker. The number of users rarely makes any financial difference to developers, and with freely redistributable software, it’s near-impossible to count users anyway. There are other incentives — impressing future employers, or getting your software included in a popular OS — but they’re rather oblique.

Why Free Software has Poor Usability, and How to Improve It

Here FDD creeps in.  Somebody makes a free program that because it is free, displaces it closed source competitors which may be superior.  It survives because it is just good enough to be usable.  A free bad program can be just good enough for people to want to use it in preference to a costed close source solution that provides financial incentives to maintain.  Eric S. Raymond praises these open source works as 'category killers'.

Some very successful projects become category killers; nobody wants to homestead anywhere near them because competing against the established base for the attention of hackers would be too hard. People who might otherwise found their own distinct efforts end up, instead, adding extensions for these big, successful projects. The classic category killer example is GNU Emacs; its variants fill the ecological niche for a fully-programmable editor so completely that no competitor has gotten much beyond the one-man project stage since the early 1980s. Instead, people write Emacs modes.

That's ironic, because a lot of people think that Emacs is outdated, but like Linux because it is open source and widely used it is likely to survive.  Free, widely known, derivative and mediocre can displace sophisticated, innovative, costly and good.

Why Do Corporations Support Open Source?

Corporations like Microsoft were initially afraid of open source as a stealer for their products.  Microsoft were very concerned about Linux as a competitor to Windows.  Thus in 2002 Tech Report wrote.

Behind the war of words, analysts say, is evidence that Microsoft is increasingly concerned about Linux and its growing popularity. The Unix-like operating system "has clearly emerged as the spoiler that will prevent Microsoft from achieving a dominant position" in the worldwide server operating-system market, IDC analyst Al Gillen concludes in a forthcoming report.

While Microsoft's overall operating-system market leadership is by no means in jeopardy, Linux's continued gains make it harder for Microsoft to further its core plan for the future, Microsoft.Net.

Why Microsoft is Wary of Open Source

As desktop Linux disintegrated into a welter of forks and abandonware, Microsoft relaxed; it was safe.  But now Microsoft and other corporations like Google positively embrace open source.  Why?

They embrace it because open source allows them to monetise work without paying for it.  So corporations use open source and discard it when it does not serve their purpose.  Chris Hoffman observes.

Google doesn’t really care about Android as a full open-source project, either, which is why more and more parts of the “Android Open Source Project” (or “AOSP”) are being left behind. Google wants to keep Android open so it’s easy for manufacturers to customize, but open source applications like the keyboard and dialler are becoming more and more outdated. On a consumer Android device, Google just bundles its own closed source keyboard, dialler, and other apps. Google seems committed to an Android open-source core, but not an entire open-source operating system people can use without Google’s software and services. After all, improving the Android Open Source Project just helps Amazon’s Fire OS, a competitor to Google’s Android devices. What’s the point of that?

The Downsides of Open Source

John Mark indicts open source as a vehicle for positive social change.

When we were but wee lads and lasses on the forefront of this thing we called free software and eventually open source, we knew that this was dangerous stuff. It was destined to set fire to an entire industry, undermining entrenched monopoly powers and establishing a more equitable approach to building wealth around the tools that would power humanity in the 21st century. It was about the democratization of software and would smash what we then called the “digital divide”. That premise was entirely false. The crux of this essay is thus: not only did open source not stem or stall the redistribution of wealth and power upwards, but rather it aided and abetted the redistribution of wealth and power upwards. To be an open source proponent at this time without acknowledging this very real and most unfortunate consequence is to be a cog in a much larger machine; a stooge; a very useful idiot.

Why Open Source Failed.

Which brings us to one of the offshoots of the open source movement - the dislike of effective copyright and ownership. This was integral to the open source model that Raymond projected. But copyright law exists to protect innovators from companies like Microsoft who would otherwise exploit their work and give nothing to the innovator. It is the function of law, properly conceived, to act as the great leveller, allowing the weak to stand next the strong under the protection of law. Remove the protection of law and the Golden Rule applies; he who has the most gold makes the rules.

Prior to the open source movement, companies like Microsoft and Google had to spend millions of dollars on R&D to keep up. To have a market model where innovators who cannot capitalise their ideas, freely share their best ideas and code with the corporations who can take advantage of them is great for corporations. Open source programmers became self-basting turkeys for the corporate ovens.

Social Groups in Open Source

A small group of genuinely idealistic and creative people, often young, fall for the claims about freedom and the battle against corporate control made by open source advocates.  We can call them the givers.  These young people are mainly ignorant of the failures of the movement and are generally exploited until they burn out.   They often power significant projects.  When they burn out, they are replaced or the project dies.  This model of using people was acknowledged right from the beginning with a sly wink to Linus Torvalds from Eric S. Raymond in The Cathedral and the Bazaar.

In fact, I think Linus’s cleverest and most consequential hack was not the construction of the Linux kernel itself, but rather his invention of the Linux development model. When I expressed this opinion in his presence once, he smiled and quietly repeated something he has often said: “I’m basically a very lazy person who likes to get credit for things other people actually do.