Open Source is Ceasing to be Cool

Dr Mark Tarver, 2016

This essay is a lightly edited transcript of a series of posts I made to the Shen news group in the middle of 2015.  They highlighted the failure of the open source initiative in the project, analysed the reasons and suggested a successful formula for moving the project forward.  The posts led to the formation of the much more dynamic closed source Shen Professional.  The lessons here are widely applicable to many open source projects and developers may find the material interesting.

Evaluating the Open Source Initiative

I said that I would give the open source initiative 6 months from BSD day (February 2015)before passing judgment; but the 6 months is now well up. So it's time to look at what has been achieved and what lessons are to be learned.  Stay focused; what follows is important.

Six months have passed since BSD day and it is time to take stock of achievements. In that time the OS initiative has accomplished bugger all. Am I surprised? No. But even my admittedly low expectations were surpassed by the non-performance of this approach. I hope that we're not going to follow the Maoists of the period of the Great Leap Forward - who could not accept that their unworkable ideas did not work and resorted to show trials of people to explain their failure.

Also I hope that people are not going to try to save this discredited ideology from being led to the scaffold. Take heart, I am in fact only executing a corpse in what I am going to say in the next messages.

If there is a value in this period, and it can be a very substantial one, it is seeing why ideas fail, which can point to the right direction. We are going to take the red pill. But before moving forward, the whole open source strategy must be deconstructed.

Now I will warn you that in the next paragraphs open source is going to be shredded. Why am I doing this? Just to exercise an itch? No.  In Taoism when you embark on energy cultivation (qigong) you don't just put the new energy on top of what is already there; the old stale, bad qi needs to be cleaned out. Think of having to paint over a surface with lumpy old paint; you sand down the surface first and then apply the new paint otherwise your new paint job is screwed by the old one. So systematically I'm going to remove some myths and clean out the bad stuff.

Myth #1 Mindshare = Success

Defining mindshare as popular support on chat sites and having lots of users is an OS idea of success. If you define 'success' as such and that is your goal, then 'mindshare is success' is a tautology. But if you've graduated beyond this desire and you have other ideas of success - for example livelihood - then this is one myth that needs to be busted.

We know now that though giving people stuff for free and achieving traction is not at all synonymous with being economically successful. We know this because we now have examples of projects that got lots of freeloaders and still struggle for money - most notoriously OpenSSL, but also OpenBSD and many others less well known. We also know that in any project of significant size and complexity, money is needed on a regular basis to pay for the kind of professional development and support needed to deliver a really good product. This is the case with Shen.

The other problem in defining success as mindshare is that this creates an almost futureproof model for failure. I can think of nothing more likely to fuck your mind than going for mindshare. Why? Because simply the world is a very big place full of people with very small brains who can barely hold one idea at a time. The nature of things is that the chances of your idea conquering the world is slight. So if you're building mindshare into your success model you're building disappointment, heartbreak and abandonment into your success strategy.

There is a certain character you see in the West; the sort of girl who paints her nails, works at KFC and dreams of being selected for X-Factor because she sang karaoke at her friend's hen night. We laugh at this because we know that for her the chances of getting mindshare and being the next Beyonce are slight. Everyone can dream; but when the girl goes on to neglect her education and development because she's stuck on this dream, then something has gone very wrong in her head. The aspirations of the OS fanatic hooked on mindshare are often no more realistic than the neurotic checkout girl at KFC.

Out of all the dubious achievements of open source none has been so damaging as the general promotion of this neurotic definition of success.

Myth #2: Open Source is Creative

This is an odd one. Saying OS is creative is what Gilbert Ryle called a category mistake and in programming we call a type error; open source is no more creative than people are divisible by 2 or knowledge wants to be free. Creativity is really a feature of intelligent life of which humans are supposed to be exemplars. We can only talk about institutions being creative to the extent that they foster and encourage creativity. Universities to an extent, used to do that, but they are corporate machines today; they are not creative places to work.

OS has never really been creative in this sense for the simple reason that mindshare philosophy dominates and the quickest way to achieve mindshare is to copy some closed source work that has already achieved it. Hence Linux, Gimp, Open Office etc. This is not to say that nothing under an OS license has ever been creative; it just means that the zeitgeist of the movement does not especially reward originality.

I predict that no tool of any kind which too greatly amplifies the productivity of an individual will ever be permitted to most developers.  In this they shall follow in the maximally deskilled assembly-line footsteps of their grandparents.  “They'll time your every breath.” As for the “free software” world, it eagerly opposes industrial dogmas in rhetoric but not at all in practice. No concept shunned by cube farm hells has ever gained real traction among the amateur masses.

Where Lisp Fails: at Turning People into Fungible Cogs by Stanislav

Shen simply sticks out as contrary to that trend - it is original and hard to understand. So immediately we should at least begin to question the goals of mindshare in connection with Shen. In fact the coupling of Shen with mindshare thinking has been nothing short of tragic in its consequences for this project. I want to get stuck into specifics next.

The Consequences of Adopting Mindshare on the Development of Shen

I think a lot of people in the Shen group bought into mindshare and it informed a lot of their work. Greg wrote to me and said that there was no future unless Shen was more widely adopted. He then nobly set himself to write an open source dissertation on Shen-YACC. Bruno created a wiki which I attached to the Shen front page to allow people to create their own documentation. People created the Shen reddit - a very mixed blessing in some ways - and Twitter was used. All these are products of a generation for whom social networking via the web and the philosophy of mindshare are very much intertwined. Shen needed mindshare to live and thrive.

But actually if you look at the most productive phase from 2011-2012, the 'mindshare' consisted of a handful of people. You could have seated us all round a table in Starbucks. Perhaps one day we need to do that.

The philosophy of mindshare is one of heartbreak because significant mindshare is hard to achieve. Hence the people here were laying up trouble and disappointment for themselves. But the task was made even more monumental by very strengths of Shen; the advanced type system, sequent calculus and the incredible multilingual potential which collectively the software community did not and still do not understand. So in effect a very small group of idealistic people with hardly any resources undertook to re-educate the planet.

Giving Away Your Power

When this decision to educate the planet about Shen was taken, and I don't say it was a conscious decision, the centre of gravity of the project moved outwards and the energy was dissipated on a fruitless goal. Effectively the philosophy of mindshare had redefined success. Shen was in fact an outstanding success, almost a miracle; with the minimum of funding, under harsh conditions, it had achieved exactly what it set out to do and a small group of people had made it run under nearly every major platform. But now success was redefined as mindshare.

This is the time when reddit made an appearance. I blessed the Shen subreddit because I thought that reddit was some polite ladies bookclub - obviously showing my ignorance of it. It is nothing of the sort of course and some of its most vicious sites like Coontown - the reddit for racists only recently closed - shows what is so often like. Reddit is a shit hole; and to a lesser degree so are SlashDot, and Hacker News.

Unfortunately some members of the Shen group, imbibing the mindshare poison, gave much power away to the opinions on reddit. The OS fanatics relished their power to bully and shame and people used to mail me and say 'You need to say something because XYZ has said such and such on reddit'. I did on occasion call such people 'complete prats', but really this was whack-a-mole because the supply of arseholes was unending.

Effectively the mindshare philosophy delivered the energy and soul of the project to those who had reasons to dislike it. These people were stupid and could have been dismissed as such but the mindshare poison had so undermined resistance that people here could not fight them. These online voices represented or claimed to represent the community - they were the mindshare.

Having bowed to open source as they demanded, the group discovered that these people have melted away and are not to be found. There is no open source army on the march that Tycho hoped for. There never was. What you had was a bunch of useless people getting their kicks out of manipulating a small group of much cleverer people. It was simply a power trip. The nadir was reached when two members of the open source community crawled out from under their stone to conspire to expel Shen from Wikipedia by announcing Shen was dead.

Taking Back the Power

When you've given away your power you generally end up energetically or literally 'broke' and in a dead-end. The way this project felt mid-2015. You look around and ask "How did I get here?" and as Willi did "What now?". I've answered the first question and now I move to the second. The short answer is:

you take back the power

This is the first step of getting things going. You let go of the strategies and mind-stuff that led you to the position of helplessness and being stuck. In this case we let go of the philosophy of mindshare. Success for Shen is not mindshare.

When we do that immediately the power returns to the centre where it belongs; inside the project and to the people who made it happen. Since mindshare is no longer operating, the opinions of the open source community are no longer important. You don't have to give a shit about them unless they have something significant to say - which, reading reddit, is not often.

Doing this puts them in the proper perspective; rather like when you were a kid and there was a huge creepy figure in your bedroom. You turn on the light and the huge creepy figure is your dressing gown hanging up. Your mind gave it power and made it into something huge. The freetards want you to believe that they are huge, powerful and you cannot succeed without their approval, but actually they have no power unless we developers choose to give it to them. When you realise it is a mind-trick induced by the mindshare pill they dwindle to their true size.

Shen is a Powertool not Designed for the Average Programmer

As long as people were hooked on the blue pill there was very little chance of anything useful happening. The OS fanatics on Wikipedia talked about Shen being dead. The smell of death around the Shen project is not coming from Shen, it is coming from the decomposing body of the OS philosophy based on mindshare and the faltering economics used to power the project.

Underneath Shen is amazingly dynamic, untouched and better documented than in TBoS 3rd edition than it has ever been. And the OS 6 month period of total failure has had its purpose; at least people are waking up to something that does not work and asking 'What can we do?'. People are actually questioning the central assumptions of the OS initiative. The failure of the OS initiative is a time of huge opportunity for Shen.

Open source philosophy is dead, last century and discredited and we developers need to take back our power from it. But where do we go and what do we do with that power? Well, first we must acknowledge that Shen is itself hugely powerful, that potentially as a productivity tool it can reap enormous benefits if rightly handled and developed, that it is in many ways unique, but that being able to extract and lever that power requires programming skills that most programmers do not have and never will. Think of the skills needed to dogfight and pilot the F-16 to the edge of the atmosphere and the skills needed to fly a family twin seater under autopilot. That's the difference we're looking at between a Shen programmer and the stock programmer assembling glue code and cranking code generated from packages.

Using Shen Effectively in a Commercial Setting

If we are looking at the commercial development of this language beyond a hobbyist implementation then we are looking at an elite group.

I quote Paul Graham's famous essay, Beating the Averages, by Paul Graham. This is a must-read if you haven't read it and it is a must-must read for Shenturians. Several of the passages apply directly to Shen. Read it because I'm only excerpting with comments here. All I've done is substitute 'Shen' for 'Lisp'.

The purpose of this article is not to change anyone's mind, but to reassure people already interested in using Shen -- people who know that Shen is a powerful language, but worry because it isn't widely used. In a competitive situation, that's an advantage. Shen's power is multiplied by the fact that your competitors don't get it.

If you think of using Shen in a startup, you shouldn't worry that it isn't widely understood. You should hope that
it stays that way.

Bam! A precise reversal of OS philosophy. Competitive edge requires access to technology your competitors don't understand.

.... programming languages are different: programming languages are not just technology, but what programmers think in. They're half technology and half religion. And so the median language, meaning whatever language the median programmer uses, moves as slow as an iceberg. Garbage collection, introduced by Lisp in about 1960, is now widely considered to be a good thing. Runtime typing, ditto, is growing in popularity. Lexical closures, introduced by Lisp in the early 1970s, are now, just barely, on the radar screen. Macros, introduced by Lisp in the mid 1960s, are still terra incognita.

Bam! Cognitive social change is very very slow. Tying your project to it is as good as ossification.

You will have a hard time convincing the pointy-haired boss to let you build things in Shen, when he has just read in the paper that some other language is poised, like Ada was twenty years ago, to take over the world. But if you work for a startup that doesn't have pointy-haired bosses yet, you can, like we did, turn the Blub paradox to your advantage: you can use technology that your competitors, glued immovably to the median language, will never be able to match.

The OS idea that technology can only be levered to advantage by mindshare is historically naive. The French were annihilated at Agincourt by a technology that had very little mindshare outside England.

Closed Source Shen Professional Leaves Open Source Behind

In September 2015, after analysing the reasons why OS Shen was massively underperforming, I moved my development to closed source. Since that time the closed source Shen, Shen Professional, has developed rapidly with funding from a small group of professional programmers and investors. Concurrency, compiler optimisation, large file handling, HTML generation, and graphics have emerged or are emerging within a 6-8 month interval. We're beginning our first commercial application - a health app for yoga enthusiasts. Private money and market economics worked their usual magic.

Lessons for Developers

I think there is a useful lesson for developers to learn from the Shen project; especially when they are working at novel computing paradigms. OS may not empower your project and indeed with a 98% failure rate is unlikely to do so. As David MacIver very rightly remarked after learning the weakness of OS.

Plans which are predicated on changing the world before anyone will pay you any money are decidedly bad plans.

Throwing in the Towel David MacIver

If you've worked in an OS environment, you'll quickly discover that only about 2% of your users will financially support your work. So it makes sense to concentrate on that percentage. Open Source may be useful in establishing your reputation, but once you are established you need to capitalise on that reputation and be leery of fairy economics. At some point, you'll have to think about close sourcing either code or documentation or training because open source is not a business model.

Stuff to Read

The Problems of Open Source was a pioneering essay I wrote back in 2009 laying out the flaws in the OS approach.  I revamped it in 2014. 

Open Source is Not a Business Model

Open Source is Not A Business Model (and again)

Open Source is Not a Business Model (and yet again)

That's three articles with the same title!  Are people starting to learn?  Seems so; more of the same.

The Money in Open Source Software

Corruption and hypocrisy in the open source movement.  Read here

7 Reasons Why I do not Support the FSF