http://www.synack.net/~bbraun/writing/osfail.html

Why Darwin Failed

Apple's Open Source efforts with the Darwin project have been a
failure. Apple failed to build a community around Darwin in the 7
years since its original release because it was not a corporate
direction, but rather a marketing stunt. Culturally, Apple did not
and does not understand what it means to be open source or to build a
working community.

When Apple started shipping Mac OS X Server, it included some GPL'd
projects in addition to many other open source projects. A few
engineers working on these projects were able to convince some
marketing people and some managers that since they were required to
release some of the source anyway, why not be good citizens and
release modifications to all of their open source software? From
there it was one short marketing leap away from releasing the low
level components of Mac OS X as open source, and one large,
consistent project to release the entire base of Mac OS X as a
standalone OS. During the heyday of the dot com days, open source was
good. Everyone wanted a piece of the open source pie, and this was
Apple's way of getting it. This was great all the way around, the
marketing department wins, the engineers win, and the open source
world (appears to) win. However, just throwing some source out there
and calling it good isn't enough. Darwin 0.3 was source only and
didn't attract much attention. Several Apple engineers worked,
largely spending more time and effort that they needed, to build a
standalone OS based on the sources. They came up with a build system
that was compatible with Apple's internal build system, built up a
dependency graph for the projects, and ultimately were able to
release an OS. This was something concrete that they could show. This
is what Apple is offering the open source community, a new, free
operating system.

With something to show for their efforts, one Apple engineer in
particular went above and beyond calling attention to and attracting
developers for their new operating system. This was great, they now
had something to build a community around, they just needed to get
the word out. The word definitely got out, and people started
arriving to take a look. Darwin was brand new, this was its first
release, and it wasn't as polished as the established OS' and
certainly wouldn't be mistaken as a corporate product. It isn't easy
to take a collection of code that doesn't build, and create an OS out
of it, but a small number of Apple engineers did so, often on their
own time, and it showed. To install the OS, you needed a Mac with OS
9 to dump a backup image onto a partition. The source required click
through registration to acquire, and when you did finally download
it, it was impossible to build. However, some people did make it
through all the hurdles, understood it was a brand new project, and
were excited by the possibility of running a free UNIX based OS on
their Mac. Some were just excited to be part of a brand new project,
getting in on the ground level. Everyone involved could feel the
potential of project. Apple was a large company with lots of
engineers, and if they got behind an open source effort, this could
be great.

The potential was slow to be realized, however. Darwin 1.2 came, as
did Mac OS X Public Beta. With the release of Mac OS X Public Beta,
the Darwin community got a hint that Apple doesn't care about Darwin
and it's community, but rather the product Mac OS X. More people came
to the Darwin lists that were not running Darwin, but Mac OS X Public
Beta. Some of the Darwin source started having dependencies on
non-open source components, and became increasingly difficult to use
on Darwin. Darwin improved slightly between 1.0 and 1.2, but for each
step forward, there was a step back. When Darwin 1.3 was release
along with Mac OS X 10.0, we found Darwin had lost hardware support
that was in Darwin 1.0 and Darwin 1.2. This was highly unusual for an
open source project to remove old hardware support. This of course
was done to Mac OS X as a business decision, and Darwin got what Mac
OS X got. Several drivers became binary only, and we no longer had
the complete source to Darwin. This clearly established that Apple as
a whole worked on Mac OS X. Darwin got engineering leftovers.
However, with Darwin 1.3, x86 became a "supported" platform. Not only
that, Darwin/x86 1.3 booted from a CD and installed rather than
needing to be restored from a back on OS 9. This opened up
possibilities of doing upgrade installs, and potential freedom from
our Mac OS X oppressors, by being able to work on a platform Mac OS X
didn't have. Additionally, it opened Darwin up to a whole new world
of contributors: those without Macs. Clearly, if you had a Mac and
were able to run Mac OS X (Darwin and Mac OS X had the same hardware
requirements), you'd run Mac OS X. But now, people without Macs could
run Darwin, and it gave us hope.

However, there was a problem. Already, over the course of a year,
Apple was starting to alienate some of it's users and contributors.
Apple was developing a reputation for not taking contributions,
particularly not new features or additions. Small bug fixes which
were easily reviewed might be accepted back, but medium sized
features, such as SysV IPC support, were not getting traction. People
were getting upset at spending all of this time working on their
project which was ultimately supposed to benefit Apple, and having
all of their efforts ignored. None of these contributions got a
"sorry, we're not interested" or even an acknowledgment from Apple
engineers. These contributions were almost universally met with
silence, as if they went to /dev/null.

During this time period, Apple ended up hiring over half of their
most active contributors, which amounted to roughly 3-4 people. This
drained the contributor pool significantly, and effectively muffled
the ones that got hired. Once these contributors became employees,
they signed NDA agreements, intellectual property agreements, and
were otherwise restrained from much of the unrestricted participation
they enjoyed prior to employment. They discovered the reason for the
lack of communication, being a culture of secrecy. Even being
perceived as someone who might leak information, even accidentally,
was a liability, and it was best to just not communicate with those
outside Apple, or even outside your own group. Additionally, they
discovered the reason why no one wanted to accept large patches:
whoever submitted them to the build became personally responsible for
the feature, and everyone had more than enough to work on with the
features given to them by managers, let alone people not in their
management hierarchy. Engineers were rated based on how they met
their management's goals, and management's goals did not include
anything related to Darwin or an external community.

As time went on, Darwin proper did not improve. It had the same
crappy shell script installer, poor terminal support, and poor
hardware support on non-Mac OS X supported systems. The major
problems of Darwin did not seem to be getting any attention, source
started disappearing, first from leaf projects, then from
increasingly important projects. The source that was there was
getting increasingly difficult to build as "integration" happened and
formerly standalone projects started having more and more
dependencies, many of which on closed source components. By this
time, pretty much anyone that was interested in Darwin enough to
overcome these obstacles had either been hired by Apple, or left the
community. The Darwin lists were almost entirely populated by people
running Mac OS X and had questions about UNIX problems.

For what it was worth, Apple recognized this problem and tried to act
to correct it. Some people were hired specifically to create a better
Darwin community. For over a year this problem was discussed,
solutions were proposed and rejected, and countless meetings were
held. All the while, less source was available, and what was there
was harder to build. In the end, it was decided an independent
organization to promote Darwin and it's development was in order. It
was clear Apple its self was incapable of creating and sustaining the
community it desired. The end result was the OpenDarwin project,
jointly created by Apple and ISC. Apple provided the hardware, ISC
provided connectivity and power. Much of OpenDarwin's initial content
came from a site called darwinfo.org, which I ran until I became
employed by Apple and was asked to remove much of the functionality.

OpenDarwin was off to a confusing start, but it was going to be the
solution to all our problems. OpenDarwin started with no clear
objective. Was OpenDarwin to repackage Apple's standalone OS and
extend it in ways Apple was unwilling to spend the resources to do?
Was OpenDarwin to provide the necessary tools and environment to
allow people to hack up their Mac OS X systems? Or was OpenDarwin
simply there to provide hosting and resources to projects that were
built on or around Darwin and Mac OS X? Over time, this became a
process of elimination. Providing tools and an environment to allow
people to hack up their Mac OS X systems was largely unsustainable
since any system we came up with would be blown away by the next
Software Update or Dev Tools/XCode Update. Repackaging Apple's
standalone OS became increasingly difficult and time consuming. It
didn't help that most of OpenDarwin's initial members were the former
Darwin contributors now working for Apple on Darwin, and didn't have
the time to devote to such projects. Not to mention having just
worked on Darwin all day, and wanting a distraction when off the
clock. Even just maintaining the infrastructure for hosting projects
was burdensome. The OpenDarwin project was able to attract a marginal
community, but few in the community had the time or dedication to
make any meaningful contributions. Those that did, ended up taking
everything upon themselves and quickly getting burnt out. The
OpenDarwin Project continues to exist in a state of limbo.

With the creation of the OpenDarwin Project, Apple was able to push
open source to the back burner. They were able to check off their
open source agenda item with hardware donations instead of cultural
shift and true interaction. Source was pushed out when binary forms
of Mac OS X were made available to the public, the need to provide
buildable source was diminished, since that was OpenDarwin's problem,
and the engineers were generally happy to have one less thing to be
harassed about.

Just one problem: over the course of several years, the original goal
of creating a free standalone OS and a community around it had
failed. There was no community, and the free standalone OS had become
composed of binary packages and source no one could build. Even
worse, instead of acknowledging the problem, Apple and most of the
world simply forgot about Darwin.

So, how did it fail? What was the problem? The problem was Darwin was
a marketing ploy, and nothing more. Marketing thought that by
throwing some source files over the wall, they could increase their
market share and revenue. And they were right. You go to almost any
technical conference these days, and there is an astonishing number
of Mac laptops in the crowd. Prior to Mac OS X, people would have
been laughed out of the building for having a Mac, but now, with Mac
OS X and the magic of open source, it was suddenly OK or even cool.
Managerially and culturally, Apple had no intention of becoming
involved with any community outside its own walls. There were some
well intentioned engineers and even a few low level managers that
were duped by their own marketing department, and that was all. These
people had given substantial amounts of their free time under the
illusion they were doing something Right, something Good, but in the
end were simply used by their own company's marketing department.

Feb. 23 2006