- This page describes open software development. For openness in aid and international development, see Open International Development.
Open Source Software is defined not just by the type of licenses, but also by a certain kind of development and governance that is open to the community of developers.
- 1 Description
- 2 Characteristics
- 3 Typology
- 4 Discussion
- 5 Interview
- 6 Example
- 7 Key Book to Read
- 8 More Information
"Instead of a management hierarchy dictating how and on what each developer works, volunteers find tasks that interest them and work when they find time. Instead of having the software’s source code (essentially its design) hidden as a company’s proprietary secret, the open source programmers publish their code on the internet for others to read, learn, experiment, modify, improve, and reuse. Instead of erecting organizational barriers to control their development process and their turf, the open source development model brings all its artefacts in the open: discussions between developers and users, problem reports, the historical record of changes made to the software, even results of failed tests. Under this model transparency and self-regulation nurture each other, fostering organic growth and taming complexity." (http://www.re-public.gr/en/?p=97)
Weber suggests eight principles to capture the essence of the open source process:
“Make it interesting and make sure it happens.” -- Combinations of attraction and persuasion point participants toward problems.
“Scratch an itch.” -- Address an immediate problem or opportunity.
“Minimize how many times you have to reinvent the wheel.” -- Freed from worry about “lock-in,” adopt others’ foundation solutions.
“Solve problems through parallel work processes whenever possible.” -- Traditional development process relies upon an engineering archetype, in which an authority decides the path to be followed in development. Open source process follows an evolutionary archetype, in which the community allows multiple parallel paths to generate multiple alternative solutions, from which successful solutions are later selected.
“Leverage the law of large numbers.” -- Open source projects engage many bug-hunters and bug-fixers.
“Document what you do.” -- The larger, dispersed community is more reliant on documentation.
“Release early and release often.” -- The evolutionary model rewards frequent iterations (generations), but puts a strain on those that must select among many submitted solutions.
“Talk a lot.” -- Direct communication within the community, with lots of open conflict, typifies open source processes." (http://www.dougsimpson.com/blog/archives/000363.html)
For more, see also our page on the Open Development Method.
"I would rather see the OSI, or an entirely new entity, patronise this concept, which should be fairly easy (though not trivial) to protect as there might be some objective criteria to tell open development from plain Open Source. A first stab at what these criteria could be:
1. an Open Source license, of course;
2. a non-discriminatory access to the developer’s community;
3. a well-defined and stated process for people to get involved;
4. a neutral and self-elected governing body;
5. (more difficult, could mean having a preferential lane) a neutral party such as a foundation owning the code.
The OSI (or its spinoff, or whatever) would assert whether a community deserves the status of “open development”, much like today the OSI is approving licenses. The process could be standardised by providing “open development processes” in terms of bylaws and charters which could then be just applied via find/replace much like what happens today with licenses. The specific authorization to use the “brand” should be temporary and subject to arbitration if someone complains about a community not respecting the self-inflicted rules." (http://feather.planetapache.org/2006/03/08/should-osi-redefine-the-label-open-source/)
From a SAP White Paper, "three key characteristics of open source projects:
Open source is said to be based on the principle of meritocracy. We have found that the principle of meritocracy is used as an umbrella term for the following three more specific principles of open source:
Egalitarian. Everyone can contribute, because open source projects are accessible on the Internet and the project community is typically inclusive to anyone who wants to help.
=> Project members need to be inclusive of whoever comes along to help rather than viewing them as a foreign element. Documenting the project with future readers in mind is a core aspect of this.
Meritocratic. Contributions are judged transparently and based on their merits. All decisions are discussed publicly on mailing lists and can be looked up for reference.
=> Project members need to realize that important input and contributions can come from all across the organization based on perspectives that may be unfamiliar to the original developers.
Self-organizing. There is typically no defined process imposed from the outside so the project community itself determines how to go about its work. "
=> projects need to be accommodating of their volunteers and respectful of their time. (http://www.riehle.org/publications/2008/bringing-open-source-best-practices-into-corporations-using-a-software-forge/)
3. Paul Anderson:
"Ross Gardler, fomer Manager of OSS Watch, believes that ODM can be formalised by looking at a number of key attributes that characterise an open development community, namely:
- a deep level of user engagement: if you don’t have users then there is no point having a project.
- transparency: being open in what the community is undertaking and the way decisions are made.
- collaboration: a means of working within a diverse group of people, something that the Internet has obviously made easier.
- agility: once work begins and there is a serious engagement with users, ideas and plans may need to change.
- sustainability: having the capacity to keep developing an application solution over the necessary period of time.
- tools: wikis, bug- and version-trackers and email lists to support the development of the community and
- keep track of intellectual property rights, if relevant."
Peter Hoddinott and Tony Bailetti  define four possible scenario’s mixing open source code with an open or closed process of developing it:
“the production of the code lacked key open source characteristics such as:
• No external contributors: all code was developed in-house prior to being published on the Internet
• No visibility of who developed the code and when they developed it
• No mechanisms were available to the general public for
(i) contributing to the production of the code prior to its release in Sourceforge.net or
(ii) participating in the governance structure of the organization that produced it
In short, the code was open, but the process used to produce it was closed. There was no public community behind the production of the code, and no accommodation for such a community.
Our discussion then turned to the ambiguity in the usage of the word "source". Does source mean the computer code written in a recognized programming language, or the process used to produce the code, or something else? If we allow source to mean two different things:
(i) the process used to produce the code, and
(ii) the computer code, four cases are possible:
1. Open process and open computer code
2. Closed process and open computer code
3. Open process and closed computer code
4. Closed process and closed computer code
We surmise that many use the term open source with case 1 in mind. For example, the Eclipse code is licensed under the OSI approved Eclipse Public License (EPL). The central repository for the code base is available and the general public can, with ease, track the pedigree of the code. Release dates are known and published. The general public is encouraged to contribute to the organization itself, to define projects, and to write code. Moreover, the governance structure used to manage all the Eclipse projects is transparent.
Case 2 was outlined above, and it characterizes what our software professional called "open code" but not "open source". A fundamental contradiction seems to exist when an open source asset is developed using a process controlled by a single party. For example, the Open Office project has been criticized. for encouraging a development culture that differs radically from the open-source norm The majority of the contributors to the Open Office project work for Sun Microsystems.
Case 3 includes instances of organizations and individuals that produce a reference implementation of some standard to accelerate that standard's adoption. The code of the reference implementation is typically produced using open processes but the code itself may not be released under an open source licence. The reason is straightforward: releasing the code under an open source licence would potentially weaken the purpose and authority of the standard by facilitating the ease by which deviations of the standard may be introduced.
Case 4 includes instances which are typically referred to as proprietary or closed software. The process used to produce the code is closed, and the software is released using a non-open source licence. This case includes instances where several organizations create a consortium to produce code that only those with membership within the consortium have visibility and access to. Typically, such consortiums have a tiered membership that stipulates the members' rights with respect to the code.
It can also be argued that the use of "source" does not distinguish between the three types of code, any of which could be open or closed. Source can mean:
(i) the code used to implement a system or component,
(ii) the interface where what we open is the application programming interface (API), or
(iii) the data underlying the implementation where most any application or system creates value from the data underlying it.
It can also be argued that source is not limited to computer code. One could extend the four cases described above to combine open and closed processes with hardware schematics or documentation, two examples of assets which are increasingly being considered as open.” (http://www.osbr.ca/archive.php?issue=10§ion=Ar#A2)
The four factors of openness
Peter Hoddinott and Tony Bailetti :
“we had occasion to find ourselves struggling as we tried to make sense of instances of the word open in the context of community code. For example, we found instances where open meant that releases of the code were made available to the general public (i.e., non-members of a consortium); however, releases to the general public were delayed 12 months from the time it was available to the members of the consortium. We also found instances where what open meant depended upon the level of membership. The more expensive memberships provided these members more privileges to participate in and influence the processes, for example with veto power. In these examples, open is not equated with full access; instead, open is a matter of degree and that degree is metered out in a distinctly defined hierarchy of privilege.
This seeming confusion and differences about what is open and what is source and the use of open source to refer to phenomena that fall well outside the OSD, led us to conclude that we need to better understand the characteristics of the systems in which open source assets are produced, used and distributed.
We conceptualize any such system as being comprised of four components:
1. Network: the network of individuals and organizations that produce, use and distribute an asset
2. Processes; the processes, approaches, rules and understandings that lead to the production, use and distribution of an asset
3. Governance: the governance structure of the organization and the projects within the organization
4. Value: value created through collaboration and value appropriated through competition
We observe that a healthy open source system is required to compete with a strong proprietary system; that is, an open source system cannot compete by virtue of the distribution license alone.
What one would generally agree upon as being an open source system could be expressed in terms of the health of its four components. Such a definition would not be static, but would arguably be more accurate and useful in determining the true value of an open source system. For example, one could speak in terms of a system not having reached the status of being open source until it is deemed to be healthy. And an open source system may subsequently cease to be an open source system if its health deteriorates beyond some point. This line of inquiry could then further leverage health of the four components as a means for distinguishing other types of systems such as closed systems and community systems.
Our initial suggestion as to what is relevant for assessing the health of each of the four components of a system is as follows:
1. Network: Large, distributed and diverse: We distinguish between an asset produced by a well developed network from an asset produced by a small number of collocated producers who have similar characteristics. A general reference model for an open source asset would be one that is produced by a well developed network that is able to integrate, test, and quality assure contributions from a large number of diverse individuals and organizations dispersed throughout the world
2. Process: Includes meritocracy where one is recognized for the quality of their contributions; transparency in communications and guidelines; recruitment and promotion methods; and mechanisms for dealing with difficult people
3. Governance: Includes participation; relationship between contribution and the influence that can be asserted; membership's influence over a project, influence over the overall system governance, and ability to alter the governance structure
4. Value creation and appropriation: Usefulness of the asset; how free-riders are addressed--if it is too easy to appropriate value no one would pay for a membership or undergo an apprenticeship to move from being a developer/contributor who writes code or documentation to a committer with write access to the codebase; access to the asset by virtue of the license.” (http://www.osbr.ca/archive.php?issue=10§ion=Ar#A2)
The Role of Community in Development
Nelson Ko :
"Understanding the nature of the community that produces the open source software is a critical part of evaluation that is often overlooked by users who are new to open source software. Unlike closed source alternatives, key support options for open source software include non-commercial channels provided by an extended community. Therefore, important criteria that should be considered when evaluating open source software include the size and vibrancy of the community, the availability of online documentation, and access to support via mailing lists, forums, and IRC (Internet Relay Chat). One source for such information is statistics on sites such as SourceForge and Ohloh. However, both sites focus on the contribution of developers and it is important to note that contribution to an open source community is more than just commits to the codebase. As in commercial software, production of good open source software also requires documentation, testing, support, training, and the incorporation of user feedback. An understanding of the maturity of the community can help to answer questions such as "what support mechanisms are available if we roll out this software?" and "how difficult will it be to install and use this software?"An evaluation of an open source community should also consider the broader ecosystem in which it exists. An environment in which open source is prevalent results in consumer choice that is arguably unparalleled in closed source ecosystems. Open source software has a clear advantage in some domains, for example, in the area of wikis. Freely downloadable open source solutions are plentiful and comparable in terms of quality to their expensive, closed source counterparts.
A deeper understanding of the nature of the community is also critical in determining the nature of support that will be available for an open source software. It is important to consider the nature of the resources the community has to offer in relation to the capabilities of your organization. For example, the Tikiwiki and Drupal communities have a large number of highly technical members who can provide support at a high level of technical sophistication. On the other hand, another popular content management system Joomla, has a relatively smaller team of technical experts combined with a larger community of less technical but more design oriented individuals such as graphic designers and web designers. Largely as a result of this structure, the Joomla support forums tend to become somewhat overwhelmed with questions of a "how-to" nature. Discussions over more complex technical issues are therefore less prominent. On the other hand, it is much easier to find consultants that are able to make low cost design customizations for Joomla than it is for Drupal or Tikiwiki. Another benefit of open source software like Joomla that is exposed to a wider swath of mainstream users is that they tend to be more user-friendly, although this usually comes at a price of reduced functionality." (http://www.osbr.ca/archive.php?issue=10§ion=Ar#A6)
" at the end of the day, the most crucial issue is the development community. It is the strength and the diversity of the development community which is the best indicator for the health and the well-being of an Open Source project.
But what about end-users, I hear people cry? End users are important, to the extent that they provide ego-strokes to the developers, and to the extent that they provide testing and bug reports to the developers, and to the extent that they provide an economic justification to companies who employ open source developers to continue to do so. But ultimately, the effects of end-users on an open source project is only in a very indirect way.
Moreover, if you ask commercial end users what they value about Open Source, a survey by Computer Economics indicated that the number one reason why customers valued open source was “reduced dependence on software vendors”, which end users valued 2 to 1 over “lower total cost of ownership”. What’s important to commercial end users is that they be able to avoid the effects of vendor lock-in, which implies that if all of the developers are employed by one vendor, it doesn’t provide the value the end users were looking for.
This is why whether a project’s developers are dominated by employees from a single company is so important. The license under which the code is released is merely just the outward trappings of an open source project. What’s really critical is the extent to which the development costs are shared across a vast global community of developers who have many different means of support. This saves costs to the companies who are using a product being developed in such a fashion; it gives choice to customers about whether they can get their support from company A or company B; programmers who don’t like the way things are going at one company have an easier time changing jobs while still working on the same project; it’s a win-win-win scenario.
In contrast, if a project decides to release its code under an open source license, but nearly all the developers remain employed by a single company, it doesn’t really change the dynamic compared to when the project was previously under a closed-source license. It is a necessary but not sufficient step towards attracting outside contributors, and eventually migrating towards having a true open source development community. But if those further steps are not taken, the hopes that users will think that some project is “cool” because it is under an open-source license will ultimately be in vain. The “Generation Y”/Millennial Generation in particular are very sensitive indeed to Astroturfing-style marketing tactics." (http://thunk.org/tytso/blog/2008/04/26/organic-vs-non-organic-open-source-revisited/)
Six Limitations of the current model
Felix Stadtler :
"Six Limitations to the Current Open Source Development Methodology
The "Open Source Approach" to develop informational goods has been spectacularly successful, particularly in the area for which it was developed, software. Also beyond software, there are important, successfull Open Source projects such as the free Encyclopedia, Wikipedia; collaborative sites writing/publishing projects such as koro5hin.org; and the Distributed Proofreading Project, attached to the Gutenberg Project.
However, particularly outside the software domain, the Open Source projects remain relatively marginal. Why? Some of it can be explained by the relative newness of the approach. It takes time for new ideas to take hold and to be transferred successfully from one context to another. But this is only part of the story. The other part is that the current development model is based on a number of specific, yet unacknowledged conditions that limit its applicability to more diverse contexts, say the music distribution or drug research.
The boundaries to the open production model as it has been established in the last decade are set by six conditions characterizing virtually all of the success stories of what Benkler called "commons-based peer production." The following list is a conceptual abstraction, a kind of ideal-type. The actual configuration and relative importance of each condition varies from project to project, but taken together they indicate the boundaries of the current model. In this elaboration, I draw from examples of free and open source software, but it would be simple to illustrate these limitation based on open content projects.
1) Producers are not sellers
The majority professional, i.e. highly-skilled, programmers do not draw their economic livelihood from directly selling the code they write. Many work for organizations that use software but do not sell it, for example as system administrators. For them the efficient solution of particular problems is of interest, and if that solution can be found and maintained by collaborating with others, the sharing of code is not an issue. For others employed in private sector companies, for example at IBM, the development of free software is the basis for selling services based on that code. The fact that some people can use that code without purchasing the services is more than off-set by being able to base the service on the collective creativity of the developer community at large. From IBM's point of view, the costs of participating in open software development can be regarded as 'capital investment' necessary for the selling of the resulting product: services.
For members of academia (faculty and students) writing code, but not selling (often explicitly prohibited), contributes to their professional goals, be it as part of their education, be it as part of their professional reputation-building. For them, sharing of code is not only part of their professional advancement, but an integral part of the professional culture that sustains them also economically,. in form of salaries for the faculty and stipends for the (graduate) students.
Last but not least are all those who use their professional skills outside the professional setting, for example at home on evenings and weekends. Having already secured their financial stability, they can now pursue other interests using the same skill set.
2) Limited capital investment
Particularly the last, and very important group of people, whose who work outside the institutional framework on projects based on their own idiosyncratic interests, can only exist due to the fact that the means of production are extraordinarily inexpensive and accessible. Materially, all that is needed is a standard computer (often even a substandard one would already suffice) and a fast, reliable connection to the communication forums of the community. Of course, the computer and the network rely on a level of infrastructure that cannot be taken for granted in large parts of the world, but for most people in the centers of development, they are within relatively easy reach.
Once this access to be means of communication is secured, the skills necessary to participate in the development of code can also be acquired collaboratively, free of charge. The number of self-taught programmers is significant. Since no expensive diplomas are necessary to become active, the financial hurdle is, indeed, extraordinarily low.
3) High number of potential contributors
Programming knowledge is becoming relatively common knowledge, no longer restricted to an engineering elite, but widely distributed throughout society. Of course, truly great programmers are rare as truly great artists are, but average professional knowledge is widely available. This has a quantitative and a qualitative dimensions. Quantitatively, the number of able programmers is in the millions, and rising. Qualitatively, the range of people capable programmers is also unusually wide, not the least because the material hurdles are so low and the learning can take place outside of institutions with entry exams and tuition fees. This large and diversified pool of talents makes it possible to create the critical mass of contributors out of only a fraction of population.
4) Modularized Production
A large software program consists of many smaller code segments (libraries, plug-ins etc.)This makes it possible to break down the production process into many small steps which can be carried out by distributed contributors. If the act of integration is relatively straight forward, it allows to keep the amount of work that each has to contribute highly flexible and also make use of smaller contributions (bug reports, patches). Furthermore, the modularity of the production process allows a high number of people to work in parallel without creating significant interferences.
5) Producers Are Users
According to Eric S. Raymond, a good open source projects starts with a programmer scratching his own itch and finding out in the process that there are many others with the same problem. Wanting to use a program is a great motivation of contributing to developing it. Often, it's much more efficient that waiting, hoping that someone will write and sell a program that will address one's particular need.
6) No Liability
Last, but not least, software has no product liability. Paragraph 11 of the GPL states, similar to most other licenses, that "the copyright holders and/or other parties provide the program 'as is' without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose" (GPL, v2). The absence of liability makes it possible to produce a program without having to assign clear ownership, or other markers allowing to determine liability, to it.
The space delimited by these condition is large and still not fully explored. We can expect that the current open production model will find additional niches in which it can thrive. Few could have predicted the success of Wikipedia only three years ago, even though Open Source Software had already been very successful at the time. However, it is also clear that many information goods fall outside of this space. Not always are the means of production inexpensive and readily available or the production process modular. Sometimes, the number of potential producers is small, more often than not are the producers not the users of their own products, and, in many cases, product liability is desirable.
This does not mean that the "open source model" cannot apply to, say, the production of literary works, music, or medical drugs. What it means, however, is that to make it viable, another round of social innovation is required. This is slowly happening. The growth of "Open Access Journals" or discussions around "compulsory licensing" are good, though very early examples." (http://amsterdam.nettime.org/Lists-Archives/nettime-l-0308/msg00043.html)
Does its model survive corporate cooptation
Excerpts taken from an article by Doc Searls in Linux Journal, April 2008, at http://www.linuxjournal.com/content/linux-now-slave-corporate-masters
To what degree does the fact that free software programmers now get paid, change the internal dynamics of peer production? The following debate, taken from Linux Journal, effectively answers that question,while at the same time confirming the huge commercialization that has been going on.
On April 30, the Linux Journal asked an important question: Is Linux now a slave to corporate masters? Does it matter who pays the salaries of Linux kernel developers? If so, how much, and in what ways?.
This question was prompted by a report from the Linux Foundation, on the characteristics of those working on the Linux kernel.
Tom Slee has summarized the findings:
“One of the highlights: “over 70% of all kernel development is demonstrably done by developers who are being paid for their work”. 14% is contributed by developers who are known to be unpaid and independent, and 13% by people who may or may not be paid (unknown), so the amount done by paid workers may be as high as 85%. The Linux kernel, then, is largely the product of professionals, not volunteers.
So Linux has become an economic joint venture of a set of companies, in the same way that Visa is an economic joint venture of a set of financial institutions. As the Linux Foundation report makes clear, the companies are participating for a diverse set of commercial reasons. Some want to make sure that Linux runs on their hardward. Others want to make sure that the basis of their distribution business is solid. And so on, and none of these companies could achieve their goals independently. In the same way, Visa provides services in many different locations around the world in different sizes and types of stores. Some banks need their service mainly in one country, some in another, but when they work together they all get to provide their services all around the world.
…the Linux Foundation report has made clear that open source has crossed its commercial Rubicon, and there is probably no going back.” (http://whimsley.typepad.com/whimsley/2008/04/linux-grows-up.html)
Nick Carr predictably concludes from this:
“The shift in Linux kernel development from unpaid to paid labor, from volunteers to employees, suggests that the Net doesn’t necessarily weaken the hand of central management or repeal all the old truths about business organization.” (http://www.roughtype.com/archives/2008/04/open_source_as_1.php)
This specific argument is addressed by Timothy Lee, who is essentially saying that the corporatization of Linux has not changed its underlying organisational model:
“For starters, most of the people contributing to the kernel are professional programmers, and most professional programmers have jobs in the software industry. So it’s totally unsurprising that most kernel contributors work for software companies. But Carr’s observation also misses the point in a deeper way. What makes the open source model unique isn’t who (if anyone) signs the contributors’ paychecks. Rather, what matters is the way open source projects are organized internally. In a traditional software project, there’s a project manager who decides what features the product will have and allocates employees to work on various features. In contrast, there’s nobody directing the overall development of the Linux kernel. Yes, Linus Torvalds and his lieutenants decide which patches will ultimately make it into the kernel, but the Red Hat, IBM, and Novell employees who work on the Linux kernel don’t take their orders from them. They work on whatever they (and their respective clients) think is most important, and Torvalds’s only authority is deciding whether the patches they submit are good enough to make it into the kernel. Carr suggests that the non-volunteer status of Linux contributors proves that the Internet “doesn’t necessarily weaken the hand of central management,” but that’s precisely what the open source development model has done. There is no “central management” for the Linux kernel, and it would probably be a less successful project if there were.” (http://www.technologyowl.com/i88997-c134-rss)
Ed Cone confirms:
“What that kind of analysis is missing is that IBM is paying engineers to work on projects that IBM doesn’t own, or solely direct. You pay these engineers — but of all the relationships between senior management and line employees, the fact you are paying them is about the least important, institutionally. The idea that the minute you pay people to do something, you have the right to manage them and the right to completely take over that work for the benefit of the company — that’s not true. IBM is not producing that code, IBM engineers are. IBM is paying those people because it’s getting value out of them — Linux creates value for the enterprise, it lowers our cost of managing software, it increases peoples’ budgets for hardware and services — but there’s this crazy middle step where Linux is not now and cannot be owned or controlled by IBM. Linux is a brutal technical meritocracy, and there is no senior manager at IBM who can say, “I don’t care what the kernel engineers think, I want this.” They can’t put it into the product without appealing to people who don’t work for them. If they announced a strategic change in the kernel they would be laughed out of the room. They have given up the right to manage the projects they are paying for, and their competitors have immediate access to everything they do. It’s not IBM’s product.
There is a kind of perverse misreading of the change here to suggest that as long there are paid programmers working on the project, it’s not developing in any way different from what’s going on inside traditional organizations. It badly misunderstands how radical it is to have IBM and Novell effectively collaborating with no contractual agreement between them, and no right to expect that their programmers’ work is going to be contributed to the kernel if people external to those organizations don’t like it. And that’s a huge change.
When people read those statistics, they think, If there’s a salary, then all the other trappings of management must go along with it. Not only is that not true, it’s actually blinds you to the fact that paying someone a salary without being able to direct their work is probably the biggest challenge to managerial culture within a business that one can imagine.” (http://blogs.cioinsight.com/knowitall/content001/decoding_the_professionalization_of_linux.html)
Doc Searls then concludes the article with some personal observation and especially quoting a personal testimony by Andrew Morton, confirming the indepedence of kernel programmers:
“Andrew went out of his way to make clear, without irony, that the symbiosis between large vendors and the Linux kernel puts no commercial pressure on the kernel whatsoever. Each symbiote has its own responsibilities. To illustrate, he gave the case of one large company application.” (http://www.linuxjournal.com/article/8664)
Open Source Versus Open Development
ustin Erenkrantz, Apache Software Foundation’s president, interviewed by Paul Anderson, in 2009:
"Justin believes open development is crucially important for long-term success and notes that two ‘flavours’ of open source are emerging. He says: ‘We are seeing a difference between open source and open development. So what we are seeing is some projects and entities who are saying here is the source code, it is under Apache licence or GPL or whatever but the decisions about how the code got to that state are behind some wall—it is not in public’. This means that a developer has all the requisite rights to edit, modify and redistribute the code, but doesn’t have any real understanding as to how or why the code developed that way in the first place. Justin says: ‘So you run into some bug and you are saying why is the code this way? There is no historical context or archiving or rationale…So, [with the open development method] what we are seeing is, that in addition to “here is the code” we are seeing “here are all the decisions being made in public”.’
The other strength of the open development method is the business-like way the ASF develops software and the peer review processes that are in place. As Justin notes, ‘We see here with the establishment of the ASF, all the work [is] done in the open, everything is peer-reviewed and this [has] led to very high quality’. This focus on the work of the developer and the need to perform to the very highest standards, in public, is reflected in the structure of the organisation.
The very first stage for a developer in the Apache development model is to be a simple ‘User’—making use of the software and offering feedback, bug reports and ideas. Evolution to the next stage is by proving your worth and getting more involved in the development process. Justin describes it as: ‘Proving over time. Being able to say you can work with the community, you can send in patches, you can add to the documentation or whatever it is.’ People at this level are known as ‘developers’ and they can suggest modifications to the code. They cannot, however, directly edit the core code in the software repository, a job that is allocated to those with more experience who are known as ‘committers’. Finally, after having been elected by others, you can become a ‘member’. This entails becoming a ‘shareholder’ of the not-for-profit ASF charitable foundation and acquiring voting rights on new members and the Board of Directors.
As a way of providing some idea of the scale of the organisation Justin says: ‘There are around 260 or so members of the Foundation and 1,700 committers who can contribute directly to some part of the code base3. If these guys stick with it then at some point they will become members.’
Loyalty, talent and sheer hard work are core Apache traits and the ASF website was recently retagged with the slogan Meritocracy in Action. In actual fact, the term ‘meritocracy’ was originally coined by British social theorist Michael Young in a famous book, The Rise of the Meritocracy. His vision of a society based purely on merit was actually a dystopian warning. When I point this out, Justin laughs loudly and says: ‘I believe Roy Fielding, who was one of the first chairs of Apache, has a background in social science and was keenly aware of that.’ Justin argues that the term neatly sums up their overall approach, saying: ‘Actually it is a good representative and [is] basically saying “If you want to do something and you have demonstrated to the community that you are trustworthy and you have this merit then we will try and give you the environment to let you do the things you want to do”.’
Pragmatic Open Source Another distinction between open source and open development comes to light when Justin talks about proprietary software. This touches on a hotly debated bone of contention between the ‘free’ (as exemplified by Richard Stallman and the Free Software Foundation, or FSF) and ‘open’ software movements, and manifests itself in the different approaches taken to the development of code.
For example, the FSF is quite clear that ‘free’ software respects an individual user’s freedom. A program can only be classified as free software if its licence provides the user with four essential freedoms: the freedom to run the program as they wish; the freedom to study how the program works and adapt it; the freedom to redistribute copies; the freedom to improve the program and release improvements to the public. The Apache License Version 2.0, however, makes it clear that the source code can be modified and then incorporated into a proprietary product. This means that while Apache-licensed code qualifies as free software, products built using that code may not be free software.
Justin argues that this is simply being pragmatic and reflects what the developers want, and says: ‘By and large you’ll see that Apache projects tend to be focused on developers. These developers are primarily our target audience. One of the things with Richard Stallman’s view is that he wants the users to be free. Our attitude within Apache is that we are looking for the developers to have the freedom. They are saying: “You know what, if you want to make this into a closed source thing or if you want to make it into an open source thing then that’s fine”.’ And he adds: ‘We don’t really care necessarily how or where you use it as long as you abide by our licence, that you give us credit etc. But we are not necessarily going to say “hey you have to use it in a free and open source product”.’" (http://oss-watch.ac.uk/resources/erenkrantz)
"How do new features find their way into "legacy infrastructure" open source projects such as the Linux kernel? In other words, what is the requirements analysis and planning process?
- First up, with a legacy project, the feature set tends to be well understood.
We're implementing 30-year-old technology, so we're working to all that prior understanding of how these things should traditionally operate. This removes a lot of uncertainty from the design process.
And to a large extent we're strongly guided by well-established standards: POSIX, IEEE, IETF, PCI, various hardware specs, etc. There's little room for controversy here.
- Generally, new features are small (less than one person-year) and can be handled by one or two individuals. This is especially true of the kernel which, although a huge project is really an agglomeration of thousands of small projects. Linus has always been fanatical about maintaining the quality and sanity of interfaces between subsystems, and this stands us in good stead when adding new components.
This agglomeration of many small subsystems fits well into the disconnected, distributed development team model which we use.
If the project was a large greenfield thing, such as, say, an integrated security system for the whole of San Jose airport then open source development methodologies would, I suspect, simply come undone: the amount of up-front planning and the team and schedule coordination to deliver such a greenfield product is much higher than with "legacy infrastructure" products.
The resourcing of projects in the open source "legacy infrastructure" world is interesting. We find that the assignment of engineering resources to feature work is very much self-levelling. In that if someone out there has sufficient need for a new feature, then they will put the financial and engineering resources into its development. And if nobody ends up contributing a particular feature, well, it turns out that nobody really wanted the feature anyway, so we don't want it in the kernel. The system is quite self-correcting in that regard.
Of course, the same happens in conventional commercial software development: if management keeps on putting engineers onto features which nobody actually wants then they won't be in management for very long. One hopes. But in the open source world we really do spend zero time being concerned with programmer resource allocation issues -- the top-level kernel developers never sit around a table deciding which features deserve our finite engineering resources for the next financial year. Either features come at us or they do not. We just don't get involved at that level.
And this works. Again, because of the nature of the product: a bundle of well-specified and relatively decoupled features. If one day we decided that we needed to undertake a massive rewrite of major subsystems which required 15 person years of effort then yes, we'd have a big management problem. But that doesn't happen with "legacy infrastructure" projects.
Development processes and workflow
- All work is performed via email. Preferably on public mailing lists so a record of discussions is available on the various web archives. I dislike private design discussions because it cuts people out of the loop, reduces the opportunity for others to correct mistakes and you just end up repeating yourself when the end product of the discussion comes out.
- Internet messaging via the IRC system is used a little bit, but nothing serious happens there -- for a start it's unarchived so for the previously mentioned reasons I and others tend to chop IRC design discussions off and ask that they be taken to email.
- We never ever use phone conferences.
- The emphasis upon email is, incidentally, a great leveller for people who are not comfortable with English -- they can take as much time as they need understanding and composing communications with the rest of the team.
- Contributors send their code submissions as source code patches to the relevant mailing lists for review and testing by other developers. The review process is very important. Especially to top-level maintainers such as myself. I don't understand the whole kernel and I don't have the time or expertise to go through every patch. But I very much like to see that someone I trust has given a patch a good look-over.
- When a patch has passed the review process it will be merged into one of the many kernel development trees out there. The USB tree, the SCSI tree, the ia64 tree, the audio driver tree, etc. Each one of these trees has a single top-level maintainer.
I run a uber-tree called the "mm kernels" which integrates the latest version of Linus's tree with all the other top-level trees (32 at the last count). On top of that I add all the patches which I've collected from various other people or have written myself -- this ranges from 200 to 700 extra patches. I bundle the whole lot together and push it out for testing maybe twice a week.
When we're confident that a particular set of patches has had sufficient test and review we will push that down into Linus's tree, which is the core public kernel, at kernel.org.
- Vendors such as Red Hat and Suse will occasionally take a kernel.org kernel and will add various fixes and features. They will go through a several month QA cycle and will then release the final output as their production kernel.
- The preferred form of bug reports from testers is an email to the relevant mailing list. We go through a diagnosis and resolution process on the public list, hopefully resulting in a fix. This whole process follows a many-to-many model: everyone gets to see the problem resolution in progress and people can and do chip in with additional suggestions and insights.
This process turns out to be quite effective.
- We do have a formal web-based kernel bug reporting system, using bugzilla. But the bugzilla process is one-to-one rather than many-to-many and ends up being much less effective because of this. I screen all bugzilla entries and usually I'll bounce them directly to up email if I think the problem needs attention.
The mailing lists are high volume and it does take some time to follow them. But if a company wishes their engineering staff to become as effective as possible with the open source products, reading and contributing to the development lists is an important function and engineer time should be set aside for this." (http://www.groklaw.net/article.php?story=20041122035814276)
Citations on the role of Emergence in the Linux Development process
Selected by Karim Lakhani:
Rik van Riel: “It seems like Linux really isn't going anywhere in particular and seems to make progress through sheer luck”
Linus (in several emails in a longer thread):
“Hey, that's not a bug, that's a FEATURE![his emphasis]
“Do I direct some stuff? Yes. But, quite frankly, so do many others. Alan, Al, David, even you. And a lot of companies are part of the evolution whether they realize it or not. And all the users end up being part of the ‘fitness testing’....
“A strong vision and a sure hand sound good on paper. It's just that I have never met a technical person (including me) whom I would trust to know what is really the right thing to do in the long run....
“Too strong a strong vision can kill you-- you'll walk right over the edge firm in the knowledge of the path in front of you...
“I'd much rather have ‘brownian motion,’ where a lot of microscopic directed improvements end up pushing the system slowly in a direction that none of the individual developers really had the vision to see on their own.”
(Source: Linux kernel email list )
Key Book to Read
Code Quality: The Open Source Perspective. Diomidis Spinellis. Addison Wesley, 2006. ISBN 0-321-16607-8.
- Open Source Software ; Free Software
- Open Development Communities
- The Appropriate Development methodology
- The Open Development Method
- The Extreme Manufacturing method
Managing Open Source Software
- Mastering the chaos of the community: the development process of Firefox and the Mozilla community
- A Highly Efficient Waste of Effort: Open Source Software Development as a Specific System of Collective Production by Jochen Gläser (Research School of Social Sciences; The Australian National University)
"It appears to be very difficult to capture the distinctness of open source software production in a general theoretical framework. The only theoretical framework that compares alternative modes of collective production is transaction cost theory, which is an economic rather than sociological approach to the organization of work and therefore tends to neglect the problem of coordination. This paper describes open source software production in a sociological framework of collective production, i.e. in a framework that emphasizes the actor constellations and the systems of actions that characterize modes of collective production. A generalized sociological description of open source software production is derived from published empirical studies. On the basis of this description, open source software production can be compared to the known systems of collective production, namely markets, organizations and networks. The comparisons reveal necessary conditions for the functioning as well as specific advantages of producing communities."
"The ideology of open source software development is spearheading a shift in the way we approach the process of software development. Not only is it changing the perception of costs associated with projects, but also the management aspect of these development processes. The management of open source projects is very different from a traditional project due to the inherent nature of the objectives, team structure and the benefits involved. Several studies have examined various issues related to the management of these projects. However there is a lack of a study that puts together all the findings so that the interrelationships between these findings can be explored. This study tries to overcome this shortcoming and present the findings of other studies in a comprehensive manner and at the same time look at the entire process from a bird' eye point of view."