Collective Choice Systems

The following material is from an extensive overview and investigation of Collective Choice governance systems at the Life with Alacrity blog. The focus is on how the different methods can be applied online.

General introduction at http://www.lifewithalacrity.com/2005/12/systems_for_col.html

This summary is no substitute for reading the full four-part series.

The Context

" Collective choice systems have been around for a long time. Since at least the birth of democracy in ancient Greece people have made joint decisions about important issues, and since at least the knightly tournaments of the late Middle Age people have competed to be ranked against their peers. Today Western culture especially values diversity of input when implementing any type of choice, believing that wide input from a variety of people provides the fairest result.

The Internet expands this long history of collective choice. However, as we bring collective choice systems onto the Internet, quantifying and programming them, we discover the need to be more analytical and more methodical in the techniques used. Thus we're beginning to learn that we don't know nearly as much about these collective choice systems as we should. There is a need to analyze and study them further, to understand their strengths and weaknesses, and to evaluate their social impact. Fortunately, the social software and online games on the Internet provides the perfect petri dish for doing so.

Before any analysis can occur, however, there is a need for a categorization of systems and a definition of terms. That is the purpose of this article: to lay out at least some of the ways in which collective choices can be made, to organize them, to define them, and to briefly consider them." (http://www.lifewithalacrity.com/2005/12/systems_for_col.html)

Typology of Methods

Broadly, there seem to be three methods of collective choice, divided by the intended result: selection, opinion, or comparison.

Selection Systems

Representative Systems: In a representative system, individuals cast a ballot for someone who will represent their interests.

Deliberative Systems: In a deliberative system, individuals directly make a decision, rather than selecting a representative to do so. Deliberative systems do not have to include voting, and the subcategory of consensus systems described below technically don't, however most modern deliberative sytems do. A deliberative system is the heart of true democracy.

Consensus Systems: In consensus systems people jointly come to a consensus as a group through group interactions. This sort of decision making theoretically avoids the "tyranny of the majority" and likewise can produce more informed decision making. It's a variant of the broader deliberative systems, but one with more group and less individual power. (http://www.lifewithalacrity.com/2005/12/systems_for_col.html)

Opinion Systems

"Opinion systems are a clear subsidiary category to selection systems. An opinion system's main use is as a decision indicator, to show how people will decide or did decide in a representative system, a deliberative system, or both. Current opinion systems tend to be oriented toward actual votes, as opposed to more freeform selection systems (though the delphic polling system shows a more freeform version of the category itself). Opinion systems tend to be push-based (meaning people are asked for their opinions rather than actively offering them), but this isn't required.

All opinion systems tend to have the same general problem, which is figuring out how to use scientific means to determine the actual results of a decision.

There are two general categories of opinion systems: pre-voting (subjective) polling systems and post-voting (objective) polling systems.

Pre-voting Polling Systems: These are polls made before a vote is cast. They're often called "opinion polls" and tend to be conducted via phone. They try and isolate "likely voters" and determine how they will vote.

Post-voting Polling Systems: These are polls taken after a vote is cast. They're typically called "exit polls", as most are conducted as people are leaving a "polling" station (where they cast a vote).

Delphic Polling Systems: An interesting polling method applicable to all sorts of opinion systems is the "delphi poll". This is a specific method of polling which is iterative and anonymous and which supports confidence ratings and feedback. The general idea is that people are polled on a question using not just binary responses, but a full confidence rating (e.g., you would state that you are 60% sure that Bush would be elected, rather than stating that you think Bush would be elected). After polls are collected, the anonymous results--or at least a summary of those results--are shared with the participants, who then poll again. This iterative process continues until a consistent answer is settled upon. By incorporating feedback into the polling process there's the possibility for greatly increased reliability." (http://www.lifewithalacrity.com/2005/12/systems_for_col.html)

Comparison Systems

"Comparison systems allow individual items to be measured up against each other. There are three general categories: comparison ranking systems, which are largely objective and which typically rank people; and comparison rating systems, which more often mix subjective and objective opinions, and which more frequently rate things; and reputation rating systems, which again tend to rank people, but also have a subject and objective mix.

Comparison Ranking Systems: In a ranking system, items in a hierarchy (most frequently people) rise or fall based upon specific, objective, and well-known rules. This is the heart of most multiplayer competitive systems.

Rating Systems: In a rating system, the value of individual items (most frequently goods) rise or fall based upon the largely subjective judgment of individual users. Amazon and Netflix are two examples of stores which provide subjective rating systems.

General Rating Systems are covered in detail here and five-star based systems are discussed specifically here, and Competitive Ranking Systems are covered here.

Reputation Systems: Finally, reputation systems are very similar to ranking systems: items in a hierarchy (most frequently people) rise or fall based upon specific and well-known rules. However, unlike true ranking systems, reputation systems instead base their rules for rise and fall upon other user feedback.

Discussion

Three Methods to Improve Collective Choice systems

Christopher Allen & Shannon Appelcline suggest three main methods to improve ratings systems:

"Granular Ratings: Based on the clumping of ratings to high values, we believed that ratings could be made more useful by increasing the size of a rating scale. Most rating scales are 5-point ranges, so we suggested a 10-point range instead.

Distinct Ratings: Raters can be somewhat arbitrary in how they rate items, varying both from each other and even from themselves (usually over multiple sessions). Thus we believed that providing explicit statements of what each number meant could improve ratings.

Statistical Ratings: Finally we stated that in low volumes ratings could be biased by various quirks of data entry, either malevolent or not, and that ratings could be improved with strong statistical methods being used to polish up data and automatically keep "bad" data in line with "good". (http://www.lifewithalacrity.com/2007/01/collective_choi.html)

The authors say they have applied these principles here at http://index.rpg.net/ which is a reworking of http://www.rpg.net/reviews/